The Loop Is the Runtime

Prompt engineering asks: what should the model say next?

Loop Engineering asks harder questions:

• What state is true now?
• Which transition is legal?
• What evidence would prove it happened?
• How much more work may be spent?
• What happens after a timeout or partial side effect?
• When must the system stop?

The moment a model calls a tool and waits for the world to answer, the prompt is no longer the system. It is one computation inside a runtime.

An agent is therefore not “an LLM with tools.” It is a stateful program that uses inference to select or construct transitions under uncertainty.

A single call predicts; a loop operates

A single inference call maps input context to output tokens:

y ~ model(context, policy, sampling)

A loop repeatedly combines model output with environment change:

state_t
  -> observation_t
  -> proposed_action_t
  -> validation_t
  -> environment_result_t
  -> verified_state_t+1

The environment result may be a database receipt, test output, browser state, human approval, repository diff, API error, or timeout. It can contradict the model's prediction.

This difference is the foundation. A model can write “the deployment succeeded.” Only the deployment system's receipt can establish that state.

ReAct studies interleaving reasoning traces and actions, allowing observations to inform later reasoning. The research mechanism was important because it moved beyond producing a complete answer before interacting. A production runtime must add typed state, authority, durability, budgets, and recovery that are not established merely by an interleaved prompt format.

Define the state before defining the prompt

State is the smallest typed representation needed to decide the next legal transition and verify completion.

For a repository-change agent:

{
  "task_id": "task_7",
  "goal": "fix failing CSV export test",
  "repo_revision": "4e91...",
  "phase": "failed_check",
  "changed_files": ["src/export.py", "tests/test_export.py"],
  "last_check": {"command_id": "check_19", "exit": 1},
  "open_failures": ["test_escaped_newline"],
  "budgets": {"model_calls_left": 5, "wall_seconds_left": 240},
  "authority": {"write_repo": true, "push": false, "deploy": false},
  "event_cursor": 31
}

This is not the whole transcript. It excludes speculative thoughts and superseded plans. Each field has provenance in the event log.

Model-visible context is assembled from state, current evidence, eligible tools, and recent events. The model may suggest a state update, but the reducer accepts it only after validating the associated event.

Events are facts; reflections are hypotheses

An append-only event can say:

{
  "event_id": "evt_32",
  "kind": "check.completed",
  "command_id": "check_19",
  "exit_code": 1,
  "artifact": "logs/check_19.txt",
  "started_at": "...",
  "finished_at": "..."
}

A reflection can say:

The failure is probably caused by newline escaping in the CSV writer.

The event is an observed fact. The reflection is a useful hypothesis that may guide the next action. Do not store both in the same field under “memory.”

Reflexion studies verbal feedback retained across trials. The mechanism can help an agent avoid repeating a mistake in evaluated settings. In production, a verbal reflection needs source links, expiry, and a confidence status. A wrong diagnosis repeated in later context can lock the loop into a failure attractor.

A transition is a contract

Represent each transition with:

name
allowed source states
preconditions
typed action
required authority
budget charge
timeout and retry class
success postcondition
failure states
compensation or escalation

Example:

transition: apply_patch
from: inspected | failed_check
precondition: base revision unchanged; patch touches allowed paths
authority: repository.write
charge: one mutation attempt
success: tool receipt contains new tree hash and changed paths
failure: conflict | invalid_patch | policy_denied | timeout_unknown

timeout_unknown matters. A timeout does not prove no effect occurred. The runtime checks an idempotency key or current repository tree before retrying.

This is why tools should be narrow typed operations rather than a universal shell with prose instructions. The action language defines what the loop can do.

Plans are revisable, not executable truth

Plan-and-Solve separates generating a plan from executing subtasks in evaluated reasoning tasks. Planning is useful because it exposes missing steps and budgets before action.

An agent plan should contain:

• objective and success evidence;
• ordered or dependent steps;
• information gaps;
• expected tool and model cost;
• risky or irreversible transitions;
• checkpoints and stop conditions.

The plan is invalidated when observations change its assumptions. If inspection reveals a generated file, editing it directly may be the wrong step. If the repository revision changes, the patch plan needs rebase or restart.

Do not grant authority to the whole plan. Authorize each transition against current state. A human approving “prepare a release” has not necessarily approved deleting a database discovered three steps later.

Tool use needs a control plane

Toolformer trains a model to decide which APIs to call, when, with what arguments, and how to incorporate results. That addresses capability selection inside model behavior.

Production control remains external:

model proposes tool + arguments
runtime checks schema
runtime checks tenant, principal, object, budget, and workflow state
tool executes with idempotency and timeout
runtime validates receipt
event log records effect
model observes bounded result

The tool registry should describe:

• input and output schemas;
• read or write effect;
• required permissions;
• idempotency semantics;
• timeout ambiguity;
• rate and cost budget;
• data classification;
• side-effect reversibility;
• health and version.

A tool being technically callable does not make it eligible for the current step.

Stop conditions are executable policy

“Stop when done” is not a stop condition. It delegates the terminal decision to the same uncertain model that is generating work.

Define terminal states:

Verified success

Named postconditions hold. Tests pass, target artifact exists, citations support claims, or transaction receipt is committed.

Infeasible

Required evidence, capability, permission, or dependency is unavailable. More retries cannot change it inside the task boundary.

Budget exhausted

Maximum model calls, tokens, tool cost, mutations, elapsed time, or human review is reached.

Policy denied

The requested or discovered action violates authority or safety rules.

Repeated state

The loop revisits an equivalent failure state without new evidence. A cycle detector stops or escalates.

Deadline missed

No remaining path can finish before the outcome deadline.

Human escalation

Consequence, ambiguity, or approval policy requires a person.

The loop returns a typed terminal result with evidence. A controlled needs_approval is not the same as task failure.

Trace one complete task

Consider an illustrative repository repair. The numbers are invented, not customer data.

Step	Transition	Calls	Input	Output	Tool time	Result
1	accepted -> scoped	1	2,200	420	0.3 s	plan
2	scoped -> inspected	2	7,400	710	2.2 s	relevant files/tests
3	inspected -> changed	1	6,100	980	0.8 s	first patch
4	changed -> failed_check	1	4,800	260	11.0 s	one failing test
5	failed_check -> changed	2	9,600	880	1.3 s	diagnosis and fix
6	changed -> verified	1	5,200	210	12.4 s	checks pass
7	verified -> committed	0	0	0	0.4 s	receipt stored

Totals:

8 model calls
35,300 input tokens
3,460 output tokens
28.4 seconds tool time
2 mutations
1 failed verification
1 verified terminal outcome

The first patch was not a failed task. It was a failed transition attempt that produced evidence. Whether that evidence was worth its cost depends on how efficiently the loop used it.

State growth can quietly become the bottleneck

If the full transcript is replayed on every call, input tokens grow roughly with accumulated history. A loop with ten calls can pay for early content ten times.

Use:

• stable prefix caching for policy and tool schemas;
• typed current state instead of full replay;
• event retrieval by relevance and provenance;
• recent exact observations where wording matters;
• summaries that link back to original events;
• separate scratch space with short lifetime.

Track context composition per call. If the same failed log is sent six times after its cause is resolved, the reducer is not maintaining state.

The context rules from The Window Is Not the Memory become runtime memory rules here.

Parallel branches are child runtimes, not extra thoughts

An agent may launch three searches, two candidate patches, or several independent analyses. Parallelism can reduce critical-path latency and increase exploration. It also creates ownership questions:

• Which branch may mutate shared state?
• How is a winning branch selected?
• What cancels the losers?
• Can outputs be combined safely?
• What budget does each child inherit?
• What happens if a loser finishes after the task commits?

Give every branch an identity, parent, budget, state snapshot, and merge rule:

{
  "branch_id": "br_7b",
  "parent_event": "evt_18",
  "base_revision": "4e91...",
  "purpose": "candidate patch B",
  "authority": {"sandbox_write": true, "shared_write": false},
  "budget": {"calls": 3, "seconds": 90},
  "merge_policy": "verified_winner_only"
}

Read-only searches can often run concurrently. Mutating branches should work in isolated sandboxes or use optimistic concurrency against a declared base revision. Do not allow two model branches to write the same live object and hope the later prompt resolves it.

The coordinator verifies candidates under the same test and policy contract. It does not choose the most persuasive explanation. When one candidate wins, the runtime cancels remaining work and records cancellation receipts. Measure tokens and tool time computed after cancellation, because late cancellation turns parallel exploration into invisible capacity waste.

Merging is a transition with preconditions. Two individually valid patches may conflict or jointly fail tests. Two retrieved facts may refer to different revisions. Re-run verification on the combined artifact.

Human-in-the-loop is an asynchronous state, not a modal dialog

A workflow that needs approval can wait minutes or days. The process may restart, credentials may expire, prices may change, and another actor may modify the target while a person reviews.

Represent approval as a durable transition:

proposed -> awaiting_approval -> approved | rejected | expired | superseded

The approval request binds:

• normalized action and artifact hash;
• evidence and verifier result shown to the reviewer;
• principal requesting the action;
• reviewer role required;
• maximum amount or scope;
• expiry time;
• state or revision precondition;
• whether changes require a new approval.

An approval for patch hash abc does not authorize a regenerated patch def. Approval for a transfer at one balance snapshot may need revalidation before commit. The runtime checks current state after approval and before execution.

Human feedback also has different meanings. “Looks good” may approve an effect, label an outcome, suggest a new requirement, or merely acknowledge a preview. Use typed decisions rather than inserting the sentence into context and guessing.

Human latency belongs in the trace and ROI model. Automating 30 seconds of inference around a six-hour approval queue may have little lead-time value, though better evidence can reduce reviewer time. Measure:

time awaiting reviewer
active review time
approval and rejection rate
requests returned for missing evidence
expired or superseded approvals
post-approval validation failures

Design escalation packets to minimize reconstruction: current state, proposed action, consequence, evidence, alternatives, budget spent, and the exact decision requested. A person should not need to read the full agent transcript.

Verification is a transition, not a final ceremony

Do not reserve verification for the end. Place cheap checks after transitions where error becomes expensive to carry.

parse after generation
type-check after edit
unit check after local change
integration check before merge
policy check before authority expansion
smoke check after deployment
outcome monitor after release

Verification depth should grow with consequence and accumulated work. Re-running a full suite after every character is wasteful; waiting until deployment to discover a syntax error is worse.

Each verifier returns a typed result with scope. “Tests passed” is incomplete unless the runtime knows which tests, revision, environment, and time. A local unit test does not prove production health. A model critic is not independent evidence unless its limitations are accepted for that gate.

Failed verification changes state. The next call receives the failure artifact and remaining budget, not a vague instruction to “try again.” If the same verifier fails on the same signature without new evidence, cycle detection stops the loop.

Recovery is part of every action

The loop can fail between any two observations:

• model request times out;
• tool accepts action but response is lost;
• process crashes after effect but before event append;
• state reducer fails;
• human approval arrives after task deadline;
• another actor changes the object concurrently.

Every effect needs an idempotency key and a way to query its status. The runtime records intent before dispatch and receipt after observation. On recovery, it reconciles unknown attempts instead of blindly replaying.

Durable execution receives a full treatment later in the AI-native development article. The minimum Loop Engineering rule is simple: if the runtime cannot answer “did this action happen?” after restart, the action is not safely automatable.

A builder playbook

1. Draw the state machine

List typed phases, legal transitions, terminal states, and failure edges. If the diagram is only a circle labelled think-act-observe, it is incomplete.

2. Define outcome evidence

Write machine-checkable postconditions for success where possible. Name when human judgment is required.

3. Separate events, state, and reflection

Events are immutable observations. State is a validated reduction. Reflections and plans are model-authored hypotheses with provenance and expiry.

4. Build the eligible tool registry

Include schemas, permissions, effects, idempotency, timeouts, costs, and receipts. Select tools from trusted workflow state before exposing them to the model.

5. Install budgets and cycle detection

Limit calls, tokens, mutations, tool cost, elapsed time, and repeated equivalent states. Stop before a retry storm becomes the product.

6. Trace transitions, not just model calls

Record state before/after, evidence, proposal, validator result, authority decision, tool receipt, cost, latency, and terminal outcome.

7. Replay failures

Run the runtime against stored event traces with model and tool stubs. Test timeouts before effect, timeouts after effect, concurrent state change, invalid receipt, denied authority, and crash recovery.

The decision

Use a single call when one bounded prediction can be independently checked. Build a loop when the task must gather new evidence, perform multiple transitions, or recover from intermediate failure.

Then treat the loop as the runtime: typed state, append-only evidence, constrained actions, current authority, explicit budgets, verified postconditions, and executable stops.

Prompts can make the next proposal better. Loop Engineering makes the whole task survivable.

References

• ReAct: Synergizing Reasoning and Acting in Language Models, Yao et al., ICLR 2023. Establishes interleaved reasoning, action, and observation in evaluated language agents.
• Toolformer: Language Models Can Teach Themselves to Use Tools, Schick et al., NeurIPS 2023. Studies learned decisions about API use.
• Reflexion: Language Agents with Verbal Reinforcement Learning, Shinn et al., NeurIPS 2023. Explores verbal feedback retained across agent trials.
• Plan-and-Solve Prompting, Wang et al., ACL 2023. Separates planning from subtask execution in reasoning prompts.

The workflow trace is illustrative. Exact invented inputs and totals are retained in the claim ledger.