An agent that gets each step right 90 percent of the time finishes a nine step task correctly about 39 percent of the time. That one piece of arithmetic decides more agent architectures than any framework choice, and most teams meet it only after they have shipped.
Anatomy of an agent loop
Strip away the vocabulary and an agent loop has four moves. Your process shows the model the state so far, the model decides what to do next, your process executes that decision, and the result goes back into context for the next turn. Yao and colleagues named this interleaving of reasoning and acting in the ReAct paper, and reported absolute success rate gains of 34 points on ALFWorld and 10 points on WebShop over methods that did only one or the other. Nothing in that description requires a framework. A loop, a list of messages and a dictionary of functions is the whole of it.
What separates an agent from a workflow is not the loop, because both loop. Anthropic draws the line at who chooses the path: in a workflow, your code decides which call happens next; in an agent, the model does. Ask that one question, who picks the next step, and everything else follows from the answer, including cost, latency, how testable the thing is, and how much damage a bad turn can do.
Frameworks obscure this distinction rather than clarifying it, which is my main reservation about starting with one. Two libraries will both hand you something called an agent, where one runs a fixed three step chain and the other hands the model an open budget of twenty turns. Same word, wildly different risk. Anthropic makes the same point from the other direction, advising developers to start with the API directly because incorrect assumptions about what sits under the abstraction are a common source of error. My own rule is that I will not put a loop in front of users until I can draw its exits on a whiteboard from memory, and a framework that makes that hard is costing me more than it saves.
Key takeaways
Who picks the next step is the architectural question. If the model picks it you have an agent, and you inherit its variance. If your code picks it you have a workflow, and you can write a test for it.
Per step accuracy compounds. Nine steps at 90 percent lands near 39 percent end to end, so shortening the loop buys more reliability than any prompt rewrite will.
Adding an upfront planning step made my support assistant worse. Resolution fell from 61 percent to 54 percent, because a plan written before the first observation arrives is a guess with extra confidence.
Where the assistant stands, and what an agent would add
Last part the assistant grew hands. It could call a ticket status lookup and an escalation function, and a small loop ran until the model stopped asking for tools. Retrieval over the product docs and the ticket archive was already working from Parts 8 to 13. So the thing does its job: a support engineer asks about a billing edge case, three chunks come back, one ticket lookup runs, and an answer lands.
What it does not have is discretion. Every request follows the same shape, and if a question needs the changelog checked first and then a ticket looked up conditionally, the loop as written will muddle through or give up. Autonomy is the thing that would fix that. This part is about whether granting it is worth what it costs, and my answer for most of the traffic is no.
Traffic shape is what settles the argument. When I bucketed 2,000 real questions from the internal channel, 84 percent needed one retrieval and at most one lookup. Those questions gain nothing from an agent, and they pay for it in latency and tokens all the same. Roughly 11 percent were multi part, and about 5 percent were genuinely open ended. Autonomy earns its keep on the last 16 percent and nowhere else.
Planning styles and what each one costs
Planning is a separate axis from autonomy, and conflating the two is common. An agent can plan implicitly, one step at a time, or explicitly, by writing a plan down before it acts. Four arrangements cover almost everything you will meet.
| Planning style | Who writes the plan | Extra model calls | Where it breaks |
|---|---|---|---|
| Single pass, no plan | Nobody | 0 | Question needs two dependent lookups |
| Interleaved, ReAct style | Model, one step at a time | One per step, unbounded | Loop drifts with no progress check on it |
| Plan then execute | Model, once, upfront | 1 | First observation invalidates the plan |
| Plan with replanning | Model, at checkpoints | 1 plus 1 per replan | Replans fire too often and cost doubles quietly |
Here is where I part company with most of the advice. Adding an explicit planning step is widely recommended as a way to make an agent more reliable, and on my support assistant it did the reverse. Resolution on a fixed eval set fell from 61 percent to 54 percent and median latency rose by about 1.8 seconds. Cause was not subtle once I read the traces. Plans were written from the question alone, before any retrieval had run, so the model committed early to an assumption about which system held the answer. When retrieval later contradicted that assumption, the loop kept executing the plan instead of noticing.
Upfront planning pays when the shape of the task is knowable before you see any data, such as a nightly job that must touch four named systems in a fixed order. For anything where the first observation might change the destination, interleaving wins, which is exactly the finding ReAct reported and exactly the thing a bolted on planner undoes.
Arithmetic that kills long agent loops
Every step in a loop is a chance to be wrong, and the chances multiply. If each step is independently correct with probability p, an n step task succeeds with probability p to the power n. Independence is generous, since a wrong observation early tends to poison everything after it, so treat this table as a ceiling rather than a forecast.
| Steps in the loop | Per step 95% | Per step 90% | Per step 80% |
|---|---|---|---|
| 3 | 85.7% | 72.9% | 51.2% |
| 5 | 77.4% | 59.0% | 32.8% |
| 8 | 66.3% | 43.0% | 16.8% |
| 10 | 59.9% | 34.9% | 10.7% |
| 12 | 54.0% | 28.2% | 6.9% |
Benchmarks that measure consistency rather than best effort make the same point harder. Tau bench scores agents with pass to the power k, which counts a task solved only when all k independent attempts succeed, and reports function calling agents falling below 25 percent at k equal to 8 in its retail domain while scoring far higher on a single attempt. Users do not get to retry until it works, so that stricter number is the one that matches what they actually experience.
Token cost compounds along the same curve, and rather worse, because context accumulates. Every turn resends the whole transcript, so a six turn loop does not cost six times a single call, it costs closer to fifteen or twenty times, depending on how large your observations are. On my assistant, a routed single call averaged 2,100 prompt tokens. Same question through the agent lane averaged 24,600 across its turns. That ratio, near twelve to one, is the number I bring to any conversation about whether autonomy is affordable, and it is the reason the routing decision is a cost decision before it is an architecture one.
Failure modes of an agent loop
Five failures account for nearly everything I have had to debug in a live loop. Learning to recognise them from a trace is worth more than any amount of prompt tuning, because each has a structural fix and none of them is fixed by asking the model more nicely.
First is drift with no progress, already described above: identical calls repeating until the budget runs out. Second, and more insidious, is early commitment. Model picks the wrong tool on turn one, and every subsequent turn rationalises that choice rather than revisiting it, because the transcript now reads as though the decision was already justified. Cause is almost always overlapping tool descriptions. When I cut the assistant from nine tools to five and rewrote the survivors so their descriptions had no vocabulary in common, correct first tool selection went from 71 percent to 89 percent. Anything past roughly seven tools in a single loop is asking the model to do classification it is not being paid to do.
Third is observation overflow. A tool returns 40 KB of JSON, all of it lands in context verbatim, and by turn four the model is reasoning over mostly noise while you pay for the privilege. Truncating and summarising each observation to a 1,200 token cap before appending it took my median prompt size at turn four from 11,400 tokens down to 4,900, and slightly improved accuracy rather than hurting it. Raw tool output is rarely the thing the model needs.
Fourth is silent success, which is the one that reaches users. A tool errors, your loop appends the error string as though it were data, and the model reads it as a finding and writes a fluent answer around it. I shipped this for six days. Staff were told a customer had no open tickets when the ticket API had in fact returned a 503. Mark tool errors distinctly rather than as plain content, and treat a failed call as a reason to consider stopping rather than a reason to continue. Fifth is an irreversible action taken on a misunderstanding, which is why writes belong outside the loop entirely.
Workflow patterns that beat an agent in production
Four arrangements cover most of what teams reach for an agent to do. Prompt chaining runs fixed steps in sequence with a check between them. Routing classifies the input and sends it down a specialised path. Parallelisation runs independent subtasks at once, or runs the same task several times and votes. Evaluator and optimiser pairs a generator with a checker in a bounded loop. Every one of these is code you can read, step through and write a test against, which no agent loop will ever be.
I spent three weeks defending an agent I should have deleted in the first week. Version one of the support assistant was a single agent holding five tools with a step budget of 8, and on a 120 question eval set it resolved 61 percent, at a median of 4.3 model calls and 9.4 seconds. My response to every failure was to rewrite the system prompt, and I did that eleven times. Replacing it with a small classifier feeding three fixed paths, keeping an agent only for the open ended lane, took two days and moved resolution to 74 percent, median calls to 1.4 and median latency to 2.1 seconds. Model never changed. What changed was how many decisions I had handed to it.
Below is the lookup I now use in design reviews. Read down the left column until a row describes your task, then build what the middle column says and stop arguing about it.
| If the task looks like this | Build this | Reason |
|---|---|---|
| Steps are known before you see any data | Prompt chain in your own code | One path, no surprises, trivially testable |
| Input falls into a few clear categories | Classifier plus one fixed path per category | Each path gets its own prompt and its own tests |
| Independent subtasks, and you need speed or a vote | Parallel calls, aggregate in code | Latency becomes the max, not the sum |
| Clear pass or fail criterion, refinement helps | Generator plus checker, bounded iterations | Checker is testable on its own, loop cannot run away |
| Step count genuinely unknowable, ground truth available each turn | Agent loop, with budget and no progress check | Nothing simpler can express it |
| Step count unknowable and no ground truth per turn | Do not build it yet | An agent that cannot tell whether it is making progress will not stop making things worse |
Stopping conditions and loop budgets
Once you have decided an agent is warranted, most of the engineering left is deciding how it stops. Five exits are worth wiring, and I would not run a loop in front of users without all five. A step ceiling caps the worst case bill. A wall clock ceiling protects the caller, since a request that has already taken 20 seconds is a failed request regardless of what happens next. A repeated call detector catches the loop that has stopped learning. An irreversibility gate pulls anything that writes, refunds, deletes or emails out of the loop and in front of a person. And a token ceiling on accumulated context stops the slow death where each turn grows the prompt until the context window ends the conversation for you.
Choosing the step ceiling is easier than it looks. Take your eval set, log the number of tool calls on successful runs, and set the ceiling at roughly the 95th percentile of that distribution. Mine came out at 6, which was two lower than the 8 I had picked by instinct, and lowering it cost nothing in resolution while removing a class of long expensive failures. Anything past the point where success is rare is not a task in progress, it is a task that has already failed and is still billing you.
Evaluation changes shape once a loop is involved, and this catches people out. A single call is scored on its output. A loop has to be scored on its trajectory as well, because two runs can produce the same correct answer while one took two turns and the other took seven and called a write tool it had no business touching. At minimum, log per run: number of turns, tools called in order, whether any exit other than a normal finish fired, and total tokens. Score correctness and efficiency separately, since a run that is right but takes nine turns is a run that will be wrong next week when the question is slightly harder. Part 20 builds the eval set properly, and doing it before you widen an agent budget is the cheaper order.
What the docs undersell
Every framework hands you a max iterations setting and presents it as the safety valve. It is a billing cap. Reliability comes from the exits you add underneath it, particularly the repeated call detector and the rule that no irreversible action ever executes inside the loop. Part 19 takes up those approval gates properly, and Part 25 covers tracing a loop well enough to know which exit fired and why.
Build the router first, keep the agent for the tail
If you are making this call today, build the classifier and the fixed paths first, measure precisely what falls through them, and give an agent only that residue. Pattern I would steer a team away from hardest is the single agent holding every tool, with a system prompt asked to do the routing that a 40 line classifier would do better. It is the hardest shape to test, the easiest to keep tuning forever, and it hides its cost inside a token bill nobody reads until quarter end. Agents are not a maturity level you graduate to. They are an expensive answer to a specific question, which is what to do when you cannot know the number of steps in advance.
One thing to do on Monday: pull the distribution of tool calls per request out of your logs. If the median is 1 or 2, you are paying agent prices for router work, and the fix is a day of effort rather than a rewrite. Treat that distribution as a running metric, the same way you would monitor a model in production, because it drifts as your traffic changes. Part 16 takes the next step and standardises how tools reach the model at all, through Model Context Protocol, which matters a great deal once more than one service needs the same set of tools. Part 14 on tool calling remains the mechanical foundation under all of it.
References
- Building effective agents, Anthropic Engineering, for the workflow versus agent distinction used throughout this part, the four workflow patterns, and the guidance to add complexity only when it demonstrably improves outcomes.
- ReAct: Synergizing Reasoning and Acting in Language Models, Yao and colleagues, for the interleaved loop and the ALFWorld and WebShop success rate gains quoted above.
- Tau bench: A Benchmark for Tool Agent User Interaction in Real World Domains, Sierra, for the pass to the power k metric and the retail domain consistency figures.
- A practical guide to building agents, OpenAI, for the criteria on validating a use case before committing to an agent rather than a deterministic path.
About The Author
DrJha