MMohan Vishe
All projects
EvaluationAgentsWriting

Decisions in AI Systems

Suppose you switch an agent to a cheaper model to cut costs, and it gets more expensive. I built the agent and measured when that happens — and the measurement corrected my own explanation.

View code on GitHubPythonMIT

The number that lies

In an agentic loop you don't pay per token. You pay per completed task, and the two move in opposite directions often enough to be dangerous.

A cheaper model at the orchestrator — the node that plans, picks a tool, reads the result and decides again — takes more turns to get there, and fails more often. Both effects land on the same bill. In the worked example below, a model 4× cheaper per token comes out 16% more expensive per successful task, while a cost dashboard, which counts calls rather than outcomes, reports a 23% saving.

Then I put prompt caching into the cost model, and the answer flipped.

... ... Same two models, three different answers Orchestrator node, illustrative parameters. The cheaper model is 4x cheaper per token in every column. $0.00 $0.01 $0.02 $0.03 $0.04 $0.0300 $0.0231 cheaper model -23% Cost per attempt what a cost dashboard shows $0.0333 $0.0385 cheaper model +16% Cost per successful task what you actually pay for $0.0157 $0.0115 cheaper model -27% Per successful task, with prompt caching capable model cheaper model
The same two models and the same workload, priced three ways. Only the middle column is what you actually pay for — and with caching even that reverses.

Most APIs now bill re-sent context at a steep discount, and almost everything an agent turn sends was sent the turn before. With cache reads at a tenth of the normal price, the cheaper model turns into a 27% saving per successful task. What doesn't change: it still fails 40% of tasks against 10%, and still makes seven round trips against three.

So the honest claim is narrower than "cheap models cost more", and more useful:

A cheaper model does not automatically reduce the cost of an agentic system, and a more capable one doesn't automatically improve it. Which case you're in depends on turns, success rate, caching and latency — and you only learn those by measuring.

Why the loop punishes a weaker model

Three multipliers sit between the price per token and the price per task.

Turn inflation. A weaker orchestrator picks a plausible but wrong tool, passes an argument in the wrong shape, reads the error and re-plans. Each misstep is at least one more turn.

Context growth. Chat APIs are stateless, so every turn re-sends the whole conversation: system prompt, tool definitions, and every call and result so far. Tokens grow with the square of the turns, and the extra turns are the expensive ones because they come last.

... ... Every turn re-sends everything before it Input tokens per turn of one attempt. Illustrative parameters from cost_model.py. 0 2,500 5,000 7,500 10,000 1 2 3 capable model: 3 turns 14,400 input tokens 1 2 3 4 5 6 7 cheaper model: 7 turns 44,800 input tokens system prompt + tools + request earlier turns, re-sent 2.3x the turns, 3.1x the tokens
2.3× the turns, 3.1× the tokens. The grey block is re-sent on every single turn.

Failure. Cost per successful task is cost per attempt divided by the success rate, so a weaker model's bill gets divided by a smaller number.

Together they give one rule you can apply before committing to a swap. A cheaper model is only cheaper per successful task if price ratio > token ratio × success ratio. The pricing page gives you the first. Only an evaluation gives you the other two.

What an evaluation has to cover

"Run an eval" is easy to say and easy to do badly. A single prompt run once measures one sample of a random process. A benchmark measures someone else's tasks. A cost dashboard measures calls, not outcomes.

An agent conversation can be graded at five levels, and each one answers a question the others can't.

... ... What to evaluate in one multi-turn conversation A conversation from the test agent. Each band is a level you can grade, and each needs its own checks. Customer Orchestrator Tools turn 1 call find call orders reply turn 2 call update reply 1 Node one call 2 Trajectory tools, arguments, order, retries 3 Turn was this reply right? 4 Conversation end state + every reply, across turns 5 Reliability: run the whole conversation k times and count how often all k succeed.
One conversation from the test agent. Each band is a level you can grade — and each needs its own checks.

Every level gets measured on the same three axes, always per completed task rather than per call: accuracy, latency and cost. Two of those deserve emphasis, because they're the ones teams skip:

  • Reliability, not just success. Models sample, so run each case k times. pass^k — every one of k runs succeeding — is the number that matters for an agent acting on a user's behalf, because every user gets one run. An agent that's right 90% of the time is right five times running only 59% of the time.
  • Latency per task and at the tail. Seven fast round trips can lose to three slow ones, and a median hides it. In the experiment below the median said the cheap model was faster while the time per successful task was three times worse.

Then decide with gates, in order: quality, then latency, then cost. A configuration that fails the quality bar is out, however cheap it is. Cost only ranks what survives. A weighted score sounds more balanced, but it lets a precise cost number outvote a noisy quality one.

A test subject, and two runs of 240 measured conversations

To keep this from being an argument about arithmetic, I built the agent it describes and evaluated it: a customer-support agent for a made-up online store, with 8 tools, 4 policy rules, and 24 graded conversations including multi-turn ones where the customer changes their mind. Grading is deterministic, on the store's end state rather than on wording.

The headline comes from the second run. With the graders frozen first, documenting every tool parameter doubled the 3B agent's task success (20% → 41%) and cut the price advantage it needs to break even from 4.1× to 1.5× (95% CI 1.0–2.2×), still above the router's 1.1×. The first run comes first here, because it's what showed where to look.

Every conversation ran five times on each of two sizes of the same open model family, locally: 120 conversations per model, plus a separate 48-message set for the leaf node below. The first run took 56 minutes on one consumer GPU and cost nothing.

... ... Measured: the same agent on a 3B and a 7B model 24 conversations x 5 runs each, local Qwen2.5-Coder models. Green dot = better. No dot = neither is better. ORCHESTRATOR (decision node) Success rate 3B 20% 7B 62.5% All 5 runs succeed (pass^5) 3B 4% 7B 50% Identified the customer first 3B 67% 7B 98% Model calls / conversation 3B 4.3 7B 4.3 Rejected tool calls / conv. 3B 3.14 7B 0.04 Tokens / successful task 3B 30,274 7B 7,382 Latency / call (~2 s fixed) 3B 2.63 s 7B 2.58 s Conversation, p95 3B 25.0 s 7B 20.8 s Time / successful task 3B 56.0 s 7B 17.8 s ROUTER (leaf node) Accuracy 3B 87% 7B 98% Tokens / correct call 3B 149 7B 132 Latency / call (~2 s fixed) 3B 2.25 s 7B 2.37 s
First run, both models, both nodes. On call counts and average latency they look identical. On outcomes they are not close.

In the first run the two models are indistinguishable on the metrics a dashboard shows: 4.3 model calls per conversation each, about 11 seconds of model time each (most of it a fixed ~2 s per call in the harness, which flatters the slower model). Per outcome, the 3B needed 4.1× cheaper tokens just to break even (95% CI 2.4–7.5× over 24 tasks), and it never gets that far, because it fails the quality gate first: 20% of tasks against 62.5%.

Pricing prompt caching into the measured traces doesn't rescue it either — break-even moves to 4.24×, slightly wider. Caching flips the worked example above but not the real one, because the measured gap comes from the success rate, not from context growth.

... ... Reliability: how often every one of k runs succeeds (pass^k) Same tasks, repeated. A model that is right 'most of the time' is wrong for somebody every day. 0% 25% 50% 75% 100% k=1 k=2 k=3 k=4 k=5 3B 4% 20% 7B 50% 62.5%
First run. Ask for the same conversation to work five times running and the small model holds up 4% of the time.

The mechanism was also not quite the one I predicted, which is the best argument for measuring rather than reasoning. Model calls per conversation were identical. The extra cost came from rejected tool calls — 3.14 per conversation against 0.04 — pushing errors into a context that every later call re-sends, and from the small model giving up rather than looping: answering from a failed lookup, inventing a SKU, or quitting after one error. In 18% of conversations it hit the call cap and never answered at all.

Then the same two models on a separate leaf-node task in the same domain — an intent router: one call, fixed output, no loop:

Intent router3B7B
Accuracy (95% CI)87% (80–93%)98% (94–100%)
Tokens per correct call149132
Latency per call (about 2 s fixed overhead)2.25 s2.37 s

Break-even collapses from 4.1× to 1.13× (95% CI 1.04–1.24×). Almost the entire price gap passes straight through, and on cost the small model wins comfortably. Whether you'd ship it is a quality question about what happens after a misroute, not a cost one.

Same two models, two kinds of node, opposite answers. Which is the whole argument.

... ... Where each model tends to belong Decision nodes compound errors across turns; leaf nodes do one narrow job in one call. Customer message Router classify the intent, 1 call Orchestrator plan, pick tools, read results, decide again Tools orders, stock, tickets Formatter tool results to a fixed reply schema loop: many turns decision node: start capable leaf node: a small model is often enough "Tends to", not "always": the evaluation decides each one.
Where each model tends to belong — 'tends to', because the evaluation decides each node.

The second run: document the tools

About a third of those rejected calls had a plain cause: only one of the agent's 13 tool parameters had a description, and 117 of the small model's 377 rejected calls copied a parameter's schema into the argument. So the second run described all 13, each with an example value that isn't in the store. It also moved the router to temperature 0 and put it in front of the agent, and logged Ollama's own timings. The two models, the 24 tasks and the seeds stayed the same, and the graders were frozen and committed before the first trace.

First runSecond run95% CI
3B task success20%40.8%27–55%
7B task success62.5%68.3%52–83%
All 5 runs succeed, 3B / 7B4% / 50%12% / 54%
Rejected tool calls per conversation, 3B3.141.43
Calls that copied a schema into the argument, 3B1170
Break-even, orchestrator4.10×1.47×1.01–2.16×
Break-even with the re-sent prefix at 10%4.24×1.74×1.26–2.51×
Break-even, router1.13×1.09×1.02–1.21×

The gap narrowed and the ordering held: the orchestrator is still further from parity than the router, and the quality gate still decides first, at 41% against 68%. The whole pipeline, router included, also breaks even at 1.47×, because the router is only about 3% of its tokens.

Several setup changes went in together, so this run doesn't say how much each contributed. The parameter docs are the most likely driver, since the schema-copying calls went to zero, but the router's label entered the context in the same run. Separating them is the next test. The fixed 2 seconds per call also turned out to be the harness: measured against Ollama's own timings, request overhead is 0.03 s.

The same argument as a product decision

The companion piece is the non-quantitative version: setting the bar for a model change on an internal knowledge product, where accuracy gates before cost is considered at all, rather than the two being weighed together.

Weighing them sounds balanced. It's also how teams talk themselves into a worse product, because the cost figure is precise, the quality figure is noisy, and the precise number wins the argument regardless of which one matters.

Reproducing it

Everything is public and runs on a laptop. The cost model needs only the standard library; the experiment needs Ollama and two free local models. Every figure on this page is generated by the same code that produced the numbers, and the raw traces for all 480 conversations across both runs are in the repository.

python cost_model.py                      # every illustrative number, including the caching case
cd experiment && python analyze.py        # recompute the first run from the published traces
EVAL_RESULTS=results-run2 python analyze.py   # the same for the second run

Both write-ups describe how decisions were made rather than the systems behind them: no proprietary architecture, no client details, and worked examples that say where they're illustrative.