What one AI question actually costs — an instrumented routing pilot

September 2026. About a nine-minute read.

Our design partner runs a fleet-operations platform. Inside it lives an AI assistant that answers owners' questions — driver performance, attendance, coaching, vehicle compliance — by calling ~23 internal data tools in a loop and synthesizing the results.

Like most production assistants, it was pinned to one model: every question, easy or hard, went to a fast, inexpensive model. That's the rational default — until you look closely at what it does to the hard questions. So we looked closely. This is the full pilot: what we built, how we tested it on the customer's real question set, and every number we measured along the way.

The problem, stated honestly

Three facts collide in any production assistant:

Most questions are easy. "How many active drivers do we have?" is a lookup. A fast model answers it perfectly, quickly, and almost for free.
Some questions are genuinely hard — cross-referencing two data domains, aggregating raw weekly records, writing a fair 360 review of an employee. On these, the fast model's mistakes are the quiet kind: a miscounted total, a missing caveat. The answer reads fluently. Nobody re-adds the raw rows to check it.
Upgrading everything to the strong model costs ~3–4× on every question — easy ones included. For this workload we measured it: the strong model everywhere would have cost ~3× the pinned baseline.

The answer is routing: size each question first, then pick the model. The interesting part is doing that accurately, deterministically, and cheaply — and being able to prove all three from the ledger.

What we built

A judge in front of the model pool. Every incoming question is first read by a small, fast model — the judge — which classifies it into a tier (SIMPLE / MEDIUM / COMPLEX / REASONING). Tiers map to models: for this pilot, SIMPLE and MEDIUM → claude-haiku-4-5, COMPLEX and REASONING → claude-sonnet-5. Five design decisions did the heavy lifting:

1. The judge reads the application, not just the question. Here's the thing question-wording can't tell you. Compare:

"Which employees called off last week, and why?"
"Who are my worst 3 quality-flagged drivers, and why?"

Identical shape. Opposite difficulty — because in this customer's app, the call-off tool returns rows that already include the reason, while the quality tool returns raw week-by-week entries the model must aggregate itself. The first is reading a list; the second is real analysis. So the judge's system prompt is a domain profile: a compact encoding of the customer's tool catalog — which lookups come pre-joined, which return raw data, which questions require stitching two domains together. Tier = the work left over after the tools. When we benchmarked wording-only approaches on this workload, a surface heuristic scored 39% and keyword rules collapsed from 93% (on the set they were tuned on) to 62% on real queries. The domain-profile judge scored 21/21 on the customer's labeled question sheet and 8/8 on hold-out questions it had never seen.

2. Deterministic by construction. Judges default to sampling — which means a borderline question can route differently between runs. We measured exactly that before fixing it: the same cross-domain question classified MEDIUM four runs out of five and COMPLEX the fifth. The judge now runs at temperature 0 with a bounded output: same question, same route, every time (we re-ran the borderline set 5× each — zero flips).

3. Classify once per turn, not per hop. The assistant is an agentic tool loop — one user question can mean five model calls as tools return data. The judge's verdict is cached for the turn, so a 5-hop question pays for classification once. On real traffic this cut classification overhead ~5× versus judging every hop.

4. Fail toward cheap, never silently expensive. If the judge times out, a static fallback classifies; if everything fails, the request runs on the fast model. Routing can never block a request or quietly upgrade one.

5. Everything gets a receipt. Every judge call lands in the usage ledger as its own labeled line item — its cost, its verdict, and a pointer to the request that triggered it. Every answer records both the model requested and the model served. The entire pilot below is reconstructed from those receipts, not from our notes.

One question, fully instrumented

Here is a single real turn from the pilot — a genuinely hard question ("who has the most attendance coachings and the worst quality score?") asked through the customer's production assistant, with routing on:

Step	What happened	Tokens in / out	Cost	Time
Routing check	judge reads the question → verdict: COMPLEX	944 / 10	$0.000994	1.7 s
Hop 1	question + tool definitions → strong model; primes the prompt cache	21,555 / 445	$0.058278	5.8 s
Hop 2	first tool results (21,435 tokens read from cache)	21,929 / 163	$0.006905	1.7 s
Hop 3	tool loop continues (21,435 cached)	22,108 / 147	$0.007103	1.7 s
Hop 4	tool loop continues (21,435 cached)	22,271 / 139	$0.007349	2.0 s
Hop 5	final answer (21,435 cached)	22,367 / 78	$0.006931	1.7 s
Turn	1 check + 5 hops	111,174 / 982	$0.0876	~11.6 s

Four things this table proves at once: the app asked for its fast tier and the gateway served the strong model (that requested-vs-served pair is the receipt); classification cost 1.1% of the turn and ran once, not five times; provider prompt caching survived routing (hops 2–5 read 21k tokens at ~1/10 price); and every line is independently auditable in the ledger.

The test: the customer's real questions, both ways

We took the customer's own 21-question ground-truth sheet — each question pre-labeled with its intended tier by analyzing what their tools would actually have to do — and ran it through the live assistant twice: once pinned (the old setup), once routed.

Routing accuracy: 15/15. Of the 21, fifteen were answerable by the deployed tool set; the judge routed every one correctly — 12 stayed on the fast model, 3 upgraded — including the trap question the sheet's authors predicted wording-based routers would get wrong (a "both understaffed AND underperforming" question that sounds like cross-domain analysis but is served by a single pre-joined tool, so it's mid-tier at most; the judge kept it on the fast model).

The other six questions? The assistant had no tool for that data at all — and declined cleanly every time, in fresh threads and long ones, never inventing a number. (Routing kept those on the cheapest path, where a decline should live. The gap itself became roadmap input for the customer.)

What the upgrades bought — checked against the raw data. On the three hard questions, we compared the fast model's answers and the strong model's answers against the raw tool tables:

Silent miscounts, caught. On the "worst 3 quality-flagged drivers, and why" question, the fast model reported 3 blurry-photo events where the raw rows sum to 4, and 5 missed-package events where the rows sum to 8. The strong model got every count right — and where the fast model gave one generic recommendation, it diagnosed a different failure pattern per driver (navigation accuracy vs. scan workflow vs. a broad mix) with a targeted fix for each.
Over-claiming, caught. On an employee 360, the fast model asserted "all 13 coachings still open" — a claim the underlying system can't actually support (it doesn't track resolution). The strong model flagged exactly that, plus a known undercount in the shift denominator. Same numbers, honest error bars.
And one round to the fast model — see the failure section below, because it's the most useful finding in the pilot.

The failure we found (and why it made the product better)

On one cross-domain question, the strong model failed to answer at all: it worked more cautiously than the fast model — five narrow tool calls instead of two broad ones — and ran into the assistant's 5-step tool budget before it could synthesize. $0.127 spent, no answer, while the fast model had answered the same question correctly in 3 hops for $0.027.

This wasn't a routing mistake (the classification was right) or a model-quality problem. It was a client-harness interaction: step limits tuned for one model's tool-calling style can strangle another's. It's exactly the kind of thing you only find by instrumenting real turns — and exactly why we pilot before flipping production traffic. The fix was a one-setting change on the customer's side (a tier-aware step budget), verified by re-running the question. Found in testing, fixed in testing, never seen by an end user.

The economics, without spin

Measure	Value
One routing check	$0.00097
Per 1,000 questions, with per-turn caching	~$0.19–0.97
Check as share of a real analysis turn	1.1%
Check as share of the full 15-question sweep	3.4%
Added latency, once per question (not per hop)	~1 s
Pinned-cheap sweep vs routed sweep	$0.23 → $0.43
Routed vs pinning everything to the strong model	~35% cheaper

Read that middle row honestly: routing made this workload more expensive than pinning everything to the cheap model — because its job here was to fix quality, and the delta is almost entirely the three hard questions now served properly (the check itself is noise). Against the alternative way of getting that quality — the strong model everywhere — routing is ~35% cheaper, and the policy is a dial: a cost-leaning customer can upgrade only the hardest tier and pull the ratio back toward 1×.

For perspective on the latency line: during the same pilot we recorded an 11-second provider-side stall on a plain, unrouted call. Raw model variance dwarfs a 1-second check that runs once per question.

What this adds up to

Classification accuracy is achievable and provable: 21/21 ground truth, 8/8 hold-out, 15/15 live, deterministic on repeat — but only when the router knows the application, not just the words.
The overhead is a fixed penny-and-a-second per question — largest in relative terms exactly where the absolute stakes are smallest.
The failures worth worrying about are quiet ones — miscounts, over-claims, harness interactions — and the only way to catch them is per-request receipts you can check against raw data.
Every claim in this write-up traces to a ledger row.