Agent observability & evaluation

Is the cheaper model good enough? Evaluating LLM routing before it takes traffic

A cheaper model is good enough when a gate says it is, before it takes traffic, and per class of request rather than on average. The gate runs the candidate configuration and the one in production on the same fixed set of real requests and compares cost per request, a quality score and latency. It uses a judge you have checked against people, and thresholds you wrote down before the run. Once it passes, shadow or canary traffic confirms the result on requests the set did not contain.

Why a cost cut needs a quality number

At an EU FinTech start-up running a second-hand marketplace, I cut Claude inference cost by 38%. Routing between models gave the most: Haiku for classification, Sonnet for the middle, Opus only where output quality justified it. Two engineers ran the platform, and inference cost was growing faster than revenue.

The 38% only counts because quality held, and that was verified through evaluation pipelines. Without the second half, a cost reduction is a claim that you made the product worse for less money, and nobody can say how much worse. The whole story of that cut is in cutting Claude inference costs by 38%.

Two of the things that kept the prototype out of production were the lack of observability and the lack of a safe way to release prompt or routing changes. So prompts, agents and routing logic shipped through gated releases in a GitOps pipeline, like code. The sections below describe how to build that kind of gate. They are general practice, not a transcript of that client's pipeline.

Build the evaluation set from real traffic

Synthetic prompts test what you imagined users would ask. The routing decision depends on what they actually ask, so the set should come from production requests:

Refresh the set on a schedule as a new version, and keep the old one so this quarter's result can still be compared with last quarter's.

Score per class, not on the average

A mean quality score hides exactly the failure routing causes: one class of requests goes to a model that cannot handle it, and the other classes pay for the average.

An illustrative example: a 10-prompt set with 5 simple, 3 moderate and 2 complex prompts. Say the simple prompts score 0.99, the moderate ones 0.88, and the complex ones, still routed to the top model, 0.97. The mean is (4.95 + 2.64 + 1.94) / 10 = 0.953. That passes a 0.95 gate while the moderate class sits at 0.88.

So the gate runs per class and compares four things with the configuration in production:

Measure Why it is in the gate
Cost per request The saving you are claiming, computed from token counts and the price list.
Quality score The thing you must not trade away, scored against the same labels.
Latency, p50 and p95 A smaller model is often faster, but retries and fallbacks add time at the tail.
Escalation rate How often the cheap attempt fails and the request moves up the chain.

The last row matters with a fallback chain like Haiku → Sonnet → Opus. An escalated request pays for at least two calls, so a router that escalates often saves less than its routing table suggests, and adds latency on every escalation.

Pick the scorer by task

Not every class needs a judge model.

Check the judge before you trust it

An LLM judge is a measuring instrument with known biases. The MT-Bench study of LLM judges (Zheng et al., 2023) documents position, verbosity and self-enhancement bias, and found that strong judges agreed with human preferences over 80% of the time, about as often as humans agree with each other. That is a good research result. It also means up to one verdict in five may disagree with a person.

Checking the judge costs little compared with trusting it blindly:

  1. Have people label a sample of the evaluation set with the rubric the judge uses.
  2. Measure judge-to-human agreement per class. Use Cohen's kappa rather than raw agreement: on a set where 90% of answers are fine, a judge that always says "fine" scores 90%.
  3. When the judge compares two answers, run it in both orders and count a win only if it survives the swap.
  4. Hide which model wrote which answer. If the judge and a candidate come from the same model family, self-enhancement bias is a real risk.
  5. Repeat the check whenever the judge model or the judge prompt changes. A new judge is a new instrument.

Measure the noise floor as well: run the production configuration against itself twice. Model output varies between runs, and a threshold tighter than that variation fails good releases at random.

Set thresholds before the run

Write the thresholds into the gate before you see the numbers. Deciding afterwards turns the gate into a negotiation. A minimal version:

# Sketch of a release gate, not production code.
def gate(candidate: dict, baseline: dict, noise: dict) -> list[str]:
    failures = []
    for cls in baseline:  # e.g. "simple", "moderate", "complex"
        c, b = candidate[cls], baseline[cls]
        if c["quality"] < b["quality"] - noise[cls]:
            failures.append(f"{cls}: quality {c['quality']:.3f} vs {b['quality']:.3f}")
        if c["p95_ms"] > b["p95_ms"] * 1.2:
            failures.append(f"{cls}: p95 {c['p95_ms']} ms vs {b['p95_ms']} ms")
    if sum(v["cost"] for v in candidate.values()) >= sum(v["cost"] for v in baseline.values()):
        failures.append("no cost saving")
    return failures  # empty: the change may ship

The shape matters more than the numbers. Quality is checked per class and may drop by no more than the measured noise, latency has a budget, and the release has to save money in total to be worth the risk. The 1.2 is an example; pick your own tolerance.

The comparison in its simplest form

I later open-sourced the distilled routing logic as llm-router (MIT). Its benchmark design, in benchmarks/run_benchmark.py, sets out the cost/quality comparison in its simplest form:

  2. For each of three configurations:
        (a) baseline_opus_only       — every prompt to Opus, no cache, no compression
        (b) router_only              — Router with classifier, no cache, no compression
        (c) router_full              — Router with classifier + cache + compression
     Record per-prompt: model_used, input_tokens, output_tokens, cost_usd, latency_ms, response_text.
  3. Compute quality metric: cosine similarity between (a)'s response and (b)/(c)'s response,
     using sentence-transformers all-MiniLM-L6-v2 or anthropic embeddings if available.
  4. Aggregate: total_cost, avg_latency, mean_quality_vs_baseline.
  5. Emit summary table to stdout. Save detailed JSON to benchmarks/results/{timestamp}.json.

Acceptance: configuration (c) should achieve ≥30% cost reduction vs (a) with quality ≥0.95.

Three parts of it are worth copying. Separating (b) from (c) isolates what routing alone buys from what caching and compression add. The design records tokens, cost and latency next to each response, so cost and quality come from the same run. And the acceptance criterion is two numbers, written down in advance.

Before using that design for a release decision, I would add three things. The corpus already tags each prompt with an expected_complexity, so aggregating per class is a small step. The quality metric is similarity to the Opus answer: cheap and judge-free, but it rewards sounding like Opus rather than being right, so a correct answer worded differently scores low. And there is no noise floor: two Opus runs of the same prompt do not always produce the same text.

The repository's tests check the routing decision itself: a short question goes to Haiku, a prompt containing "step by step" goes to Opus, and a prompt between the two length thresholds with no multi-step markers goes to Sonnet. CI runs them on every push and pull request to main, so a threshold change that moves one of those prompts to another model fails a check instead of surprising someone in production.

Shadow first, then canary

An evaluation set is a snapshot. Two steps close the gap to live traffic.

Shadow traffic. Copy a sample of live requests to the candidate configuration, discard its answers, and score them offline against what production served. Users see nothing, and you find the request types the set missed. It costs extra inference for the sampled share, and the copies need the same PII masking and retention rules as the primary path.

Canary. Send a small share of real traffic, or a few tenants, to the candidate. Watch cost per request, latency, escalation rate and a judged sample of answers against the rest of traffic, with the rollback condition written down before the canary starts. When routing logic is configuration shipped through GitOps, rollback is a revert.

After full rollout, the same measures stay on the dashboard, because traffic keeps changing after the gate has passed. The agent observability hub covers the traces and logs those signals come from, and an audit trail for every LLM call covers the record of what each call did. The routing code is in routing between Haiku, Sonnet and Opus, and the cost side of all this is under LLM cost engineering.

What I would do differently