LLM cost engineering

Routing between Haiku, Sonnet and Opus: a decision table that works

Route by task where you know it, by request shape where you do not, and escalate one tier at a time when a check fails. That is the whole policy. The table below is how llm-router implements the shape-based part. It is the open-source starter I distilled from a production routing layer, MIT-licensed. The rest of this post covers what the code does, where it goes wrong, and what to measure.

It is a starter, not a finished library. The classifier, the response cache and whitespace compression are implemented, with unit tests. Router.route() lays out the call path as numbered TODO steps for you to wire to your own client. Install it from the repository, not by package name:

git clone https://github.com/Narayanaraj/llm-router-starter
cd llm-router-starter
pip install -e ".[dev]"
pytest

The decision table

The classifier works on an estimate of the size of the system prompt and user prompt together, at 4 characters per token, with two thresholds: 200 tokens and 2,000 tokens.

The checks run in that order: Haiku first, then Opus, and Sonnet catches the rest. A 10-token prompt that says "prove" goes to Opus. A 150-token prompt with a code block goes to Sonnet, unless it also asks for step-by-step reasoning.

The production layer this came from split by task rather than by shape: Haiku for classification, Sonnet for the middle, Opus only where output quality justified it. A library cannot see your tasks, so it approximates them with shape.

What the classifier looks at

Three signals, all deterministic, so classifying a request costs no model call. The reasoning markers are regular expressions:

MULTISTEP_MARKERS = [
    r"\bstep[- ]?by[- ]?step\b",
    r"\bfirst.{0,40}then\b",
    r"\bexplain (your )?reasoning\b",
    r"\bprove\b",
    r"\bderive\b",
    r"\banalyze (in )?(detail|depth)\b",
    r"\bcompare and contrast\b",
]

All three are computed over the system prompt and the user prompt together:

full_text = f"{system or ''}\n{prompt}".strip()
token_estimate = self._estimate_tokens(full_text)

has_multistep = bool(self._multistep_re.search(full_text))
has_code = bool(CODE_BLOCK_PATTERN.search(full_text))

The size signal is the crude one: len(text) // 4. Its docstring says to replace it for production. For Claude, the token counting endpoint returns the real count, and the thresholds are only as good as the count behind them.

Where the heuristic goes wrong

The repository ships 10 sample prompts in benchmarks/prompts.json, each labeled with an expected complexity. Run the classifier over them:

import json
from llm_router.classifier import Classifier

c = Classifier()
for p in json.load(open("benchmarks/prompts.json")):
    r = c.classify(p["prompt"], p["system"])
    print(p["id"], p["expected_complexity"], "->", r.complexity)

5 of the 10 match their label. The misses are the useful part:

• long-system-001, labeled simple. The user says "Hello, can you help me?". The system prompt includes "You break down complex topics step by step." Because the classifier reads the system prompt too, the marker fires and a greeting goes to Opus. Any feature whose system prompt mentions step-by-step reasoning sends every request to the most expensive tier.
• summary-001 and summary-002, labeled moderate. A one-sentence and a three-bullet summary, estimated at 68 and 93 tokens, go to Haiku.
• code-001, labeled moderate. "Write a Python function that returns the nth Fibonacci number" has no code in it, is 25 tokens long, and goes to Haiku.
• code-002, labeled simple. A one-line function in an inline code span is longer than 20 characters, so it counts as code and goes to Sonnet.

A disagreement is not automatically a failure. Whether Haiku writes a good one-sentence summary is a question for evaluation, not for labeling, and Is the cheaper model good enough covers how to answer it before the route takes traffic. The system-prompt case is different: it is a flaw in the approach, and the fix is to classify the user turn only.

Route by task first

The strongest routing signal is not in the text. A call site that sorts a message into one of 6 intents, for example, knows what it is doing before any text arrives, and can name its tier in configuration. That is what "Haiku for classification" means in practice. Keep the text heuristics for free-form requests, where the call site cannot know in advance how hard the request is.

Escalation and fallback

In the repository, fallback_chain is an ordered list of model IDs, and the classifier picks one by index. The README's configuration example shows it:

[router]
default_model = "claude-haiku-4-5-20251001"
fallback_chain = ["claude-haiku-4-5-20251001", "claude-sonnet-4-6", "claude-opus-4-6"]
classifier_threshold_simple = 200      # tokens
classifier_threshold_complex = 2000    # tokens

There is no retry or escalation code in the repository yet. The production layer had SLA-aware retries, timeouts, latency budgets and model fallback from Haiku to Sonnet to Opus, chosen per prompt complexity. These are the rules that make escalation safe:

1. Escalate on a check that failed. A response that fails schema validation, a parser, or a test the calling code runs is evidence. A vague sense that the answer could be better is not.
2. One tier at a time, with a cap. Haiku to Sonnet, Sonnet to Opus, then stop and return an error the caller can handle.
3. Keep errors apart from quality. A timeout or a rate-limit error is a reason to retry with backoff, or to fall back for availability, within the latency budget. It is not evidence that the request needed a larger model, so do not count it as a quality escalation.
4. Spend from a latency budget. Every attempt uses part of the request's budget. When it is gone, fail cleanly.
5. Log every escalation with its route, tier and reason.

A sketch of rules 1, 2, 4 and 5. It is neither code from the repository nor the production code, and call() is assumed to handle rule 3 with its own retries:

TIERS = ["haiku", "sonnet", "opus"]  # mapped to model IDs in configuration

def answer(request, start_tier, budget_ms):
    deadline = now_ms() + budget_ms
    for tier in TIERS[TIERS.index(start_tier):]:
        remaining = deadline - now_ms()
        if remaining <= 0:
            break
        response = call(tier, request, timeout_ms=remaining)
        if passes_checks(request, response):
            return response
        log_escalation(request.route, tier, response)
    raise EscalationExhausted(request.route)

Escalation is not free. An escalated request pays for the cheap attempt and the expensive one. Here is an example with made-up traffic: say a route starts on Haiku and a share r of its requests escalate to Sonnet. With the same token counts on both tiers, the route costs Haiku + r × Sonnet per request, against Sonnet alone for going straight there. The break-even is r = 1 − (Haiku price ÷ Sonnet price). With Claude Haiku 4.5 at half the price of Claude Sonnet 5, that is 50%. Against Claude Sonnet 4.6, at three times Haiku's price, it is about 67%. Above the break-even, route that class of requests to the larger tier from the start. Anthropic's cost guidance describes the same pattern with effort levels: run at a low setting, and re-run only the failures at a higher one.

The tiers as of 2026-09-19

Sources: Anthropic's models overview and pricing pages, checked 2026-09-19. The repository's defaults name claude-sonnet-4-6 and claude-opus-4-6, which Anthropic lists as legacy models that are still available, at 3 / 15 and 5 / 25. There is also a tier above Opus, Claude Fable 5.1 at 10 / 50, and the table extends to a fourth row the same way.

Three things move the arithmetic away from list price:

• Output costs 5 times input on every tier. On a route that writes long answers, output price dominates.
• Token counts differ between models. Anthropic's pricing page notes that Claude 4.7 and later models use a newer tokenizer that produces about 30% more tokens for the same text. Compare tiers on cost per request measured from the API's usage fields, not on price per token.
• Prompt caching has a minimum length per model: 4,096 tokens on Claude Haiku 4.5, 1,024 on Claude Sonnet 5 and 512 on Claude Opus 5, according to the prompt caching documentation. A 2,000-token system prompt can be cached on Opus 5 but not on Haiku 4.5, which narrows the gap on routes with long, stable prompts.

What to measure

Per route, and reviewed after every routing change:

• The share of requests per tier, so you can see drift when traffic changes.
• The escalation rate per tier, with reasons. Compare it with the break-even above.
• Cost per request and per completed task, from the input, output, cache-write and cache-read token counts the API returns, attributed to tenant and feature.
• p95 latency per tier, with escalated requests counted end to end.
• Quality on a labeled sample, compared with the previous routing, as a gate before the change takes traffic.
• The response-cache hit rate. The repository's cache key includes the model, so retuning a threshold re-routes prompts, and their cached entries stop matching. Expect a dip after every routing change.

The repository's benchmark defines the comparison to run: an Opus-only baseline without cache or compression, the router alone, and the router with cache and compression. Quality is the cosine similarity between each response and the baseline's response, and the acceptance bar is at least 30% lower cost than the baseline at a similarity of 0.95 or more. run_benchmark.py lays that harness out as numbered steps for you to implement against your own corpus. The 10 sample prompts are a starting point, not a test set.

Similarity to the Opus answer measures sameness, not correctness. A cheaper model can phrase a correct answer differently and score lower, or be wrong in similar words and score higher. For release decisions, grade against checks specific to the task.

What I would do differently

• Classify the user turn, not the system prompt. The long-system-001 miss shows why: one line of standing instructions can move a whole feature to the top tier.
• Make routing by feature the first rule. Each call site would declare its tier in configuration, and the text classifier would handle only free-form requests.
• Count tokens properly. The thresholds deserve a real token count, not a character estimate.
• Sweep effort on one model before adding tiers. Anthropic's current guidance is to sweep effort on the model you already use, and then to price the stronger model alone at low effort, before building a multi-model setup. Haiku 4.5 does not support the effort setting, but Sonnet 5 and Opus 5 do.
• Keep model IDs and prices in dated configuration, not in code. Both change, and a price table in code goes stale without anyone noticing.