LLM cost engineering

How we cut Claude inference costs 38% in production

Routing requests between Claude models did most of the work. Together with response caching and prompt compression, it cut Claude inference cost by 38% on a production platform, and the evaluation pipelines showed that quality held. Two engineers ran that platform, and that constraint shaped the trade-offs as much as the price list did.

The situation

The client was an EU FinTech start-up running a second-hand marketplace, and I built out its Claude platform between 2024 and 2026. A prototype on the Claude API already worked. It could not go to production, for five reasons:

Behind all five sat one constraint: inference cost was growing faster than revenue, on a platform run by two engineers. A fix that cut cost but added a system those two people could not keep running was not a fix.

This post is about the cost blocker, but the others shaped it. You cannot manage the cost of a tenant you cannot see, and you cannot tune routing safely without a way to release routing changes.

Make spend visible per tenant

A provider invoice tells you what the organization spent. It does not tell you which workspace, feature or prompt spent it. So we built per-tenant token-spend dashboards, and cost became visible per workspace. Each tenant also had its own rate budget, as part of the isolation layer that used OIDC and namespace RBAC.

The mechanism is simple and worth copying: attach the tenant to every model call, record the token counts the API returns, and price them per model. The open-source distillation of the routing logic, llm-router, shows the shape. It is not the client's code. Every call returns its own cost record:

@dataclass
class RouteResponse:
    """Result of a single Router.route() call."""
    content: str
    model: str
    input_tokens: int
    output_tokens: int
    cost_usd: float
    cached: bool
    latency_ms: int

route() also takes a user_id, which the CLI describes as a "User/tenant identifier (for budget tracking)". Once every record carries a tenant, a dashboard is an aggregation. How to carry that context through traces and logs is covered in agent observability.

Three levers, and why routing gave the most

We used three levers: prompt compression, response caching keyed on prompt hashes, and routing between models. Routing gave the most.

Routing between models

The split was by the kind of work: Haiku for classification, Sonnet for the middle, and Opus only where output quality justified the price. Around it sat a routing layer with SLA-aware retries, timeouts, latency budgets and a model fallback chain from Haiku to Sonnet to Opus, chosen per prompt complexity.

There is a structural reason routing tends to win. Caching and compression remove tokens: caching pays only when a request repeats, and compression trims only the input side. Routing changes the price of every token in a request, input and output, for every request that does not need the top tier. At Anthropic's list prices on 2026-09-19, Claude Haiku 4.5 costs one fifth of Claude Opus 5 per token, in both directions (pricing).

In llm-router, the classifier makes the choice with deterministic checks:

if token_estimate <= self.simple_threshold and not has_multistep and not has_code:
    return ClassificationResult(
        complexity="simple",
        recommended_model=self.fallback_chain[0],  # Haiku
        reason=f"short ({token_estimate} tokens), no multi-step markers, no code",
    )

if token_estimate >= self.complex_threshold or has_multistep:
    return ClassificationResult(
        complexity="complex",
        recommended_model=self.fallback_chain[2],  # Opus
        reason=f"long ({token_estimate} tokens) or multi-step reasoning detected",
    )

Everything else falls through to Sonnet. The full decision table, the classifier signals and the escalation rules are in Routing between Haiku, Sonnet and Opus.

Response caching keyed on prompt hashes

An identical request should not be paid for twice. The response cache was keyed on a hash of the prompt. In llm-router the key covers the prompt, the system prompt and the model:

def make_cache_key(prompt: str, system: Optional[str], model: str) -> str:
    """Deterministic hash for (prompt, system, model) tuple."""
    payload = json.dumps(
        {"prompt": prompt, "system": system or "", "model": model},
        sort_keys=True,
    )
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()

Two details matter in a multi-tenant system. The system prompt belongs in the key, so a changed template never serves an answer written under the old one. And when a response depends on tenant data, the tenant belongs in the key as well, or one workspace can be served another workspace's answer. A TTL bounds how stale an entry can get; the repository defaults to 3,600 seconds.

This is a different mechanism from the provider's prompt caching, which discounts a repeated prompt prefix instead of skipping the call. The cost engineering overview covers both.

Prompt compression

Compression cuts input tokens. The open-source version orders its steps by risk: whitespace normalization, which is always safe; removing duplicated in-context examples; and, only for long system prompts and only if you opt in, summarization by Haiku. The code comment on that last step is direct: "Be careful — this is the level that can degrade quality."

Whatever compression you use, it is a prompt change, and it goes through the same release gate as any other prompt change.

Holding quality

A saving counts only if the answers stay as good. Quality was held with evaluation pipelines and gated releases: prompts, agents and routing logic were all released through GitOps CI/CD with gates, the same way as code.

Treating a routing rule as code matters. Moving a class of requests from Sonnet to Haiku changes the behavior every user in that class sees, even though no application code changed. How to decide whether a cheaper model is good enough before it takes traffic is its own subject, covered in Is the cheaper model good enough.

The same platform also ran PII masking, moderation filters and a conversation-level audit trail. After rollout, there were zero PII incidents in production logs. The audit side is covered in An audit trail for every LLM call.

The result

Claude inference cost fell by 38%, with quality held, and the evaluation pipelines are what verified it. The platform carried multi-tenant production traffic at SLA, with an isolated budget for each tenant.

The trade-off I chose

Fine-tuning a smaller open model would have cut cost further. We did not do it.

A fine-tuned model is not a one-off job. It needs a pipeline for training data, an evaluation run for every retrain, a retraining schedule that keeps up with the product, and serving infrastructure of its own. With two engineers, that evaluation and retraining pipeline was more operational surface than the team could carry. I traded some of the saving for a system the team could actually run.

What I would do differently