LLM cost engineering: cutting inference spend in production

Where LLM inference money actually goes, the levers that reduce it, and how to prove the saving.

The problem

LLM spend grows faster than traffic because the cost of a request is not fixed. It is the number of tokens in, plus the number of tokens out, each times the price of the model that handled them. All three drift upward unless someone pushes back.

Prompts accumulate. Every feature adds instructions, examples, retrieved documents or tool definitions. None of them is large on its own. Together they can double the input of every request, and nobody reviews the total.
Conversations resend their history. In a chat or an agent loop, each turn sends the whole conversation again. The first message of a 20-turn conversation is paid for 20 times, so the cost of a conversation grows much faster than its length.
The default model is the strongest one, because that is the model the prototype was built on.
Output is verbose by default, and output is the expensive side. On every current Claude tier, an output token costs 5 times an input token, according to Anthropic's pricing as of 2026-09-19.
Retries and fallbacks add calls that appear in no feature's design.
Nobody owns the total. The invoice arrives once a month, per organization. It does not say which tenant, feature or prompt spent it.

On a production Claude platform I built out for an EU FinTech start-up running a second-hand marketplace, unpredictable cost was one of the reasons the prototype could not go to production. Inference cost was growing faster than revenue, on a platform run by two engineers. Routing between models, response caching and prompt compression cut Claude inference cost there by 38%, with quality held, and per-tenant dashboards made the spend visible. The full case is in How we cut Claude inference costs 38% in production.

Cost engineering means turning that drift into numbers someone owns: cost per request, per tenant and per feature, each with a quality number next to it.

The levers that actually work

Five levers apply to almost any LLM product. None of them needs a new model or a training pipeline.

Routing between models

Model tiers differ in price by a multiple. On 2026-09-19, Claude Haiku 4.5 costs one fifth of Claude Opus 5 per token, input and output alike. Much production traffic is not hard: classification, extraction, short answers. Sending all of it to the top tier pays the top price for work a smaller model can do.

A router needs three parts. First, a rule per task where the call site knows the task, for example a classification endpoint pinned to the smallest tier in configuration. Second, a classifier for free-form requests, using signals such as length, reasoning markers and code. Third, escalation to the next tier when a check on the answer fails. On the marketplace platform, routing gave the most of the three levers: Haiku for classification, Sonnet for the middle, and Opus only where output quality justified it.

Two cautions. An escalated request pays for both attempts, so a route that escalates often should start on the larger tier. And the price gap between tiers changes with each model generation, so the arithmetic needs checking again when the lineup moves. Anthropic's current cost guidance is to sweep the effort setting on the model you already use before adding a second one. The decision table, the classifier code and the escalation rules are in Routing between Haiku, Sonnet and Opus.

Caching

Caching means two different mechanisms, and they pay on different traffic.

Response caching happens on your side. An identical request gets the stored answer, and no model call is made. The key is a hash of everything that shapes the answer: the prompt, the system prompt or its template version, the model, and the tenant wherever the answer depends on tenant data. A TTL bounds how stale an answer can get. It pays on repeated questions and repeated inputs, and does nothing for unique conversations. The marketplace platform keyed its response cache on prompt hashes.

Provider prompt caching stores the processed prefix of a prompt, so repeated system prompts, tool definitions, documents and conversation history are billed at a discount. On Anthropic's API, as of 2026-09-19, a cache read costs 0.1 times the base input price on most models, and a cache write costs 1.25 times it for the 5-minute lifetime or 2 times for 1 hour (prompt caching). It needs the stable content first and the varying content last, because a change anywhere in the prefix invalidates everything after it. Each model also has a minimum cacheable length, which rules short prompts out.

Either kind needs an invalidation story before it goes live: which event makes an entry wrong, and how the key or the TTL reflects that event.

Prompt compression

Compression cuts input tokens without changing what the model is asked to do. Apply it in order of risk:

Mechanical cleanup. Whitespace, duplicated examples, and instructions for features that no longer exist. This is safe.
Context trimming. Retrieve fewer and better chunks, send the fields a task needs instead of the whole record, and summarize old conversation turns instead of resending them. This is usually safe.
Rewriting instructions with a model. This can change behavior, and it needs evaluating like any other prompt change.

Every compression is a prompt change and goes through the same release gate. It also only touches the input; where output dominates the bill, the fifth lever matters more.

Budgets per tenant and per feature

A budget turns a cost number into a control. Attribute every call to a tenant and a feature at the moment it is made. Set a budget for each, and decide in advance what happens at the limit: an alert at a soft threshold, and at the hard limit one of queuing, moving the tenant to a cheaper tier, or rejecting with an error the product can explain. That choice is a product decision, not an infrastructure one.

Budgets are also isolation. Without them, one tenant's runaway usage spends the capacity and the money of everyone else. On the marketplace platform, each tenant had its own rate budget, and per-tenant token-spend dashboards made cost visible per workspace. It carried multi-tenant production traffic at SLA with isolated budgets per tenant.

Output length

Output tokens are the expensive ones, so ask for the output you will use: a label instead of a paragraph, JSON fields instead of prose, one line instead of a memo. Anthropic's cost guide reports a triage job in which a one-line answer used 39% fewer output tokens than a two-line answer and cost 14% less per run, with accuracy within run-to-run noise.

Two cautions. A low max_tokens cap is not a cost control: a truncated answer gets retried, and you pay for both attempts. And on Claude Opus 5, lowering the effort setting does not reliably shorten the visible answer, so Anthropic's effort documentation advises prompting for length directly.

How to measure it

Measure cost per unit of work, not the invoice.

Record every call. Log the tenant, the feature or route, the model, and the four token counts the API returns: input, cache write, cache read and output. Price each at its own rate. The OpenTelemetry side of carrying that context through traces is covered under agent observability.

Aggregate four ways: cost per request, per tenant, per feature, and per completed task. The last one matters most. A cheap request that fails and gets retried or escalated is not cheap, and Anthropic's cost guide recommends comparing models on cost per completed task for the same reason.

Take a baseline before changing anything. Record a period of traffic with its mix of features and tenants, so that a later drop can be traced to a lever and not to a change in traffic.

Compare models on measured cost, not list price. Tokenizers differ between models. Anthropic's pricing page notes that Claude 4.7 and later models produce about 30% more tokens for the same text, so the same request costs a different number of tokens on different tiers.

Put quality next to every cost number. Build an evaluation set from real traffic, score it per class of request, and run it as a gate on every prompt, routing and compression change. After release, score a sample of production traffic. On the marketplace platform, the 38% counted because evaluation pipelines verified that quality held, and prompts, agents and routing logic went through gated releases. How to decide whether a cheaper or routed model is good enough before it takes traffic is covered in Is the cheaper model good enough.

Common mistakes

Cutting the model price while the prompt doubles. Moving to a tier at half the price saves nothing on input if the prompt doubles in the same quarter. Tokens per request is the number to watch, next to price.

Caching without an invalidation story. A key that leaves out the prompt version, the model or the tenant serves stale answers, or another tenant's answers. On the provider side, a timestamp near the top of a system prompt quietly turns prompt caching off.

Comparing price per token instead of cost per task. Tokenizers, retries, escalations and answer length all sit between the price list and the bill.

Shipping routing or compression changes without an evaluation gate. A routing rule that moves a class of requests to a smaller model changes behavior for every user in that class. It deserves the same release process as code.

Paying interactive prices for work nobody waits on. Anthropic's Batch API takes 50% off every token in exchange for results within 24 hours, according to Anthropic's pricing and cost guidance as of 2026-09-19. Nightly jobs, backfills and evaluation runs belong there.

One budget for the whole organization. The first overrun is found on the invoice, and one tenant can spend everyone's money before anyone sees it.

Reporting a saving with no quality number. A cost cut without an evaluation behind it cannot be checked, by you or by anyone you report it to.