Token Costs, Caching and Model Routing: How Design Decides Your AI Unit Economics

Per-token prices keep falling, and enterprise AI bills keep climbing anyway. Gartner forecasts worldwide AI spending will reach $2.59 trillion in 2026, up 47 percent year over year (Gartner, 19 May 2026). Your LLM project is expensive because unit price is the smallest lever you have. Token volume, output length, repeated context, and model choice decide the bill, and all four are architecture decisions you control. You reduce AI running costs by caching stable context, routing each task to the cheapest model that can do it, and capping output, in that order.

Why does the bill keep rising when token prices keep falling?

A single model call looks cheap. A production workload is rarely a single call. An agentic task, where the model plans, calls a tool, reads the result, and revises, can fan out into ten or more model calls before it returns one answer to the user. Retrieval pipelines add their own weight: every query ships the same policy documents, product catalogue, or patient-intake schema into the context window, and you pay for those tokens on each request.

So the cost driver is compounding volume rather than the sticker price. The prototype sends 2,000 tokens and answers in 300. The production version sends 40,000 tokens of retrieved context, runs six internal steps, and streams 1,500 tokens back. Same model, same per-token rate, roughly fifty times the cost, multiplied by real traffic.

This is why Gartner expects more than 40 percent of agentic AI projects to be cancelled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls (Gartner, 25 June 2025). Cost overrun kills these projects more often than any modelling failure.

Where does the money actually go?

Start with the one asymmetry every finance-adjacent leader should internalise: output tokens cost several times more than input tokens. On Anthropic's published rates, Claude Opus 4.8 runs $5 per million input tokens against $25 per million output tokens, a five to one ratio; Claude Sonnet 4.6 is $3 and $15; Claude Haiku 4.5 is $1 and $5 (Anthropic pricing, 2026). A verbose model that writes three paragraphs where one sentence would do is the most expensive line item you have.

The four cost drivers, in the order they usually bite:

1. Output length. The most expensive tokens, and the easiest to cap with a hard limit and a prompt that asks for the answer directly.
2. Repeated input context. The same system prompt, few-shot examples, and retrieved documents resent on every request.
3. Model tier. Running your most capable model on tasks a mid-tier model would clear.
4. Step count. Multi-step agent loops that re-read their own history at every turn.

Notice that only one of the four is the price on the tariff sheet. The other three are decisions in your code.

How does prompt caching change the math?

Prompt caching is the highest-return change available to most teams, and it costs nothing beyond getting the request structure right. When a large, stable prefix repeats across requests, the provider can serve it from cache instead of reprocessing it. On Anthropic's implementation, cache reads cost roughly 0.1x the base input rate, about a 90 percent saving on the cached portion; cache writes cost 1.25x the base rate for a five-minute window and 2x for a one-hour window (Anthropic prompt caching docs, 2026).

The mechanism is a prefix match. The cache key is the exact bytes of the prompt up to a marked breakpoint, so a single changed byte anywhere in that prefix invalidates everything after it. This is where teams lose the saving without realising it. A timestamp in the system prompt, an unsorted JSON blob, a per-request ID injected near the top: each one silently defeats the cache, and the response usage quietly reports zero cache reads.

The design rule follows directly. Put the stable content first, in a deterministic order: frozen system prompt, sorted tool definitions, static retrieved context. Put the volatile content, the user's actual question, last. Verify it worked by reading the cache-read token count on the response rather than assuming. For a workload that resends 30,000 tokens of policy context on every query, caching that prefix turns the dominant cost line into a rounding error.

For work that tolerates latency, batch processing stacks on top: Anthropic's Batch API discounts token usage by 50 percent (Anthropic pricing, 2026). Overnight report generation, bulk classification, and document extraction rarely need a synchronous answer.

How should you route between models?

Model routing is the practice of sending each task to the cheapest model that meets the quality bar, rather than defaulting your whole pipeline to the most capable one. The pricing ratios make the case: Haiku 4.5 is one fifth the input price of Opus 4.8. If classification, extraction, routing decisions, and short summaries run on the small model while only genuinely hard reasoning reaches the large one, the blended cost drops sharply without a visible quality change.

The trap is measuring quality on the wrong axis. A cheaper model that produces a slightly worse answer on a task where the answer feeds a downstream step can inject an error that the expensive model then spends tokens trying to recover from. Route by measured task difficulty rather than impression. Build a small evaluation set for each task type, run both tiers against it, and promote the cheap model only where it holds the bar.

Two engineering cautions worth stating plainly. First, switching models mid-conversation invalidates the prompt cache, because caches are scoped per model. If you need a cheaper model for a sub-task, spawn a separate call for it and keep the main loop on one model, rather than swapping the model inside a cached conversation. Second, in regulated work the cheapest route is not always a legal route. Under the Digital Personal Data Protection Act 2023, where and how patient or customer data is processed and retained is a compliance question, so data residency and retention constraints can remove a model or region from the routing table before cost enters the decision.

What does a cost-aware architecture look like in production?

At Nextdot we run voice-first customer-experience agents in live hospital environments, and clinical traffic is unforgiving on both latency and cost. The economics only work when the design accounts for them from the first commit. A cost-aware build looks like this in practice:

• A frozen, cacheable prefix carrying the system prompt and stable context, with volatile input appended last.
• Output capped at the length the task needs, with prompts written to return the answer without preamble.
• A routing layer that sends routine turns to a small model and reserves the large model for hard reasoning, each tier validated against an evaluation set.
• Token usage logged per request, so a regression in cache hit rate or a creeping output length shows up on a dashboard rather than on the invoice.

That last point is the one teams skip. Unit economics is a measurement discipline before it is an optimisation problem. Log input tokens, cached tokens, output tokens, and step count per request, attribute them to a task type, and the expensive line items name themselves. You cannot cut a cost you are not measuring, and the demo that impressed everyone was almost certainly never measured this way.

The compounding cuts both ways. The same fan-out that multiplies a careless design multiplies a disciplined one. A workload that caches its context, caps its output, and routes its traffic can run at a fraction of the naive cost at the same quality, which is the difference between an AI project that reaches production and one that becomes part of the 40 percent Gartner is warning about.

Frequently asked questions