LLM Inference Costs: The Race to the Bottom and What It Actually Means

Historical text. Preserved below for context. Its dollar comparison, forecast and broad claims about caching, hardware, quantisation and self-hosting are superseded by the dated discussion above; do not use them as current rates or verified measurements.

Two years ago, processing a million tokens through GPT-4 cost roughly $60. Today you can do the same work for under $2 with models that perform comparably or better on most practical tasks. That is not incremental improvement. That is a structural shift in the economics of AI systems.

This piece looks at what is actually driving the cost curve, which assumptions it breaks for teams currently designing or rebuilding AI applications, and where the floor might be.

What Is Driving the Collapse

Several forces are compressing inference costs simultaneously.

Hardware efficiency. Each generation of GPU silicon does more useful work per watt. The H100 is not just faster than an A100 — it is dramatically more efficient at the specific matrix operations that dominate transformer inference. The H200 and B100 will continue this. Custom silicon (Google TPUs, AWS Trainium, Groq's LPU) adds pressure from different architecture directions.

Quantisation. Running models in 4-bit or 8-bit precision was once considered a significant quality trade-off. For most production workloads, it is not. INT4 quantisation of a well-trained 70B model often outperforms a 13B model in full precision at a fraction of the compute cost. The maths is straightforward: 4x fewer bits means roughly 4x more tokens per GPU memory slot.

Speculative decoding and batching improvements. Providers have become considerably better at keeping GPUs saturated. Speculative decoding — using a smaller draft model to propose tokens that a larger model verifies — reduces latency-per-token without proportional compute increases. Continuous batching allows providers to handle variable workloads without the memory waste of static batch allocation.

Competition. OpenAI's pricing monopoly has ended. Anthropic, Google, Mistral, Groq, and a range of inference-as-a-service providers are all competing on price. Open weights models have given the market a credible opt-out.

What This Breaks

Several architectural decisions made when inference was expensive no longer make sense.

Aggressive caching. Many teams built complex caching layers to avoid re-running prompts. Some of this remains valid for latency reasons. As a cost-saving measure it is increasingly hard to justify the engineering overhead.

Context compression. Techniques for aggressively summarising context before sending it to a model made sense when long contexts were expensive. At current and projected prices, the engineering cost of maintaining a compression pipeline often exceeds the token savings. Evaluate this honestly on your specific workload before building it.

Routing to smaller models. The strategy of routing "simple" queries to cheap small models and "complex" ones to expensive large models is getting harder to execute well. The classification problem is non-trivial, the quality boundary is blurry, and the price difference is shrinking. What made sense at a 30x price ratio may not make sense at 5x.

On-premises inference for cost. Running self-hosted inference made economic sense when hosted prices were high. The primary remaining arguments for self-hosting are privacy, latency, customisation, and avoiding vendor dependency — not cost per token at most scales.

What This Does Not Change

Lower inference cost does not solve the problems that actually constrain production AI systems:

• Latency. Cheap and fast are different dimensions. Many use cases need sub-second responses. Token pricing is irrelevant if the response takes four seconds.
• Context quality. What you put in determines what comes out. Retrieval quality, chunking strategy, and prompt engineering remain engineering problems that pricing does not solve.
• Evaluation. You still need to know whether the model is doing what you want. Cheap tokens do not make evaluation easier.
• Reliability. API availability, rate limits, and consistency of output are not improved by lower prices.

Where the Floor Is

Inference costs will continue to fall. The question is the gradient and the floor.

The compute floor is set by physics — you cannot run a 70B parameter model on fewer FLOPs than the matrix multiplications require. But there are multiple levers still being pulled: mixture-of-experts architectures that activate a fraction of parameters per token, improved training procedures that produce more capable smaller models, hardware advances that continue the efficiency curve, and further quantisation research including 2-bit experiments.

A reasonable projection for 2026 is another 3–5x reduction in cost per token for comparable quality. The more important question for most teams is not whether costs fall further but how to architect systems that remain sound across a wide range of future pricing rather than being optimised for today's numbers.

Practical Implications

If you are designing a new AI system now:

1. Do not over-engineer cost reduction. Build for correctness and maintainability first. Optimise token usage only where profiling shows it matters.
2. Make your model provider choice a pluggable dependency. Avoid deep coupling to provider-specific APIs. The competitive landscape will continue to shift.
3. Re-evaluate existing caching and routing layers. Some may have become complexity without benefit.
4. Consider the full cost of the system, not just inference. Embedding infrastructure, evaluation pipelines, and retrieval systems often cost more in engineering time and operational overhead than the inference itself.

The drop in inference pricing is real and ongoing. Whether it changes your immediate architecture depends on where you are in the build cycle and what your specific bottlenecks are. It should, at minimum, prompt a review of any design decision that was made primarily on cost grounds more than six months ago.