Keep building AI. Rethink the machinery.

Research and analysis · Sources checked 24 October 2026 · No original hardware benchmark claimed.

If the next breakthrough in artificial intelligence needs another warehouse, a queue of memory chips and a small power station, perhaps the breakthrough also needs an engineering meeting.

I want AI to keep advancing. I want better tools, useful discoveries and more people able to build things. I don’t want every improvement to arrive with a bigger hardware invoice.

What AI developers have achieved is genuinely impressive. Scaling models, data and computing resources helped make that progress possible. The mindset I want to challenge is treating another increase in resources as the default answer to the next problem.

There has already been substantial optimisation, as the examples below show. My argument is about priorities: “How can we do more with what we already have?” should be a starting question, not a budget-cutting exercise after the expensive version is built. Resource efficiency and consumer affordability belong alongside capability as measures of progress.

Why is the AI boom on my upgrade bill?

You don’t have to use AI to help pay for its appetite. You might just want a graphics card for games, more RAM for editing, or an SSD because the old one is full. Those ordinary upgrades sit downstream of an industry increasingly organised around much bigger buyers.

This is the consumer problem I care about: paying more for the same capacity or performance, rather than getting a better product for the extra money. The ambition to build useful AI is impressive. Making everybody else’s hardware bill harder to swallow is a less impressive side effect.

The pressure is documented. In its 30 September 2026 outlook, TrendForce reported that suppliers were prioritising server memory capacity while graphics memory remained constrained. It forecast fourth-quarter contract price increases of 10–15% for conventional DRAM and 15–20% for NAND Flash. Those are quarterly industry forecasts, not measured retail price changes. It also reported PC brands reducing mainstream SSD capacities to control costs. Sometimes the squeeze means less storage in the box, not just a bigger price tag. [27]

Micron made the change in priorities unusually explicit when announcing its Crucial consumer-business exit in December 2025: AI demand and larger strategic customers were central to its explanation. Components still reach PCs through partners and manufacturers; this was a consumer-brand exit, not the disappearance of every supply route. [18] [19]

Consumer graphics cards are not data centre accelerators, and desktop RAM is not HBM. But production capacity, memory supply and investment decisions connect these markets. AI demand is a major source of pressure, alongside manufacturers’ product choices and supply constraints. That doesn’t establish that every GPU, RAM kit or SSD has doubled or tripled in price.

You shouldn’t need to be an AI customer to end up paying its hardware bill.

Efficiency belongs in this discussion because it offers a way to do more useful work without endlessly bidding for more components. It cannot promise an immediate retail price cut. But making smarter software and suitable specialist hardware a priority is a better direction than treating consumers as whoever gets the leftovers.

The good news: engineers are already doing this

Can AI become more efficient? It already is. The harder question is whether those gains can keep ahead of growing demand.

This principle has evidence behind it. DeepMind’s 2022 Chinchilla study trained a smaller model on more data using the same training compute budget as the larger Gopher model, then reported better performance across many evaluated tasks. It showed that how compute is allocated can matter as much as simply increasing model size. It was not a direct energy comparison. [26]

The IEA’s 2026 update projects global data centre electricity demand rising from roughly 485 TWh in 2025 to 950 TWh in 2030. That covers all data centres, not chatbots alone, and is a projection. The same report describes rapid efficiency improvements alongside growing use of demanding applications such as reasoning and video generation. [1]

For the engineering comparison, hold the work and quality steady. If the same five tasks each need a quarter of the energy, total consumption falls by 75%.

Growing use can absorb savings. In this simplified example, demand can grow to less than four times the original workload and total energy still falls. At four times, it breaks even; beyond that, it rises. That does not erase the value of doing each job efficiently.

“How many GPUs?” is sometimes the second question

The first is: what are we trying to do?

Training a frontier model, transcribing a recording and recognising “turn the lights off” are different jobs. Simple code may handle the last one without a language model. A compact speech model may handle the recording.

Use a small model where it meets the quality target, escalate when it doesn’t, and avoid repeating work unnecessarily. Retrieve the relevant passage instead of dragging a document collection into every request.

The target is the least wasteful successful answer, not the cheapest failed attempt.

Count retries, stale caches and idle machines. Three failed attempts can cost more than using a stronger model once. A local computer left running all day is not automatically greener than a well-utilised shared service.

Your AI has a removalist problem

Imagine employing a brilliantly fast chef, then making them walk to a distant pantry for every ingredient. That is the problem when a processor can calculate faster than memory can deliver the numbers.

Token-by-token generation, called decode, is often limited by memory bandwidth, particularly with small interactive batches. Processing the prompt, or prefill, is typically more compute intensive. The balance depends on the model, batch and context. [21]

GPU engineers call this data-movement limit the memory wall, related to the broader von Neumann bottleneck. [17] Model weights need storage; the key-value or KV cache adds working memory as conversations and concurrent requests grow.

FlashAttention reduces memory traffic while retaining exact attention. [2] PagedAttention improves serving-cache allocation. Its 2023 paper reported two to four times the throughput of the evaluated systems at comparable latency. That is a scoped research result, not a universal electricity-saving multiplier. [3]

Both still run on GPUs. Neither required the philosophical breakthrough of buying another building: they help existing hardware do more useful work.

Sometimes the clever bit is using fewer bits

Quantisation stores numbers at lower precision. Choose the detail the job needs, rather than writing every measurement to an absurd number of decimal places.

Seven billion weights at 16 bits take about 14 decimal GB. At four bits, the raw payload is 3.5 GB. The diagram compares that storage alone. [4]

Quality still matters, and the hardware must support the format efficiently. NVIDIA’s NVFP4 work shows that GPU makers are pursuing this too. [5]

Microsoft’s BitNet b1.58 2B4T uses ternary weights but 8-bit activations. It is a purpose-trained research model with two billion parameters, not a switch that shrinks any frontier model without consequences. [6] [22]

Teaching smaller models to mimic big ones

Distillation trains a smaller model to reproduce useful behaviour from a larger one. Unlike quantisation, it changes how another model learns. Training costs something; the potential payoff is cheaper inference across many later requests, provided quality holds up. [16]

The GPU is an excellent tool. It isn’t a law of physics.

AI did not discover GPUs after crypto miners got bored. Researchers used them years before the generative AI boom. Modern data centre GPUs include dedicated tensor hardware; they aren’t old graphics cards wearing a corporate lanyard. [7]

Cerebras brings substantial memory onto wafer-scale processors. Its 22 October explanation also makes the limit clear: huge models still need multiple wafers. Bringing memory closer doesn’t abolish storage requirements or make the system a cheap home accessory. [11]

A more efficient machine can remain expensive to buy or integrate. Replacing dependence on one supplier with another does not solve affordability by itself.

Change the scoreboard, not just the chip

Define the job and an acceptable result, then compare energy, latency and cost on equal terms. Tokens per second cannot tell you whether the answer was correct or three failed attempts came first.

MLPerf’s controlled inference benchmarks are a better reference than unrelated launch slides. [14] MLPerf Power specifically addresses energy measurement alongside performance. Keep the workload and measurement boundary explicit. [24]

One actual production measurement

Google’s August 2025 paper reported a median of 0.24 Wh per Gemini Apps text prompt, using May 2025 production data. It includes accelerators, host CPUs and RAM, allocated idle capacity and data centre overhead. Google says the findings were not independently verified. [23]

That is a dated median for one service, not a universal AI average, a current figure for every Gemini model, or a lifecycle accounting of training and manufacturing. Its value is the published measurement method.

Count the surrounding infrastructure too. Cooling, clean electricity, water, grid constraints and manufacturing impacts matter. Replacing an old machine can save operating energy while adding production emissions; keeping it forever can also be wrong.

Google’s lifecycle-based compute-carbon-intensity analysis reports a 3.7-fold Ironwood improvement over TPU v5p. That is a vendor metric with stated assumptions, not a guarantee for every customer workload. [15]

What should change on Monday morning?

Builders: measure one real workflow. Try smaller models or simpler code against the same quality test. Trim irrelevant context and stop unproductive agent loops. Keep a stronger fallback where needed.

Vendors: compete on useful work per unit of memory, energy and cost. Invest in suitable hardware and portable software instead of making bigger component orders the default answer. Take the knock-on effects on consumer supply seriously.

Buyers: check model compatibility, upgrade costs and whether you can migrate later. A cheap accelerator that needs expensive memory and integration may not be cheap at all.

Conclusion

AI should keep advancing. Better algorithms, memory management and suitable hardware can reduce the resources needed for that progress. That matters to energy systems and to consumers buying ordinary GPUs, RAM and storage. Some jobs need formidable machines; every job deserves a reason for using one.

Keep the ambition. Rethink the machinery.

Before the next purchase or deployment, ask: Can it do our actual job on hardware we can afford? What does a successful result cost in energy and money? Have we tested a less demanding way?

More analysis: what drives LLM inference costs and why slowing AI is the wrong question.