Review update · 17 October 2026. Corrected causal masking, dense-attention complexity, FlashAttention memory claims and the overly rigid account of where facts are stored. Added a dated token-limit clarification and primary research links.
What the Attention Operation Computes
In standard scaled dot-product attention, each query is compared with permitted keys. After scaling and any mask, softmax produces weights used to combine the value vectors. Self-attention draws queries, keys and values from the same sequence representation; cross-attention can draw them from different sequences. The original Transformer paper describes the operation and its multi-head form.
The word “permitted” matters. In a causal language-model decoder, a token cannot attend to later tokens. Padding, local or other masks can restrict access further. It is incorrect to say every token always attends to all other tokens. PyTorch’s implementation documentation makes the masking and scaling explicit.
The machinery that implements this uses three projections of each token's representation:
- Queries (Q) — encoding "what am I looking for?"
- Keys (K) — encoding "what do I contain?"
- Values (V) — encoding "if you attend to me, what do you take?"
Attention weights are computed by taking the dot product of a query with each key (measuring compatibility), scaling by the square root of the key dimension (stabilising gradients), and applying softmax to get a probability distribution. That distribution is used to compute the weighted sum of value vectors.
Why This Is Powerful
The critical property of attention is dynamic, content-dependent aggregation. Unlike convolutional networks (fixed local neighbourhoods) or recurrent networks (fixed bottleneck), dense self-attention can connect distant positions directly when its mask permits them. Position information and masks also affect the computation, so the pattern is not determined by content alone.
This means long-range dependencies are first-class citizens. A pronoun at position 200 can attend directly to its referent at position 5 if the content relationship supports it. The same architectural operation handles both local and global dependencies.
For dense attention over a full length-n sequence, the pairwise computation scales quadratically in n at fixed dimensions. A naive implementation also materialises a quadratic score matrix. This describes the attention operation, not the cost of every part of a model or every decoding step. Cached autoregressive generation and specialised architectures need separate analysis.
Multi-Head Attention
A single attention head computes one pattern of weighted aggregation. Multi-head attention runs h attention operations in parallel, each with its own Q/K/V projections at reduced dimension, then concatenates and projects the results.
This gives the model capacity to simultaneously attend to the sequence in multiple different ways — different heads can track different types of relationships. In practice, heads do exhibit specialisation, though this is an emergent property of training rather than something explicitly designed.
The Feed-Forward Layers: The Other Half
Feed-forward layers transform each position’s representation alongside attention and residual connections. The key-value memory interpretation provides one useful account of their learned associations; it does not mean all such layers are sparse, or that every fact has a single address.
Feed-forward layers can support learned associations, but “facts live in the MLP; attention only routes” is too clean a separation. Geva and colleagues’ key-value memory interpretation is an empirical account of the models they studied, not proof of one exclusive storage location for every model. Layers interact through the residual stream.
The Scaling Hypothesis: What It Actually Claims
The scaling hypothesis makes claims that are frequently misrepresented in popular accounts.
What the cited study establishes. Chinchilla’s compute-optimal training study examined how to allocate a fixed training budget between parameters and training data. Its results do not establish a universal “20 tokens per parameter” rule for every architecture, training regime or deployment objective. Predicting language-model loss is also different from guaranteeing performance on a downstream task.
What it does not claim: that capability improvements are guaranteed at any scale; that all capabilities scale smoothly (some appear to emerge non-linearly); that scaling alone is sufficient without qualitative improvements in architecture, training procedure, and data curation; that loss on the training distribution predicts performance on out-of-distribution tasks.
The popular "we just need more scale" interpretation overreads the empirical findings. The more careful reading is that scale is a reliable lever for improving performance on tasks well-represented in the training distribution, within known architectural constraints.
Architectural Evolution: What Has Actually Changed
The basic transformer architecture from 2017 is remarkably close to what large models use today. The significant accumulated changes:
- Rotary Positional Embedding (RoPE) — a positional representation; extrapolation beyond training length requires evaluation, not an automatic reliability assumption.
- Grouped Query Attention (GQA) — groups of query heads share key/value heads, reducing KV-cache requirements compared with full multi-head attention. Quality and speed depend on the model and setup. See the GQA paper.
- RMSNorm — a normalisation variant used in some architectures; it is not evidence that all models use the same design.
- SwiGLU/GeGLU activations — gated activation choices used in some models, rather than a guarantee of a better result for every training setup.
- Flash Attention — an exact, IO-aware attention algorithm that avoids materialising the full attention matrix in high-bandwidth memory. Its linear auxiliary-memory behaviour does not remove dense attention’s quadratic arithmetic. See FlashAttention.
Architecture, data, training and inference choices all affect a system. This article does not quantify their relative contributions across recent models, and should not claim that architecture has converged permanently.
Implications for Practitioners
Context needs evaluation. Attention provides a mechanism to combine permitted context representations, not a guarantee that every relevant detail will affect the answer. Test factual use of context rather than infer it from window size.
Avoid a rigid storage-versus-routing story. Feed-forward and attention components participate in the same computation. Their interactions are more useful to understand than an absolute claim that a fact belongs to only one component.
Token update · 17 October 2026. A published context window, a maximum output allowance and the reliable use of information are different things. Check the exact model and endpoint limits. Dense full-sequence attention has quadratic work, but that does not mean doubling a prompt always quadruples the total serving bill or per-token decoding cost. Model/runtime choices and caching matter.
Size is one variable. Evaluate the model on your task, including position-sensitive retrieval, rather than treat parameter count or a benchmark average as a quality guarantee.
Understanding the mechanics of what transformers compute does not directly make you a better prompt engineer. It does make you better at reasoning about why models behave as they do — which is prerequisite to reliable system design.
Related reading: RAG in Production: Lessons from Real Deployments.