Attention

Attention is a family of operations in neural networks that computes an output by assigning data-dependent weights to a collection of representations and combining them. The weights depend on the current query or context, so the same input element can matter differently at different decoding steps, spatial positions, or layers. Attention first became prominent as a way to remove the fixed-vector bottleneck in neural machine translation and later became the main token-mixing operation in Transformers.

In machine learning, the term is a technical label rather than a claim that the operation reproduces human attention. It covers several related constructions, including recurrent encoder-decoder attention, self-attention, cross-attention, sparse attention, and multi-head attention. Their common feature is learned, input-dependent aggregation. This page is the hub for the wiki's attention coverage: it works through scaled dot-product attention with a numerical example and links the pages on multi-head variants, positional methods, the KV cache, and efficient and sparse implementations.

Definition and scope

Before attention, an influential sequence-to-sequence design encoded a source sequence into one fixed-dimensional vector and used a second recurrent network to decode the target sequence.^[1] Bahdanau, Cho, and Bengio instead let the decoder form a different context vector at each output step by weighting all encoder states. They described this as a differentiable soft search over source positions and trained the alignment and translation model jointly.^[2] This mechanism was introduced for machine translation, but the same idea soon appeared in image captioning and speech recognition.^[3][4][5]

At an abstract level, an attention layer receives:

• one or more queries, which specify what information is being requested;
• keys, which are compared with the queries;
• values, which contain the information to be aggregated;
• a scoring function and, usually, a normalization function.

For query $q_i$, keys $k_j$, and values $v_j$, a common form is:

$$e_{ij} = s(q_i, k_j)$$ $$\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{\ell \in A_i}\exp(e_{i\ell})}$$ $$o_i = \sum_{j \in A_i}\alpha_{ij}v_j$$

Here, $s$ is a learned or fixed compatibility score, $A_i$ is the set of positions that query $i$ is allowed to attend to, $\alpha_{ij}$ is an attention weight, and $o_i$ is the output. With softmax normalization, the weights over $A_i$ are nonnegative and sum to one. A mask changes $A_i$ by excluding padding positions, future positions, non-neighbors, or other disallowed connections.

This definition separates three choices that are sometimes conflated:

1. What can communicate? The attention pattern or mask determines the allowed query-key pairs.
2. How is relevance scored? The score may be additive, dot-product, bilinear, or another learned function.
3. What is transmitted? The values need not equal the keys, even when they originate from the same input.

Attention is therefore not synonymous with the Transformer. Recurrent encoder-decoder models used it before the Transformer, and attention modules can be inserted into convolutional, graph, set, and multimodal architectures.

Historical development

Encoder-decoder attention

The 2014 Bahdanau model addressed the fixed-vector bottleneck of recurrent encoder-decoder translation. At decoder step $i$, an alignment model scored the previous decoder state against each annotation produced by a bidirectional encoder. A softmax converted those scores into weights, and their weighted sum formed a step-specific context vector.^[2] The alignment model used a small feed-forward network, so this construction is commonly called additive attention or Bahdanau attention.

Luong, Pham, and Manning later compared global attention, which considers all source positions, with local attention, which restricts attention to a window. They also studied dot-product, bilinear, and concatenation-based score functions.^[3] These papers established that "attention" denotes a family of alignment and aggregation choices rather than a single formula.

The distinction between soft and hard attention concerns how elements are selected. Soft attention averages values using continuous weights and is differentiable through the weights. Hard attention samples or chooses discrete locations, which can reduce the number of evaluated locations but generally requires a gradient estimator or another training strategy. Xu and colleagues evaluated both forms for image captioning.^[4] Chorowski and colleagues adapted recurrent attention to speech and added location-aware information to help the alignment progress through long acoustic sequences.^[5]

Transformer attention

The 2017 paper "Attention Is All You Need" introduced an encoder-decoder architecture built from attention and position-wise feed-forward layers, without recurrent or convolutional sequence mixing.^[6] Its central operation is scaled dot-product attention:

$$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)V$$

The rows of $Q$ contain queries, the rows of $K$ contain keys, the rows of $V$ contain values, and $M$ is an optional mask. The scale factor $1/\sqrt{d_k}$ limits the typical magnitude of dot products as key dimension $d_k$ grows. Without that scaling, large logits can place softmax in regions with small gradients.^[6]

For an input matrix $X$, a self-attention layer usually creates the three matrices with learned projections:

$$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V$$

Transformer attention made it possible to compute representations for all sequence positions in parallel during training. It also reduced the maximum path length between two positions to a constant number of attention operations, although a full attention layer compares all pairs and therefore has quadratic score-matrix size in sequence length.^[6]

Scaled dot-product attention, step by step

This section walks through one attention layer in the notation of the 2017 Transformer paper, then computes a two-token example by hand. The sub-pages linked from the tables further down cover each variant in depth; the purpose here is to fix the shapes, the scale factor, and the order of operations that every variant inherits.

The projections: Q, K, and V

Start with an input matrix $X$ of shape $n \times d_{\text{model}}$, one row per token. Three learned matrices turn it into queries, keys, and values:

$$Q = XW^Q,\qquad K = XW^K,\qquad V = XW^V$$

$W^Q$ and $W^K$ have shape $d_{\text{model}} \times d_k$, and $W^V$ has shape $d_{\text{model}} \times d_v$. Queries and keys must share the dimension $d_k$ because they are dotted together; values may have a different width $d_v$, and the output inherits it. In the original model, $d_{\text{model}} = 512$, and with $h = 8$ heads each head used $d_k = d_v = d_{\text{model}}/h = 64$.^[6] In self-attention the same $X$ feeds all three projections; in cross-attention the queries come from one representation and the keys and values from another (see the encoder-decoder subsection below).

Scores and the scale factor

The score matrix compares every query with every key:

$$S = \frac{QK^\top}{\sqrt{d_k}}$$

$S$ has shape $n \times n$, and entry $S_{ij}$ measures how strongly token $i$ attends to token $j$. The authors of the Transformer paper reported that without scaling, additive attention outperformed dot-product attention for larger values of $d_k$, and they wrote that they "suspect" the reason is that the dot products grow large in magnitude, pushing the softmax into regions where it has extremely small gradients. Dividing by $\sqrt{d_k}$ is their remedy.^[6] Their footnote gives the intuition: if the components of $q$ and $k$ are independent random variables with mean 0 and variance 1, then $q \cdot k$ has mean 0 and variance $d_k$, so dividing by $\sqrt{d_k}$ brings the variance back to 1.^[6]

The divisor is the per-head key dimension, not the model width. With the original settings, the scores are divided by $\sqrt{64} = 8$, not by $\sqrt{512} \approx 22.6$. Using $d_{\text{model}}$ by mistake flattens the softmax far more than intended.

Softmax and masking

Softmax is applied row by row, so each query's weights over the keys are nonnegative and sum to one. A mask $M$ is added to the scores before the softmax: allowed positions get 0, disallowed positions get $-\infty$ (in practice a large negative number), so that they contribute exactly zero after exponentiation. The Transformer paper describes its causal mask this way: it prevents leftward information flow in the decoder "by masking out (setting to $-\infty$) all values in the input of the softmax which correspond to illegal connections."^[6] Applying a mask after the softmax instead of before would leave the remaining weights summing to less than one, which is a different and unintended operation.

The final step multiplies the weights by the values:

$$P = \operatorname{softmax}(S + M),\qquad O = PV$$

$O$ has shape $n \times d_v$. In a multi-head layer, this whole computation runs once per head on its own projected $Q$, $K$, and $V$, and the head outputs are concatenated and projected by $W^O$.^[6]

A softmax over large scores is computed with the row maximum subtracted first, $\operatorname{softmax}(x)_i = e^{x_i - m(x)} / \sum_j e^{x_j - m(x)}$ with $m(x) = \max_i x_i$. The result is mathematically identical, but the exponentials stay in range. FlashAttention uses exactly this form and then shows that the maximum and the normalizer can be maintained incrementally across blocks of keys, which is what lets it avoid materializing the full score matrix.^[19] The incremental form comes from Milakov and Gimelshein's online normalizer calculation.^[40]

A worked example

Take two tokens with $d_{\text{model}} = d_k = d_v = 2$, so every matrix is 2 x 2. The projection weights below are toy values chosen so that the arithmetic can be checked by hand; a trained layer learns them.

$$X = \begin{pmatrix} 1 & 2 \\ 2 & 1 \end{pmatrix},\quad W^Q = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix},\quad W^K = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix},\quad W^V = \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix}$$

The projections give $Q = X$, $K$ with its columns swapped, and $V$ with its second column negated:

$$Q = \begin{pmatrix} 1 & 2 \\ 2 & 1 \end{pmatrix},\quad K = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix},\quad V = \begin{pmatrix} 1 & -2 \\ 2 & -1 \end{pmatrix}$$

The remaining steps, with values rounded to four decimals:

Token 1 attends more to token 2 (weight 0.6698) because $q_1 \cdot k_2 = 5$ exceeds $q_1 \cdot k_1 = 4$, and its output is a blend of the two value rows leaning toward $v_2 = (2, -1)$. Skipping the scale factor would have produced weights of (0.2689, 0.7311) for token 1: the same ordering, but a sharper distribution, and the sharpening grows with $d_k$.

Causal masking

Autoregressive models add a mask that lets position $i$ see only positions $j \le i$. As a matrix, the mask has 0 on and below the diagonal and $-\infty$ above it:

$$M = \begin{pmatrix} 0 & -\infty \\ 0 & 0 \end{pmatrix}$$

Applied to the example, token 1's scaled scores become (2.8284, $-\infty$), its weights become (1, 0), and its output is exactly $v_1 = (1, -2)$. Token 2 already sees both tokens, so its row is unchanged: weights (0.6698, 0.3302) and output (1.3302, -1.6698). The general rule is that the output at position $i$ depends only on tokens up to $i$, which has two consequences. During training, all positions are computed in parallel from one masked score matrix, since the ground-truth sequence is known and each row already respects the constraint.^[13] During generation, the keys and values of earlier positions do not change when a new token is appended, so they can be stored rather than recomputed; that is the KV cache described below. Bidirectional encoders such as BERT omit this mask, and padding masks are combined with it by the same addition before the softmax. The earlier subsection on causal, bidirectional, and padding masks discusses the distinctions in more detail.

Main forms

Self-attention and cross-attention

In self-attention, queries, keys, and values are produced from the same sequence or set. Each output position can therefore combine information from other positions in that input. In cross-attention, queries come from one representation while keys and values come from another. Encoder-decoder translation uses decoder states as queries and encoder states as keys and values. The same arrangement can condition an image representation on text or connect another pair of modalities.

These terms describe the source of $Q$, $K$, and $V$, not the score function. Either self-attention or cross-attention may be single-head or multi-head, dense or sparse, causal or noncausal.

Causal, bidirectional, and padding masks

A causal mask permits position $i$ to attend only to positions at or before $i$. In a score matrix, disallowed entries are assigned a value that becomes zero after softmax, commonly implemented with negative infinity before normalization. Causal self-attention is used for autoregressive language models.

An unmasked encoder can attend in both sequence directions. For example, BERT uses bidirectional Transformer encoders and a masked-language-model objective. The word "bidirectional" here describes the allowed attention pattern, not a separate scoring rule.

Padding masks exclude placeholder positions introduced when unequal-length examples are batched. Other masks can enforce local windows, graph neighborhoods, block structure, or application-specific constraints. Correct masking is part of the mathematical definition of a layer, not merely an implementation detail.

Multi-head attention

Multi-head self-attention applies several learned query, key, and value projections in parallel:

$$\operatorname{head}_r = \operatorname{Attention}(QW_r^Q,KW_r^K,VW_r^V)$$ $$\operatorname{MHA}(Q,K,V) = \operatorname{Concat}(\operatorname{head}_1,\ldots,\operatorname{head}_h)W^O$$

The original Transformer divided the representation across heads, concatenated their outputs, and applied an output projection.^[6] Multiple heads allow distinct projected interactions to be represented in one layer. A head should not, however, be assumed to correspond to one stable linguistic or semantic concept. Head behavior depends on the model, layer, input, training run, and analytical method.

Additive and multiplicative scores

Two common compatibility functions are:

$$s_{\mathrm{add}}(q,k)=v_a^\top\tanh(W_q q+W_k k)$$ $$s_{\mathrm{dot}}(q,k)=q^\top k$$

Additive attention uses a learned feed-forward alignment model. Multiplicative attention uses a dot product or a bilinear form such as $q^\top Wk$. Additive and dot-product attention can represent different functions, and their practical cost depends on tensor shapes and hardware. Scaled dot-product attention is especially convenient because a batch of pairwise scores can be computed with matrix multiplication.^[6]

Multi-head attention and its variants

Multi-head attention runs $h$ copies of scaled dot-product attention in parallel, each with its own query, key, and value projections, and concatenates the results.^[6] The widely used variants keep the query heads but change how many distinct key and value heads exist, which in turn sets how much state a decoder has to keep per generated token. DeepSeek-V2's paper tabulates the KV cache per token for each variant by counting elements; the formulas below are copied from that table and the concrete example is derived from them.^[26]

The example column uses $n_h = 32$ query heads, $d_h = 128$, and $l = 32$ layers, which are the published dimensions of Mistral 7B; that model uses GQA with 8 key-value heads, so the GQA row is its actual per-token cache.^[31] The MLA row assumes DeepSeek-V2's choices $d_c = 4 d_h$ and $d_h^R = d_h / 2$, which the paper says makes its cache equal to GQA with 2.25 groups.^[26] Bytes are elements times two for 16-bit storage; the vLLM paper does the same arithmetic for OPT-13B and arrives at 800 KB per token.^[27]

Shazeer's motivation for MQA was a performance analysis of incremental decoding: across $n$ steps, the arithmetic is $\Theta(bnd^2)$ but the memory traffic is $\Theta(bn^2 d + nd^2)$, because the keys and values of every earlier position are reloaded at each step. The ratio of memory access to arithmetic, $\Theta(n/d + 1/b)$, approaches 1 when the sequence length is comparable to the model width or the batch is small, so memory bandwidth becomes the bottleneck. Sharing one key and value head across all query heads shrinks the tensors that have to be reloaded.^[13]

GQA divides the query heads into $G$ groups, each sharing one key head and one value head. GQA-1 is MQA and GQA-H (one group per head) is MHA, so the method interpolates between the two. The paper's other contribution is uptraining: an existing multi-head checkpoint is converted by mean-pooling the key and value heads of each group and then trained for a small fraction (5% in their experiments) of the original pre-training compute. The authors note that GQA is applied to decoder self-attention and not to encoder self-attention, where representations are computed in parallel and memory bandwidth is not the main constraint.^[14]

MLA compresses the keys and values of each token into a low-rank latent vector and reconstructs them with up-projections; in DeepSeek-V2 the up-projection for keys can be absorbed into the query projection at inference, so only the latent is cached. Because rotary position embedding rotates keys in a position-dependent way, it cannot be applied inside the compressed path, and the paper introduces a separate decoupled key per token that carries RoPE. DeepSeek-V2 used $n_h = 128$, $d_h = 128$, $d_c = 512$, and $d_h^R = 64$, and reported that MLA outperformed MHA on its benchmarks while using a much smaller cache.^[26]

Positional information

Attention on its own is permutation-equivariant, so position must be injected somewhere. The original Transformer added sinusoidal or learned position vectors to the token embeddings.^[6] Two later methods put the information into the attention scores instead, and both have their own pages.

RoPE became the default in most open decoder-only models, and its interaction with the cache-reduction methods above is not free: DeepSeek-V2 needed the decoupled key precisely because RoPE is incompatible with its low-rank compression.^[26] ALiBi's authors frame their method as an answer to the question of how a model extrapolates to sequences longer than those seen in training.^[25]

Inference: the KV cache

Once the causal mask is in place, the key and value rows of earlier tokens never change, so a decoder stores them and computes only the new token's query, key, and value at each step. The stored state is the KV cache, and its size is the per-token count in the variants table multiplied by the number of tokens in context. The vLLM paper gives a concrete figure: for a 13B-parameter OPT model, one token's cache occupies 800 KB, computed as 2 (keys and values) x 5120 (hidden size) x 40 (layers) x 2 (bytes per FP16 value), so a 2048-token request needs up to 1.6 GB and only a few tens of requests fit on a GPU with tens of gigabytes of memory.^[27]

The per-token decode cost is dominated by reading that cache. In Shazeer's analysis, each decode step reloads the full $K$ and $V$ tensors, so memory traffic grows with the square of the sequence length across a generation while arithmetic grows linearly.^[13] The PyTorch blog post introducing Flash-Decoding adds the hardware side: at decode time the query length is typically 1, FlashAttention parallelizes only across batch and query length, and with batch size 1 it uses less than 1% of an A100 GPU, which has 108 streaming multiprocessors. Flash-Decoding splits the keys and values along the sequence dimension, computes each split's attention with a log-sum-exp scalar per row, and reduces across splits; the authors measured attention up to 50x faster than FlashAttention and end-to-end decoding up to 8x faster on CodeLlama-34B at long sequence lengths.^[28]

Efficient and sparse implementations

Two different kinds of work go by the name "efficient attention". The first computes exactly the same function as dense attention but organizes memory traffic better. The second changes which query-key pairs are computed, or replaces the softmax with a cheaper operator, and therefore changes the model. The table separates them, and the earlier comparison table in the efficiency section gives the same distinction in summary form.

Each entry in the "Claim from the paper" column is the authors' own result under their own setup, not an independent measurement. Whether a sparse or linear method preserves quality depends on the task, and the FlashAttention paper's motivation was precisely that approximate methods often failed to deliver wall-clock speedups.^[19]

Cross-attention and encoder-decoder use

Cross-attention is the arrangement where queries come from one sequence and keys and values from another. It predates the Transformer: in Bahdanau, Cho, and Bengio's translation model, the decoder's previous hidden state acts as the query, the encoder's annotations act as keys and values, and a small feed-forward alignment model produces the scores, so the decoder performs a soft search over source positions at every output step.^[2] The Transformer keeps the idea in its "encoder-decoder attention" layers, where the queries come from the previous decoder layer and the keys and values come from the encoder output, so that every decoder position can attend over all input positions.^[6] The Attention Is All You Need page covers the full architecture.

Decoder-only language models have no encoder and therefore no cross-attention; the conditioning text is placed in the same sequence as the output and handled by causal self-attention. Encoder-decoder models such as T5 keep both kinds of layer, and the GQA paper's observation that memory bandwidth is not the bottleneck for the encoder applies to that layout.^[14]

Structural properties

Weighted message passing

Attention can be understood as message passing. Each key-value pair offers a message, a query determines the weights, and the output aggregates the selected values. This view extends beyond sequences. Graph Attention Networks restrict a node's attention to graph neighbors and learn unequal coefficients for their messages.^[7] Non-local neural-network blocks compute a weighted sum across positions in image or video feature maps.^[8] Set Transformer uses attention while constructing permutation-invariant models for set-valued inputs.^[9]

Order and position

Self-attention without position-dependent features is permutation equivariant: reordering input rows reorders the outputs in the same way. That property is useful for sets but insufficient when sequence order matters. Transformer models therefore add or incorporate positional encoding. The original Transformer added sinusoidal or learned position representations to the input embeddings.^[6] Later methods place relative position information in the score or modify queries and keys, but those choices are separate from the basic attention operation.

Content-dependent receptive fields

A dense self-attention layer can connect every permitted pair of positions in one step. Unlike a fixed convolution kernel, its weights depend on the current representations. This does not guarantee that a trained model will use distant information effectively. It only means the computation graph permits the connection. Optimization, training data, positional representation, masking, and numerical precision all affect what information is actually used.

Applications

Attention moved from recurrent translation systems into several model families.

• Language and speech: Encoder-decoder attention supports alignment between source and target sequences, while Transformer self-attention supplies contextual token representations. Location-aware attention was developed for speech recognition.^[5]
• Vision: Early visual attention models weighted spatial features during caption generation.^[4] Non-local blocks later applied related aggregation to image and video features.^[8] A Vision Transformer treats image patches as a sequence and processes them with Transformer encoders.^[10]
• Multimodal generation: Latent diffusion models use cross-attention to condition image-generation features on representations such as text or spatial inputs.^[11] This is an example of cross-attention joining two representation streams rather than a special new scoring function.
• Graphs and sets: Masks can restrict communication to graph edges, while unmasked set attention can model pairwise interactions without imposing an input order.^[7][9]
• Biomolecular modeling: AlphaFold 2's Evoformer contains attention-based and non-attention-based updates over multiple-sequence-alignment and residue-pair representations, and its structure module uses invariant point attention.^[12]

These examples use the same broad pattern but differ substantially in tokenization, masks, geometry, objectives, and surrounding architecture. Results from one domain do not by themselves establish that a particular attention design is best in another.

Efficiency and memory

Full attention

For sequence length $n$, a dense self-attention score matrix contains $n^2$ entries. With head dimension $d$, forming scores and applying them to values takes on the order of $n^2d$ arithmetic operations. A straightforward implementation also materializes an $n \times n$ intermediate for each head and batch element. This cost becomes important for long sequences and high-resolution spatial inputs.^[6]

During autoregressive decoding, previously computed keys and values are commonly retained in a KV cache. Caching avoids recomputing projections for earlier tokens, but the stored state grows with sequence length and the number of key-value heads.

Multi-query attention shares one set of keys and values across query heads, reducing the amount of cached and loaded key-value data during incremental decoding.^[13] Grouped-query attention uses an intermediate number of key-value heads and was proposed as a compromise between standard multi-head and multi-query attention.^[14] These variants reduce key-value memory and bandwidth; they do not remove the pairwise query-key work of dense attention.

Sparse and approximate attention

Sparse attention limits the allowed pairs. Longformer combines a sliding local window with selected global positions, making its attention cost scale linearly with sequence length when the window and number of global positions are fixed.^[15] BigBird combines local, global, and random connections and provides theoretical results for that particular sparse pattern.^[16] Sparse designs trade universal direct connectivity for a chosen communication graph, so accuracy and efficiency depend on whether the pattern fits the task.

Linear attention methods rewrite or approximate attention so that key-value aggregation can be performed before combining it with each query. Katharopoulos and colleagues used kernel feature maps and matrix associativity to obtain linear scaling in sequence length for their formulation.^[17] Performer used positive orthogonal random features to approximate softmax attention.^[18] Such methods alter the operator or approximate it, and their quality, stability, and actual speed are empirical questions rather than consequences of asymptotic notation alone.

Exact implementation improvements

Flash Attention computes exact scaled dot-product attention with an IO-aware tiled algorithm. It avoids writing the full attention matrix to high-bandwidth memory by processing blocks in faster on-chip memory and maintaining the softmax normalization across blocks.^[19] Its arithmetic dependence on sequence length remains quadratic for dense attention, but its auxiliary memory use and memory traffic are lower than a straightforward implementation.

This distinction is important:

Interpretation and limitations

Attention weights are conditional coefficients

An attention matrix records coefficients used at one layer, for one head and one input. It can reveal which value vectors were weighted strongly in that computation. It does not automatically measure a raw input feature's causal effect on the final prediction.

Jain and Wallace found that attention weights in the NLP models they studied were often weakly related to gradient-based importance measures and that substantially different attention distributions could sometimes produce similar outputs.^[20] Wiegreffe and Pinter argued that whether attention counts as explanation depends on the definition, model, and test, and proposed alternative diagnostics rather than a universal rejection.^[21] Serrano and Smith likewise found that attention magnitudes were not a fail-safe measure of importance under intervention.^[22]

In a deep Transformer, residual connections and repeated mixing further complicate interpretation. Abnar and Zuidema proposed attention rollout and attention flow to account for information propagation across layers and found higher correlations with ablation and gradient measures than raw attention in their experiments.^[23] These methods remain diagnostics, not proofs of causal responsibility.

Identifiability and saturation

Different score vectors can yield similar weighted sums, especially when value vectors are redundant. Softmax can also saturate when logits have large magnitude, leading to very peaked distributions and small derivatives. Scaling, normalization, initialization, precision, and masking therefore affect training behavior.

Quadratic connectivity is not guaranteed recall

Dense attention permits every position to interact with every other permitted position, but permission is not successful use. Long sequences can still challenge retrieval, optimization, and position handling. Conversely, a sparse or approximate layer may be adequate when relevant dependencies follow its communication pattern. Claims about long-context ability should be evaluated on the target distribution and task, not inferred from context-window size or complexity alone.

Attention is one component

Transformer behavior does not arise from attention in isolation. Feed-forward layers, residual paths, normalization, tokenization, positional representations, objectives, data, and decoding all contribute. Replacing or visualizing attention addresses only one part of the system.

Common misconceptions and pitfalls

The following errors recur in implementations and explanations. Each row cites the source that settles it.

Practical checks

When implementing or evaluating an attention layer, several checks prevent common errors:

• Apply padding and causal masks before softmax and verify that masked probabilities are zero.
• Confirm the intended axes for query length, key length, heads, and batch dimensions.
• Use numerically stable softmax and sufficient precision for normalization and accumulation.
• Distinguish training-time parallelism from autoregressive decode latency.
• Report both asymptotic cost and measured wall-clock behavior on the target hardware.
• Treat attention visualizations as hypotheses for further testing, not standalone explanations.
• Compare variants at matched parameter count, training budget, sequence length, and evaluation setting when possible.

References

1. ^Sutskever, I., Vinyals, O., and Le, Q. V. "Sequence to Sequence Learning with Neural Networks." Advances in Neural Information Processing Systems 27, 2014. proceedings.neurips.cc/...97f410bb7eca942-Abstract
2. ^1 ^2 ^3 ^4Bahdanau, D., Cho, K., and Bengio, Y. "Neural Machine Translation by Jointly Learning to Align and Translate." ICLR 2015, first posted 2014. arxiv.org/...1409.0473
3. ^1 ^2Luong, M.-T., Pham, H., and Manning, C. D. "Effective Approaches to Attention-based Neural Machine Translation." EMNLP 2015, pp. 1412-1421. aclanthology.org/D15-1166
4. ^1 ^2 ^3Xu, K., Ba, J., Kiros, R., et al. "Show, Attend and Tell: Neural Image Caption Generation with Visual Attention." ICML 2015, pp. 2048-2057. proceedings.mlr.press/...xuc15
5. ^1 ^2 ^3Chorowski, J. K., Bahdanau, D., Serdyuk, D., Cho, K., and Bengio, Y. "Attention-Based Models for Speech Recognition." Advances in Neural Information Processing Systems 28, 2015. proceedings.neurips.cc/...e9ea8072e3189e2-Abstract
6. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19Vaswani, A., Shazeer, N., Parmar, N., et al. "Attention Is All You Need." Advances in Neural Information Processing Systems 30, 2017. papers.nips.cc/...47dee91fbd053c1c4a845aa-Abstract
7. ^1 ^2Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. "Graph Attention Networks." ICLR 2018. openreview.net/forum
8. ^1 ^2Wang, X., Girshick, R., Gupta, A., and He, K. "Non-Local Neural Networks." CVPR 2018, pp. 7794-7803. openaccess.thecvf.com/..._Networks_CVPR_2018_paper
9. ^1 ^2Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., and Teh, Y. W. "Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks." ICML 2019, pp. 3744-3753. proceedings.mlr.press/...lee19d
10. ^Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. "An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale." ICLR 2021. openreview.net/forum
11. ^1 ^2Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. "High-Resolution Image Synthesis With Latent Diffusion Models." CVPR 2022, pp. 10684-10695. openaccess.thecvf.com/...on_Models_CVPR_2022_paper
12. ^Jumper, J., Evans, R., Pritzel, A., et al. "Highly accurate protein structure prediction with AlphaFold." Nature 596, 583-589, 2021. nature.com/...s41586-021-03819-2
13. ^1 ^2 ^3 ^4 ^5 ^6 ^7Shazeer, N. "Fast Transformer Decoding: One Write-Head is All You Need." 2019. arxiv.org/...1911.02150
14. ^1 ^2 ^3 ^4 ^5Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S. "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints." EMNLP 2023, pp. 4895-4901. aclanthology.org/2023.emnlp-main.298
15. ^1 ^2Beltagy, I., Peters, M. E., and Cohan, A. "Longformer: The Long-Document Transformer." 2020. arxiv.org/...2004.05150
16. ^1 ^2Zaheer, M., Guruganesh, G., Dubey, K. A., et al. "Big Bird: Transformers for Longer Sequences." Advances in Neural Information Processing Systems 33, 2020. proceedings.neurips.cc/...5f31a9a7a361ab9-Abstract
17. ^1 ^2Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention." ICML 2020, pp. 5156-5165. proceedings.mlr.press/...katharopoulos20a
18. ^1 ^2Choromanski, K., Likhosherstov, V., Dohan, D., et al. "Rethinking Attention with Performers." ICLR 2021. openreview.net/forum
19. ^1 ^2 ^3 ^4 ^5 ^6Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Re, C. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." Advances in Neural Information Processing Systems 35, 2022. proceedings.neurips.cc/...40d5-Abstract-Conference
20. ^1 ^2Jain, S., and Wallace, B. C. "Attention is not Explanation." NAACL-HLT 2019, pp. 3543-3556. aclanthology.org/N19-1357
21. ^1 ^2Wiegreffe, S., and Pinter, Y. "Attention is not not Explanation." EMNLP-IJCNLP 2019, pp. 11-20. aclanthology.org/D19-1002
22. ^Serrano, S., and Smith, N. A. "Is Attention Interpretable?" ACL 2019, pp. 2931-2951. aclanthology.org/P19-1282
23. ^Abnar, S., and Zuidema, W. "Quantifying Attention Flow in Transformers." ACL 2020, pp. 4190-4197. aclanthology.org/2020.acl-main.385
24. ^Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. "RoFormer: Enhanced Transformer with Rotary Position Embedding." arXiv:2104.09864, first posted April 2021. arxiv.org/...2104.09864
25. ^1 ^2Press, O., Smith, N. A., and Lewis, M. "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation." arXiv:2108.12409, first posted August 2021. arxiv.org/...2108.12409
26. ^1 ^2 ^3 ^4 ^5 ^6 ^7DeepSeek-AI. "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model." arXiv:2405.04434, first posted May 2024. arxiv.org/...2405.04434
27. ^1 ^2 ^3 ^4 ^5Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. "Efficient Memory Management for Large Language Model Serving with PagedAttention." arXiv:2309.06180, September 2023. arxiv.org/...2309.06180
28. ^1 ^2Dao, T., Haziza, D., Massa, F., and Sizov, G. "Flash-Decoding for long-context inference." PyTorch Blog, October 13, 2023. pytorch.org/...flash-decoding
29. ^Dao, T. "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning." arXiv:2307.08691, July 2023. arxiv.org/...2307.08691
30. ^Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision." arXiv:2407.08608, July 2024. arxiv.org/...2407.08608
31. ^1 ^2 ^3 ^4Jiang, A. Q., Sablayrolles, A., Mensch, A., et al. "Mistral 7B." arXiv:2310.06825, October 2023. arxiv.org/...2310.06825
32. ^Liu, H., Zaharia, M., and Abbeel, P. "Ring Attention with Blockwise Transformers for Near-Infinite Context." arXiv:2310.01889, first posted October 2023. arxiv.org/...2310.01889
33. ^Yuan, J., Gao, H., Dai, D., et al. "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention." arXiv:2502.11089, first posted February 2025. arxiv.org/...2502.11089
34. ^DeepSeek-AI. "DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models." arXiv:2512.02556, December 2025. arxiv.org/...2512.02556
35. ^Lu, E., Jiang, Z., Liu, J., et al. "MoBA: Mixture of Block Attention for Long-Context LLMs." arXiv:2502.13189, February 2025. arxiv.org/...2502.13189
36. ^Munkhdalai, T., Faruqui, M., and Gopal, S. "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention." arXiv:2404.07143, first posted April 2024. arxiv.org/...2404.07143
37. ^1 ^2 ^3Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. "Efficient Streaming Language Models with Attention Sinks." arXiv:2309.17453, first posted September 2023. arxiv.org/...2309.17453
38. ^Zhang, Z., Sheng, Y., Zhou, T., et al. "H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models." arXiv:2306.14048, first posted June 2023. arxiv.org/...2306.14048
39. ^Zheng, L., Yin, L., Xie, Z., et al. "SGLang: Efficient Execution of Structured Language Model Programs." arXiv:2312.07104, first posted December 2023. arxiv.org/...2312.07104
40. ^1 ^2Milakov, M., and Gimelshein, N. "Online normalizer calculation for softmax." arXiv:1805.02867, first posted May 2018. arxiv.org/...1805.02867
41. ^Qin, Z., Sun, W., Li, D., Shen, X., Sun, W., and Zhong, Y. "Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models." arXiv:2401.04658, first posted January 2024. arxiv.org/...2401.04658
42. ^MiniMax. "MiniMax-01: Scaling Foundation Models with Lightning Attention." arXiv:2501.08313, January 2025. arxiv.org/...2501.08313