Attention

Attention is a family of operations in neural networks that computes an output by assigning data-dependent weights to a collection of representations and combining them. The weights depend on the current query or context, so the same input element can matter differently at different decoding steps, spatial positions, or layers. Attention first became prominent as a way to remove the fixed-vector bottleneck in neural machine translation and later became the main token-mixing operation in Transformers.

In machine learning, the term is a technical label rather than a claim that the operation reproduces human attention. It covers several related constructions, including recurrent encoder-decoder attention, self-attention, cross-attention, sparse attention, and multi-head attention. Their common feature is learned, input-dependent aggregation.

Definition and scope

Before attention, an influential sequence-to-sequence design encoded a source sequence into one fixed-dimensional vector and used a second recurrent network to decode the target sequence.^[1] Bahdanau, Cho, and Bengio instead let the decoder form a different context vector at each output step by weighting all encoder states. They described this as a differentiable soft search over source positions and trained the alignment and translation model jointly.^[2] This mechanism was introduced for machine translation, but the same idea soon appeared in image captioning and speech recognition.^[3][4][5]

At an abstract level, an attention layer receives:

• one or more queries, which specify what information is being requested;
• keys, which are compared with the queries;
• values, which contain the information to be aggregated;
• a scoring function and, usually, a normalization function.

For query $q_i$, keys $k_j$, and values $v_j$, a common form is:

$$e_{ij} = s(q_i, k_j)$$ $$\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{\ell \in A_i}\exp(e_{i\ell})}$$ $$o_i = \sum_{j \in A_i}\alpha_{ij}v_j$$

Here, $s$ is a learned or fixed compatibility score, $A_i$ is the set of positions that query $i$ is allowed to attend to, $\alpha_{ij}$ is an attention weight, and $o_i$ is the output. With softmax normalization, the weights over $A_i$ are nonnegative and sum to one. A mask changes $A_i$ by excluding padding positions, future positions, non-neighbors, or other disallowed connections.

This definition separates three choices that are sometimes conflated:

1. What can communicate? The attention pattern or mask determines the allowed query-key pairs.
2. How is relevance scored? The score may be additive, dot-product, bilinear, or another learned function.
3. What is transmitted? The values need not equal the keys, even when they originate from the same input.

Attention is therefore not synonymous with the Transformer. Recurrent encoder-decoder models used it before the Transformer, and attention modules can be inserted into convolutional, graph, set, and multimodal architectures.

Historical development

Encoder-decoder attention

The 2014 Bahdanau model addressed the fixed-vector bottleneck of recurrent encoder-decoder translation. At decoder step $i$, an alignment model scored the previous decoder state against each annotation produced by a bidirectional encoder. A softmax converted those scores into weights, and their weighted sum formed a step-specific context vector.^[2] The alignment model used a small feed-forward network, so this construction is commonly called additive attention or Bahdanau attention.

Luong, Pham, and Manning later compared global attention, which considers all source positions, with local attention, which restricts attention to a window. They also studied dot-product, bilinear, and concatenation-based score functions.^[3] These papers established that "attention" denotes a family of alignment and aggregation choices rather than a single formula.

The distinction between soft and hard attention concerns how elements are selected. Soft attention averages values using continuous weights and is differentiable through the weights. Hard attention samples or chooses discrete locations, which can reduce the number of evaluated locations but generally requires a gradient estimator or another training strategy. Xu and colleagues evaluated both forms for image captioning.^[4] Chorowski and colleagues adapted recurrent attention to speech and added location-aware information to help the alignment progress through long acoustic sequences.^[5]

Transformer attention

The 2017 paper "Attention Is All You Need" introduced an encoder-decoder architecture built from attention and position-wise feed-forward layers, without recurrent or convolutional sequence mixing.^[6] Its central operation is scaled dot-product attention:

$$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)V$$

The rows of $Q$ contain queries, the rows of $K$ contain keys, the rows of $V$ contain values, and $M$ is an optional mask. The scale factor $1/\sqrt{d_k}$ limits the typical magnitude of dot products as key dimension $d_k$ grows. Without that scaling, large logits can place softmax in regions with small gradients.^[6]

For an input matrix $X$, a self-attention layer usually creates the three matrices with learned projections:

$$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V$$

Transformer attention made it possible to compute representations for all sequence positions in parallel during training. It also reduced the maximum path length between two positions to a constant number of attention operations, although a full attention layer compares all pairs and therefore has quadratic score-matrix size in sequence length.^[6]

Main forms

Self-attention and cross-attention

In self-attention, queries, keys, and values are produced from the same sequence or set. Each output position can therefore combine information from other positions in that input. In cross-attention, queries come from one representation while keys and values come from another. Encoder-decoder translation uses decoder states as queries and encoder states as keys and values. The same arrangement can condition an image representation on text or connect another pair of modalities.

These terms describe the source of $Q$, $K$, and $V$, not the score function. Either self-attention or cross-attention may be single-head or multi-head, dense or sparse, causal or noncausal.

Causal, bidirectional, and padding masks

A causal mask permits position $i$ to attend only to positions at or before $i$. In a score matrix, disallowed entries are assigned a value that becomes zero after softmax, commonly implemented with negative infinity before normalization. Causal self-attention is used for autoregressive language models.

An unmasked encoder can attend in both sequence directions. For example, BERT uses bidirectional Transformer encoders and a masked-language-model objective. The word "bidirectional" here describes the allowed attention pattern, not a separate scoring rule.

Padding masks exclude placeholder positions introduced when unequal-length examples are batched. Other masks can enforce local windows, graph neighborhoods, block structure, or application-specific constraints. Correct masking is part of the mathematical definition of a layer, not merely an implementation detail.

Multi-head attention

Multi-head self-attention applies several learned query, key, and value projections in parallel:

$$\operatorname{head}_r = \operatorname{Attention}(QW_r^Q,KW_r^K,VW_r^V)$$ $$\operatorname{MHA}(Q,K,V) = \operatorname{Concat}(\operatorname{head}_1,\ldots,\operatorname{head}_h)W^O$$

The original Transformer divided the representation across heads, concatenated their outputs, and applied an output projection.^[6] Multiple heads allow distinct projected interactions to be represented in one layer. A head should not, however, be assumed to correspond to one stable linguistic or semantic concept. Head behavior depends on the model, layer, input, training run, and analytical method.

Additive and multiplicative scores

Two common compatibility functions are:

$$s_{\mathrm{add}}(q,k)=v_a^\top\tanh(W_q q+W_k k)$$ $$s_{\mathrm{dot}}(q,k)=q^\top k$$

Additive attention uses a learned feed-forward alignment model. Multiplicative attention uses a dot product or a bilinear form such as $q^\top Wk$. Additive and dot-product attention can represent different functions, and their practical cost depends on tensor shapes and hardware. Scaled dot-product attention is especially convenient because a batch of pairwise scores can be computed with matrix multiplication.^[6]

Structural properties

Weighted message passing

Attention can be understood as message passing. Each key-value pair offers a message, a query determines the weights, and the output aggregates the selected values. This view extends beyond sequences. Graph Attention Networks restrict a node's attention to graph neighbors and learn unequal coefficients for their messages.^[7] Non-local neural-network blocks compute a weighted sum across positions in image or video feature maps.^[8] Set Transformer uses attention while constructing permutation-invariant models for set-valued inputs.^[9]

Order and position

Self-attention without position-dependent features is permutation equivariant: reordering input rows reorders the outputs in the same way. That property is useful for sets but insufficient when sequence order matters. Transformer models therefore add or incorporate positional encoding. The original Transformer added sinusoidal or learned position representations to the input embeddings.^[6] Later methods place relative position information in the score or modify queries and keys, but those choices are separate from the basic attention operation.

Content-dependent receptive fields

A dense self-attention layer can connect every permitted pair of positions in one step. Unlike a fixed convolution kernel, its weights depend on the current representations. This does not guarantee that a trained model will use distant information effectively. It only means the computation graph permits the connection. Optimization, training data, positional representation, masking, and numerical precision all affect what information is actually used.

Applications

Attention moved from recurrent translation systems into several model families.

• Language and speech: Encoder-decoder attention supports alignment between source and target sequences, while Transformer self-attention supplies contextual token representations. Location-aware attention was developed for speech recognition.^[5]
• Vision: Early visual attention models weighted spatial features during caption generation.^[4] Non-local blocks later applied related aggregation to image and video features.^[8] A Vision Transformer treats image patches as a sequence and processes them with Transformer encoders.^[10]
• Multimodal generation: Latent diffusion models use cross-attention to condition image-generation features on representations such as text or spatial inputs.^[11] This is an example of cross-attention joining two representation streams rather than a special new scoring function.
• Graphs and sets: Masks can restrict communication to graph edges, while unmasked set attention can model pairwise interactions without imposing an input order.^[7][9]
• Biomolecular modeling: AlphaFold 2's Evoformer contains attention-based and non-attention-based updates over multiple-sequence-alignment and residue-pair representations, and its structure module uses invariant point attention.^[12]

These examples use the same broad pattern but differ substantially in tokenization, masks, geometry, objectives, and surrounding architecture. Results from one domain do not by themselves establish that a particular attention design is best in another.

Efficiency and memory

Full attention

For sequence length $n$, a dense self-attention score matrix contains $n^2$ entries. With head dimension $d$, forming scores and applying them to values takes on the order of $n^2d$ arithmetic operations. A straightforward implementation also materializes an $n \times n$ intermediate for each head and batch element. This cost becomes important for long sequences and high-resolution spatial inputs.^[6]

During autoregressive decoding, previously computed keys and values are commonly retained in a KV cache. Caching avoids recomputing projections for earlier tokens, but the stored state grows with sequence length and the number of key-value heads.

Multi-query attention shares one set of keys and values across query heads, reducing the amount of cached and loaded key-value data during incremental decoding.^[13] Grouped-query attention uses an intermediate number of key-value heads and was proposed as a compromise between standard multi-head and multi-query attention.^[14] These variants reduce key-value memory and bandwidth; they do not remove the pairwise query-key work of dense attention.

Sparse and approximate attention

Sparse attention limits the allowed pairs. Longformer combines a sliding local window with selected global positions, making its attention cost scale linearly with sequence length when the window and number of global positions are fixed.^[15] BigBird combines local, global, and random connections and provides theoretical results for that particular sparse pattern.^[16] Sparse designs trade universal direct connectivity for a chosen communication graph, so accuracy and efficiency depend on whether the pattern fits the task.

Linear attention methods rewrite or approximate attention so that key-value aggregation can be performed before combining it with each query. Katharopoulos and colleagues used kernel feature maps and matrix associativity to obtain linear scaling in sequence length for their formulation.^[17] Performer used positive orthogonal random features to approximate softmax attention.^[18] Such methods alter the operator or approximate it, and their quality, stability, and actual speed are empirical questions rather than consequences of asymptotic notation alone.

Exact implementation improvements

Flash Attention computes exact scaled dot-product attention with an IO-aware tiled algorithm. It avoids writing the full attention matrix to high-bandwidth memory by processing blocks in faster on-chip memory and maintaining the softmax normalization across blocks.^[19] Its arithmetic dependence on sequence length remains quadratic for dense attention, but its auxiliary memory use and memory traffic are lower than a straightforward implementation.

This distinction is important:

Scroll sideways for more →

Interpretation and limitations

Attention weights are conditional coefficients

An attention matrix records coefficients used at one layer, for one head and one input. It can reveal which value vectors were weighted strongly in that computation. It does not automatically measure a raw input feature's causal effect on the final prediction.

Jain and Wallace found that attention weights in the NLP models they studied were often weakly related to gradient-based importance measures and that substantially different attention distributions could sometimes produce similar outputs.^[20] Wiegreffe and Pinter argued that whether attention counts as explanation depends on the definition, model, and test, and proposed alternative diagnostics rather than a universal rejection.^[21] Serrano and Smith likewise found that attention magnitudes were not a fail-safe measure of importance under intervention.^[22]

In a deep Transformer, residual connections and repeated mixing further complicate interpretation. Abnar and Zuidema proposed attention rollout and attention flow to account for information propagation across layers and found higher correlations with ablation and gradient measures than raw attention in their experiments.^[23] These methods remain diagnostics, not proofs of causal responsibility.

Identifiability and saturation

Different score vectors can yield similar weighted sums, especially when value vectors are redundant. Softmax can also saturate when logits have large magnitude, leading to very peaked distributions and small derivatives. Scaling, normalization, initialization, precision, and masking therefore affect training behavior.

Quadratic connectivity is not guaranteed recall

Dense attention permits every position to interact with every other permitted position, but permission is not successful use. Long sequences can still challenge retrieval, optimization, and position handling. Conversely, a sparse or approximate layer may be adequate when relevant dependencies follow its communication pattern. Claims about long-context ability should be evaluated on the target distribution and task, not inferred from context-window size or complexity alone.

Attention is one component

Transformer behavior does not arise from attention in isolation. Feed-forward layers, residual paths, normalization, tokenization, positional representations, objectives, data, and decoding all contribute. Replacing or visualizing attention addresses only one part of the system.

Practical checks

When implementing or evaluating an attention layer, several checks prevent common errors:

• Apply padding and causal masks before softmax and verify that masked probabilities are zero.
• Confirm the intended axes for query length, key length, heads, and batch dimensions.
• Use numerically stable softmax and sufficient precision for normalization and accumulation.
• Distinguish training-time parallelism from autoregressive decode latency.
• Report both asymptotic cost and measured wall-clock behavior on the target hardware.
• Treat attention visualizations as hypotheses for further testing, not standalone explanations.
• Compare variants at matched parameter count, training budget, sequence length, and evaluation setting when possible.

References

1. ^Sutskever, I., Vinyals, O., and Le, Q. V. "Sequence to Sequence Learning with Neural Networks." Advances in Neural Information Processing Systems 27, 2014. proceedings.neurips.cc/...97f410bb7eca942-Abstract
2. ^Bahdanau, D., Cho, K., and Bengio, Y. "Neural Machine Translation by Jointly Learning to Align and Translate." ICLR 2015, first posted 2014. arxiv.org/...1409.0473
3. ^Luong, M.-T., Pham, H., and Manning, C. D. "Effective Approaches to Attention-based Neural Machine Translation." EMNLP 2015, pp. 1412-1421. aclanthology.org/D15-1166
4. ^Xu, K., Ba, J., Kiros, R., et al. "Show, Attend and Tell: Neural Image Caption Generation with Visual Attention." ICML 2015, pp. 2048-2057. proceedings.mlr.press/...xuc15
5. ^Chorowski, J. K., Bahdanau, D., Serdyuk, D., Cho, K., and Bengio, Y. "Attention-Based Models for Speech Recognition." Advances in Neural Information Processing Systems 28, 2015. proceedings.neurips.cc/...e9ea8072e3189e2-Abstract
6. ^Vaswani, A., Shazeer, N., Parmar, N., et al. "Attention Is All You Need." Advances in Neural Information Processing Systems 30, 2017. papers.nips.cc/...47dee91fbd053c1c4a845aa-Abstract
7. ^Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. "Graph Attention Networks." ICLR 2018. openreview.net/forum
8. ^Wang, X., Girshick, R., Gupta, A., and He, K. "Non-Local Neural Networks." CVPR 2018, pp. 7794-7803. openaccess.thecvf.com/..._Networks_CVPR_2018_paper
9. ^Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., and Teh, Y. W. "Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks." ICML 2019, pp. 3744-3753. proceedings.mlr.press/...lee19d
10. ^Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. "An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale." ICLR 2021. openreview.net/forum
11. ^Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. "High-Resolution Image Synthesis With Latent Diffusion Models." CVPR 2022, pp. 10684-10695. openaccess.thecvf.com/...on_Models_CVPR_2022_paper
12. ^Jumper, J., Evans, R., Pritzel, A., et al. "Highly accurate protein structure prediction with AlphaFold." Nature 596, 583-589, 2021. nature.com/...s41586-021-03819-2
13. ^Shazeer, N. "Fast Transformer Decoding: One Write-Head is All You Need." 2019. arxiv.org/...1911.02150
14. ^Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S. "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints." EMNLP 2023, pp. 4895-4901. aclanthology.org/2023.emnlp-main.298
15. ^Beltagy, I., Peters, M. E., and Cohan, A. "Longformer: The Long-Document Transformer." 2020. arxiv.org/...2004.05150
16. ^Zaheer, M., Guruganesh, G., Dubey, K. A., et al. "Big Bird: Transformers for Longer Sequences." Advances in Neural Information Processing Systems 33, 2020. proceedings.neurips.cc/...5f31a9a7a361ab9-Abstract
17. ^Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention." ICML 2020, pp. 5156-5165. proceedings.mlr.press/...katharopoulos20a
18. ^Choromanski, K., Likhosherstov, V., Dohan, D., et al. "Rethinking Attention with Performers." ICLR 2021. openreview.net/forum
19. ^Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Re, C. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." Advances in Neural Information Processing Systems 35, 2022. proceedings.neurips.cc/...40d5-Abstract-Conference
20. ^Jain, S., and Wallace, B. C. "Attention is not Explanation." NAACL-HLT 2019, pp. 3543-3556. aclanthology.org/N19-1357
21. ^Wiegreffe, S., and Pinter, Y. "Attention is not not Explanation." EMNLP-IJCNLP 2019, pp. 11-20. aclanthology.org/D19-1002
22. ^Serrano, S., and Smith, N. A. "Is Attention Interpretable?" ACL 2019, pp. 2931-2951. aclanthology.org/P19-1282
23. ^Abnar, S., and Zuidema, W. "Quantifying Attention Flow in Transformers." ACL 2020, pp. 4190-4197. aclanthology.org/2020.acl-main.385