Self-attention

Self-attention is a neural network operation in which each position forms a data-dependent mixture of information from positions in the same input sequence or set. The operation projects every input representation into a query, a key, and a value. Each query is compared with the keys, the resulting scores are normalized, and the output is a weighted sum of the values. Because the weights depend on the current input, the representation of a token, image patch, speech frame, or set element can change according to the other elements processed with it.^[1]

The word "self" identifies the source of the representations. In self-attention, queries, keys, and values originate from the same collection. In cross-attention, queries originate from one collection while keys and values originate from another. Self-attention is one member of the broader attention family and one component of a Transformer. It is not, by itself, a complete Transformer layer or model. Transformer blocks also contain output projections, residual paths, normalization, and feed-forward transformations, while encoder-decoder Transformers additionally contain cross-attention.^[1]

Within-sequence attention appeared before the Transformer under names including "intra-attention" and "self-attention." Cheng, Dong, and Lapata used attention over a single sequence inside a recurrent machine-reading model in 2016, and Lin and colleagues presented a structured self-attentive sentence embedding in 2017.^[3][4] Vaswani and colleagues then defined scaled dot-product self-attention and multi-head attention as the principal token-mixing operations of the 2017 Transformer architecture.^[1] That formulation became widely used in language, vision, speech, set, scientific, and decision-sequence models, although each application supplies its own tokens, masks, positional information, and surrounding architecture.^{[18][19][20][21][22]}

Definition and scope

Let an input contain $n$ elements with representation width $d_{\text{model}}$. Place the representations in a matrix:

$$X \in \mathbb{R}^{n \times d_{\text{model}}}.$$

A standard self-attention head constructs three projections:

$$Q = XW^Q,\qquad K = XW^K,\qquad V = XW^V,$$

with:

$$W^Q \in \mathbb{R}^{d_{\text{model}} \times d_k},\qquad W^K \in \mathbb{R}^{d_{\text{model}} \times d_k},\qquad W^V \in \mathbb{R}^{d_{\text{model}} \times d_v}.$$

The rows of $Q$ are queries, the rows of $K$ are keys, and the rows of $V$ are values. Queries and keys must have the same inner dimension for a dot product, but the value dimension may differ. "Same input" does not mean that queries, keys, and values are identical. Their learned projection matrices are normally different.^[1]

For query row $q_i$ and key row $k_j$, the unnormalized compatibility score is:

$$s_{ij} = \frac{q_i k_j^\top}{\sqrt{d_k}}.$$

An optional mask or additive bias $M$ modifies the scores, and a row-wise softmax produces the attention coefficients:

$$A = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right).$$

The head output is:

$$O = AV.$$

For ordinary self-attention, $QK^\top$ and $A$ have shape $n \times n$, while $O$ has shape $n \times d_v$. Row $i$ of $A$ gives the coefficients used to combine value rows for output position $i$. With an ordinary softmax over permitted keys, those coefficients are nonnegative and sum to one. The output is still a learned representation rather than a probability statement about the input, because the value vectors and later output projection are learned.^[1]

Self-attention does not require text or even a sequence. The input can be a set, a grid flattened into patches, a series of acoustic features, or another collection. Dense self-attention permits every query to compare with every key. A mask can restrict the permitted pairs, so self-attention may also be causal, local, block-sparse, graph-structured, or otherwise constrained. The defining property is that the query, key, and value representations come from the same input collection, not that the connectivity must be dense.

Intuition

For the sentence "The cat sat on the mat because it was tired," a query produced at "it" can assign different scores to keys produced at "cat," "mat," and the other tokens. The output at "it" then combines their value vectors using those scores. A trained model may use such interactions to represent agreement, reference, position, punctuation, copying, or other patterns. The example explains the data flow, but it does not imply that one head must learn pronoun resolution or that its largest weight gives a complete explanation of the model's prediction.^[23][24]

Queries, keys, and values can also be understood as a learned retrieval interface. A query encodes what an output position is seeking, a key participates in the match, and a value carries the information that can be written to the output. Unlike a hard database lookup, softmax attention usually blends several values. Unlike a fixed nearest-neighbor system, the projections and surrounding network are trained jointly by backpropagation.

Self-attention, cross-attention, and general attention

The same scoring formula can be used with different sources for its matrices:

Scroll sideways for more →

These labels do not determine the score function, number of heads, or mask. Self-attention and cross-attention can each use dot products, multiple heads, local masks, or other variants. A causal mask changes which positions a self-attention query may use; it does not turn the operation into cross-attention.^[1]

The 2014 attention model of Bahdanau, Cho, and Bengio is an important precursor, but its decoder queries attend to encoder annotations. Under the source-based definition above, that is encoder-decoder attention rather than self-attention.^[2] The distinction matters when tracing the history of the term and when estimating cost, because self-attention usually has equal query and key sequence lengths while cross-attention need not.

Scaled dot-product computation

The Transformer paper calls its core operator scaled dot-product attention. For a batch and multiple heads, implementations add batch and head axes, but the matrix operations remain the same. The computation can be separated into four stages.^[1]

Scroll sideways for more →

The score matrix is directional. In general, $s_{ij}$ does not equal $s_{ji}$ because position $i$ contributes a query in the first score and a key in the second. Even when the same input matrix creates both, the query and key projections can differ. The normalized matrix is also generally asymmetric because every row has its own softmax denominator.

Why divide by the square root of the key dimension?

Vaswani and colleagues motivate the factor $1/\sqrt{d_k}$ with a variance calculation. Suppose, for that calculation, that the components of a query and key are independent, have mean zero, and have variance one. Their dot product is a sum of $d_k$ products:

$$qk^\top = \sum_{r=1}^{d_k} q_r k_r.$$

Under those assumptions, the sum has variance $d_k$. Dividing by $\sqrt{d_k}$ gives a score with variance near one. Without the division, increasing the head dimension tends to increase score magnitudes. Large magnitude differences can push softmax toward saturated distributions with small gradients for most entries. The argument is a motivation under simplifying assumptions, not a guarantee that trained queries and keys remain independent or unit variance.^[1]

The scale distinguishes the Transformer's operator from an unscaled dot-product score. Additive attention uses a small learned network instead of a dot product. Vaswani and colleagues reported that dot-product attention was faster in their setting because it maps efficiently to matrix multiplication, while scaling addressed the poorer behavior they observed for unscaled dot products at larger key dimensions.^[1] This comparison does not establish one scoring family as universally better across tasks or hardware.

Masks, biases, and numerical behavior

An additive mask commonly uses zero for permitted pairs and negative infinity for forbidden pairs:

$$M_{ij} = \begin{cases} 0, & \text{if key } j \text{ is available to query } i,\\ -\infty, & \text{otherwise.} \end{cases}$$

After softmax, forbidden positions receive coefficient zero. Implementations may substitute a large negative finite value, but the choice must be safe for the tensor's numeric type. A row in which every position is forbidden has no valid softmax distribution and requires explicit handling. Padding masks, causal masks, and structural masks can be combined before normalization.

Not every additive score term is a mask. A finite relative-position bias changes preferences without forbidding a connection. A binary or causal mask defines availability. Both can occupy the term $M$ in compact formulas, but they have different semantics. This distinction is useful when comparing learned relative biases, ALiBi, causal triangles, and sparse connectivity.^[6][8]

Stable implementations compute softmax after subtracting the row maximum. Fused kernels may process the row in blocks while maintaining running maxima and normalization terms. That reordering can produce the same mathematical attention result without materializing the complete score or coefficient matrix in high-bandwidth memory.^[9]

Directionality and masking

Unmasked or bidirectional self-attention

An unmasked sequence encoder allows each non-padding position to use keys on both sides of it. This pattern is often called bidirectional self-attention. BERT, for example, uses Transformer encoders that jointly condition on left and right context and trains with a masked-language-model objective.^[5] "Bidirectional" describes the allowed communication pattern. It is not a different compatibility function.

Padding remains masked even in a bidirectional encoder. If examples of different lengths share a batch, placeholder positions should neither contribute values nor receive meaningful outputs. Other application-specific positions may also be excluded.

Causal self-attention

Causal self-attention permits output position $i$ to use only positions at or before $i$. With positions numbered from zero:

$$M_{ij} = \begin{cases} 0, & j \leq i,\\ -\infty, & j > i. \end{cases}$$

The permitted score matrix is lower triangular. During teacher-forced training, the model can process all sequence positions in parallel while the mask prevents a position from reading tokens that occur later in the target sequence. The decoder of the original Transformer uses this pattern so that its prediction at a position depends only on earlier outputs.^[1]

During incremental generation, future keys do not yet exist. A decoder computes a query for the new position and attends to stored keys and values from the prefix. A causal mask is still relevant during prompt processing, batched decoding, and any computation that contains more than one newly processed position.

Local and structural masks

A self-attention mask can encode more than direction:

• A sliding-window mask permits nearby positions and excludes more distant ones.
• A block mask permits communication within or between selected chunks.
• A graph mask permits a node to attend only to specified neighbors.
• A global-token pattern gives selected positions dense connectivity while most positions remain local.

These patterns remain self-attention when all participating representations come from the same input. They change the communication graph and can reduce arithmetic if the implementation avoids work for masked pairs. Merely writing negative infinity into a dense matrix after computing every score does not realize the asymptotic saving of a sparse algorithm.^[12][13][14]

Multi-head self-attention

Multi-Head Self-Attention runs several projected attention operations in parallel. For head $r$:

$$\operatorname{head}_r = \operatorname{Attention}\left(XW_r^Q,\ XW_r^K,\ XW_r^V\right).$$

The head outputs are concatenated and projected:

$$\operatorname{MultiHead}(X) = \operatorname{Concat}\left( \operatorname{head}_1,\ldots,\operatorname{head}_h \right)W^O.$$

If every head has value width $d_v$, then $W^O$ maps from $h d_v$ to the block's output width. In the original Transformer, $d_{\text{model}}=512$, $h=8$, and $d_k=d_v=64$. The total projected width across the heads therefore equaled the model width.^[1]

Using several heads creates several independently projected score matrices and value mixtures before the final projection. This gives one layer more than one representational subspace in which to form interactions. It does not guarantee that each head learns a stable human-readable relation. Clark and colleagues found recurring positional, delimiter, and linguistic patterns in BERT attention, but also found heads with broad or heterogeneous behavior.^[23] Michel, Levy, and Neubig found in their evaluated models that many heads could be removed at test time with limited effect, while some layers and tasks depended on particular heads.^[28] These are empirical observations about studied models, not universal laws about every multi-head network.

Key-value head sharing

Standard multi-head attention gives every query head its own key and value head. Two decoding-oriented variants reduce the number of key-value heads:

Scroll sideways for more →

Shazeer proposed multi-query attention to reduce the amount of key-value data loaded during incremental decoding.^[10] Ainslie and colleagues defined grouped-query attention as the intermediate case and evaluated conversion of multi-head checkpoints through additional training.^[11] If head dimensions remain fixed, key-value cache storage scales with the number of key-value heads, so sharing can reduce cache size and memory traffic. These variants do not by themselves remove the dependence of dense attention on the number of query-key position pairs.

Claims about a universal speedup or quality loss are not justified by the head counts alone. Runtime depends on batch size, sequence length, kernel, device, parallelism, precision, and the rest of the model. Quality depends on training and evaluation conditions. The structural definitions above remain valid across those settings.

Order and positional information

Dense, unmasked self-attention without position-dependent inputs is permutation equivariant. If a permutation matrix $P$ reorders the input rows, then:

$$Q' = PQ,\qquad K' = PK,\qquad V' = PV.$$

The score matrix becomes:

$$Q'K'^\top = P(QK^\top)P^\top,$$

and row-wise softmax preserves the corresponding permutation. The output is therefore:

$$\operatorname{Attention}(PX) = P\operatorname{Attention}(X).$$

The outputs follow the reordered inputs, but the operation alone does not assign a special meaning to "first," "next," or a particular distance. This property is useful for set processing.^[20] It is insufficient for an unmasked sequence model that must distinguish order.

The qualification about masks matters. A causal or other index-dependent mask already supplies an ordered communication structure, so causal self-attention is not equivariant to an arbitrary reordering that leaves the mask fixed. Position representations add richer absolute, relative, geometric, or application-specific information.

Main position mechanisms

Several position mechanisms have been paired with self-attention:

Scroll sideways for more →

The original Transformer added position encodings to token embeddings before the encoder and decoder stacks. Its fixed version used:

$$\operatorname{PE}(pos,2i) = \sin\left(pos / 10000^{2i/d_{\text{model}}}\right),$$ $$\operatorname{PE}(pos,2i+1) = \cos\left(pos / 10000^{2i/d_{\text{model}}}\right).$$

The authors also tested learned position embeddings and reported similar results in their translation setting.^[1] Learned absolute position tables have shape determined by the supported position count and model width. They are not the same size as the token-vocabulary embedding table unless those unrelated dimensions happen to match.

Shaw, Uszkoreit, and Vaswani incorporated learned relative-position representations into self-attention, including a clipped representation of query-key distance. They also described the method as a case of relation-aware self-attention that can extend beyond linear sequences.^[6] RoPE rotates pairs of query and key coordinates by position-dependent angles. Under its construction, the inner product between a query at one position and a key at another carries relative-offset information.^[7] ALiBi adds a fixed, head-specific linear distance penalty to the score rather than adding a position embedding to the input. Its reported length-extrapolation results apply to the models and training setup evaluated in that paper.^[8]

No position scheme guarantees reliable extrapolation to every length or task. A model can accept a longer tensor while failing to use all positions effectively. Position method, training distribution, mask, optimization, and evaluation task all affect length generalization.^[8][27]

Computational cost

For one dense self-attention head with sequence length $n$, forming $QK^\top$ costs on the order of $n^2d_k$ arithmetic operations, and multiplying the coefficients by $V$ costs on the order of $n^2d_v$. A straightforward implementation stores an $n \times n$ score or coefficient matrix. With multiple heads whose total projected width stays fixed, the pairwise arithmetic is commonly summarized as:

$$O\left(n^2 d_{\text{model}}\right).$$

Projection layers add costs that scale linearly with $n$ and quadratically with representation width. Which term dominates depends on sequence length, width, batch size, kernel, and hardware. Saying that self-attention is "quadratic" refers to its dense pairwise dependence on sequence length, not to every operation in the Transformer block.^[1]

The number of entries in one dense score matrix is $n^2$. A length of 2,048 gives 4,194,304 entries per head and batch element. A length of 131,072 gives 17,179,869,184 entries. These counts explain why a naive materialization becomes impractical, but memory consumption also depends on data type, saved training activations, head count, batch size, and whether the implementation materializes the matrix at all.

Parallelism and path length

Within one layer, self-attention can compute all training positions with matrix operations. A recurrent network must ordinarily advance its recurrent state across positions in sequence. The original Transformer paper summarized the number of sequential operations per self-attention layer as constant in sequence length and the maximum path between positions as constant, while dense arithmetic remained quadratic.^[1]

This comparison concerns the computation graph of a layer. It does not mean a complete network executes in one step: Transformer layers remain sequential with respect to one another, and autoregressive decoding remains sequential across generated tokens. Nor does a short graph path guarantee that a trained model will retrieve distant information.

Autoregressive decoding and the key-value cache

In a causal decoder, earlier hidden states at a given layer do not change when one new token is appended. Recomputing their key and value projections at every generation step would repeat work. A KV Cache stores the keys and values already produced at each decoder layer. The next token produces a new query, key, and value; its query attends to the cached prefix plus the new key.

For a fixed model and batch, the attention work for one new token grows linearly with the cached prefix length. Across a generated sequence, the cumulative dense attention work grows quadratically with generated length. Cache storage also grows linearly with the number of retained positions, layers, key-value heads, and head width. The cache avoids recomputation, but it introduces a growing memory footprint and requires reading prior keys and values during decoding.

Multi-query and grouped-query attention reduce the number of stored key-value heads.^[10][11] A local or sliding-window mask can cap the number of retained positions used by each query. Quantization, eviction, paging, and prefix-sharing systems can change storage or allocation. Those are cache-management or representation strategies, not definitions of self-attention itself.

Training and decoding therefore stress hardware differently. Training over a full sequence exposes large matrix multiplications and activation storage. Single-token decoding often uses smaller matrix operations and streams a growing cache. A method that improves one regime need not improve the other by the same amount.

Efficiency methods

Methods described as "efficient attention" act on different parts of the computation. Combining them under one complexity label can be misleading.

Scroll sideways for more →

Exact tiled attention

Flash Attention is an exact, IO-aware algorithm for scaled dot-product attention. It divides the computation into tiles that fit in faster on-chip memory and maintains softmax normalization across blocks. The method reduces reads and writes between GPU high-bandwidth memory and on-chip SRAM and avoids storing the complete attention matrix in high-bandwidth memory.^[9]

FlashAttention does not make dense attention's pairwise arithmetic linear in sequence length. Its benefit comes from memory traffic, tiling, fusion, and reduced intermediate storage. The paper also describes an approximate block-sparse extension, but that extension should not be confused with the exact dense algorithm.^[9]

Sparse attention

Sparse Attention computes only a selected subset of query-key pairs. The Sparse Transformer factorized dense connectivity into strided and fixed patterns, producing paper-specific subquadratic patterns.^[12] Longformer combined a sliding local window with selected global positions; with fixed window and global counts, its attention cost scales linearly with sequence length.^[13] BigBird combined local, global, and random connections and proved expressivity results for that specific sparse construction.^[14]

Sparse methods replace a complete interaction graph with a chosen graph. Their asymptotic saving is real only if the algorithm and kernel skip absent pairs. Their accuracy depends on whether relevant information can travel through the selected pattern and enough layers. A proof for one pattern does not automatically apply to another pattern or to a particular trained model.

Linear and kernel attention

Linear Attention methods use matrix associativity after changing or approximating the attention kernel. A generic feature-map form is:

$$\operatorname{LinearAttention}(Q,K,V) = \phi(Q)\left(\phi(K)^\top V\right).$$

Computing $\phi(K)^\top V$ first avoids an explicit $n \times n$ matrix. Katharopoulos and colleagues gave a normalized kernel formulation and showed that its causal form can be updated recurrently.^[15] This operator is not identical to ordinary softmax attention unless the chosen feature representation reproduces the softmax kernel.

Performer uses positive orthogonal random features, called FAVOR+, to approximate softmax attention with linear space and time dependence on sequence length for a fixed number of features.^[16] It is inaccurate to call the approximation "exact softmax attention." Approximation error, feature count, numerical behavior, kernel implementation, and hardware determine the practical tradeoff.

Distributed exact attention

Ring Attention partitions a long sequence across devices. Each device keeps a block of queries while key-value blocks circulate around a ring, allowing the device to accumulate exact blockwise attention results. The ICLR 2024 work overlaps this communication with local block computation and also distributes feed-forward work.^[17]

Ring execution reduces per-device sequence storage and enables longer aggregate sequences. It does not reduce the total pairwise arithmetic of dense attention in the same way as a sparse or kernel method. Its behavior depends on device memory, interconnect bandwidth, block size, and communication overlap.

Role inside Transformer blocks

The original Transformer encoder layer contains a multi-head self-attention sublayer and a position-wise feed-forward sublayer, with residual connections and layer normalization around the sublayers. Its decoder layer adds a masked self-attention sublayer and an encoder-decoder attention sublayer before its feed-forward transformation.^[1] Later architectures change normalization order, activation, gating, position method, head sharing, and other details, but those changes are separate from the basic definition of self-attention.

Encoder self-attention is often unmasked except for padding or structural restrictions. Decoder self-attention is often causal. Encoder-decoder attention is cross-attention. An encoder-only model, a decoder-only model, and an encoder-decoder model can therefore use the same scaled dot-product operator with different representation sources and masks.

Self-attention supplies contextual mixing, but the surrounding block remains essential. The value and output projections determine what is written, residual paths preserve and combine streams, feed-forward layers transform each position, normalization affects optimization, and positional mechanisms supply order or geometry. Attributing a model's full capability to its attention matrices omits these components.

Applications

Self-attention can operate on any collection that a model represents as elements with compatible feature widths.

Language

Language Transformers treat subword, character, or other linguistic units as sequence elements. Bidirectional encoders can combine left and right context, while causal decoders support left-to-right generation. BERT is a documented example of the first pattern.^[5] The original Transformer is an encoder-decoder example with bidirectional encoder self-attention, causal decoder self-attention, and cross-attention between them.^[1] Self-attention is also used for tasks beyond generation, including classification, retrieval representations, tagging, and sequence-to-sequence prediction, depending on the model and objective.

Vision

A Vision Transformer divides an image into patches, maps the patches to vectors, adds position information, and processes the resulting sequence with a Transformer encoder. Dosovitskiy and colleagues evaluated this design for image classification.^[18] The patch grid makes the positional scheme and the quadratic dependence on patch count important. Later local or hierarchical vision designs alter the attention pattern, but the original ViT evidence does not by itself establish a universal advantage over convolution.

Speech

wav2vec 2.0 first encodes raw audio through a multilayer convolutional feature encoder, masks spans of the latent sequence during pretraining, and feeds the latent representations to a Transformer context network.^[19] Self-attention therefore operates on learned latent speech steps, not directly on mel-spectrogram patches as the baseline article previously stated. Speech systems vary in their front ends, masks, streaming constraints, and position methods.

Sets and scientific structures

Set Transformer uses attention modules to model interactions among elements of a set and constructs permutation-invariant outputs. It also proposes inducing-point attention to reduce the cost of processing large sets.^[20] In this setting, the lack of an inherent input order is useful rather than a defect.

AlphaFold 2's Evoformer combines attention-based and non-attention-based updates over multiple-sequence-alignment and residue-pair representations, while its structure module uses invariant point attention.^[21] These geometry-aware and pair-biased operations are specialized uses of attention inside a larger scientific architecture. They should not be reduced to ordinary text self-attention.

Decision sequences

Decision Transformer casts offline reinforcement learning as conditional sequence modeling over returns, states, and actions. Its causal Transformer predicts actions from earlier trajectory information and a desired return.^[22] This is evidence that causal self-attention can be applied to decision trajectories. It is not evidence that every reinforcement-learning problem is best solved as sequence modeling.

Attention heads and interpretation

An attention matrix is observable, but its coefficients have a narrow meaning: they are the normalized weights used to combine value vectors for one head, layer, input, and forward pass. They do not include the content of the value vectors, the output projection, residual additions, feed-forward transformations, or later layers. A high coefficient can multiply a value that contributes little in a relevant output direction, while information can also pass through residual paths.

Clark and colleagues analyzed BERT and found that some heads followed simple positional patterns, some attended strongly to delimiter tokens, and some aligned with linguistic relations more often than chance.^[23] The same study also illustrates why a clean label for every head is unwarranted. Patterns vary across heads, layers, inputs, and analytical criteria.

The explanation debate

Jain and Wallace compared learned attention weights with gradient-based importance measures in the NLP models they studied. They found frequent weak correlation and constructed substantially different attention distributions that could yield similar predictions.^[24] Their result challenges the practice of treating a raw attention map as a faithful feature-importance explanation.

Wiegreffe and Pinter argued that the conclusion depends on the definition of explanation and on tests that account for the whole model. They proposed diagnostic baselines, comparisons across random seeds, frozen-weight tests, and adversarial training as more informative evaluations.^[25] The two papers do not support either blanket statement that attention always explains a prediction or that attention can never provide useful evidence.

Abnar and Zuidema noted that repeated attention and residual mixing make raw weights at one layer especially incomplete. They proposed attention rollout and attention flow to combine information across layers, and reported higher correlations with ablation and gradient measures than raw attention in their experiments.^[26] These techniques remain post hoc diagnostics rather than proofs of causal responsibility.

Head specialization and redundancy

Multi-head attention permits heads to learn different projected interactions, but specialization is an empirical outcome rather than part of the definition. Michel and colleagues ablated heads in trained translation and BERT models. Many could be removed at test time with limited measured degradation, while particular layers and tasks were sensitive to selected heads.^[28] Head count, apparent pattern, and causal importance are therefore different questions.

For interpretability, an attention visualization is best treated as a hypothesis generator. Stronger analysis can combine attention with ablation, activation patching, gradient or attribution methods, controlled counterfactuals, and evaluation across inputs. The appropriate method depends on whether the question concerns routing, association, prediction sensitivity, or a causal mechanism.

Limitations

Dense quadratic scaling

Full self-attention forms interactions for all permitted pairs, giving quadratic arithmetic in sequence length and a quadratic intermediate in a straightforward implementation. Exact tiling can reduce intermediate storage and memory traffic, and distributed execution can spread the work, but neither changes the total dense pair count.^[9][17] Sparse and kernel methods change connectivity or the operator and must be evaluated for the target task.^{[12][13][14][15][16]}

Connectivity does not guarantee use

A dense layer gives every permitted position a direct computational path to every other position. That capability does not guarantee that training produces reliable retrieval from every location. Liu and colleagues evaluated language models on multi-document question answering and key-value retrieval and found that performance in their experiments often fell when relevant information appeared in the middle of a long context.^[27] The result is about the evaluated models and tasks, but it demonstrates why advertised context length and dense connectivity are not sufficient measures of effective context use.

Position and extrapolation

Unmasked self-attention needs position-dependent information when order matters. Causal masks supply direction but do not resolve every question of absolute distance, relative distance, or extrapolation. Position methods impose different inductive biases, and good performance inside the training range does not prove behavior outside it.^[6][7][8]

Optimization and numerical sensitivity

Dot-product scale, mask values, normalization precision, initialization, and residual architecture influence optimization. Very peaked softmax distributions can have small gradients for most entries. Lower precision and fused kernels require care with accumulation, masking, and overflow, even when they implement the same mathematical operator.^[1][9]

Formal results have assumptions

Hahn proved limitations for classes of fixed-depth self-attention models on selected formal languages under the paper's stated assumptions. For soft attention, the results include smoothness and boundedness conditions; the conclusions also distinguish whether depth, head count, or parameter magnitudes can grow with input length.^[29] These theorems are relevant to expressivity, but they do not imply that every finite practical language task is unsolvable by a Transformer or that empirical success contradicts the proof.

Attention is only one part of a model

Tokenization, embeddings, positional information, feed-forward layers, residual connections, normalization, objectives, data, and decoding all shape model behavior. Comparing "attention" with another architecture requires specifying which of those components and training conditions are held fixed. Results from a paper-specific benchmark should not be generalized into a claim that one operator is universally superior.

Historical development

Neural attention did not begin as self-attention. Bahdanau, Cho, and Bengio proposed a jointly trained soft alignment in neural machine translation in work first posted in 2014 and presented at ICLR 2015. A decoder state scored encoder annotations and formed a step-specific context vector.^[2] Since the queries and key-value representations came from different parts of the encoder-decoder model, this was cross-attention in later terminology.

Attention within one sequence developed in recurrent and sentence-representation systems. Cheng, Dong, and Lapata's 2016 machine reader used a memory network and attention within the processed text to induce relations among tokens.^[3] Lin and colleagues' ICLR 2017 model explicitly used "self-attentive" in its title and learned several attention-weighted views of bidirectional LSTM hidden states to form a matrix sentence representation.^[4] These systems used within-sequence attention but were not Transformer blocks.

The 2017 Transformer paper used the term self-attention, also noting the name intra-attention, for an operation relating positions within one sequence. It combined scaled dot-product attention, multiple heads, masks, position encodings, feed-forward sublayers, residual connections, and normalization in an architecture without recurrent or convolutional sequence mixing.^[1] The paper did not invent every antecedent attention idea, but it established the formulation that is now commonly meant by Transformer self-attention.

Later work changed different layers of the design:

Scroll sideways for more →

The table is not a claim that one line superseded the previous line. These works address different concerns: representation source, positional structure, connectivity, approximation, cache bandwidth, memory traffic, or device distribution.

Common distinctions

Scroll sideways for more →

Implementation and evaluation checks

Several checks catch common self-attention errors:

• Verify the intended batch, head, query-length, key-length, and feature axes before multiplying tensors.
• Apply causal, padding, and structural restrictions before softmax, and test that forbidden coefficients are zero.
• Handle rows with no permitted keys explicitly.
• Distinguish a finite score bias from a hard mask.
• Use stable softmax and appropriate accumulation precision.
• Confirm whether an implementation materializes the attention matrix or uses an online tiled normalization.
• Separate full-sequence training throughput, prompt processing, and single-token decoding when reporting speed.
• Report sequence length, batch size, head dimensions, data type, hardware, kernel, and whether a KV cache is used.
• Compare sparse or approximate methods at matched model size, training budget, and evaluation setting.
• Treat attention maps as intermediate coefficients and test interpretive claims with interventions or other diagnostics.

References

1. ^Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). "Attention Is All You Need." Advances in Neural Information Processing Systems 30. proceedings.neurips.cc/...bd053c1c4a845aa-Abstract
2. ^Bahdanau, D., Cho, K., and Bengio, Y. (2015). "Neural Machine Translation by Jointly Learning to Align and Translate." ICLR 2015, first posted 2014. arxiv.org/...1409.0473
3. ^Cheng, J., Dong, L., and Lapata, M. (2016). "Long Short-Term Memory-Networks for Machine Reading." Proceedings of EMNLP, 551-561. aclanthology.org/D16-1053
4. ^Lin, Z., Feng, M., dos Santos, C. N., et al. (2017). "A Structured Self-Attentive Sentence Embedding." ICLR 2017. openreview.net/forum
5. ^Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." Proceedings of NAACL-HLT, 4171-4186. aclanthology.org/N19-1423
6. ^Shaw, P., Uszkoreit, J., and Vaswani, A. (2018). "Self-Attention with Relative Position Representations." Proceedings of NAACL-HLT, 464-468. aclanthology.org/N18-2074
7. ^Su, J., Lu, Y., Pan, S., et al. (2024). "RoFormer: Enhanced Transformer with Rotary Position Embedding." Neurocomputing 568, 127063. First posted 2021. arxiv.org/...2104.09864
8. ^Press, O., Smith, N. A., and Lewis, M. (2022). "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation." ICLR 2022. openreview.net/forum
9. ^Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Re, C. (2022). "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." Advances in Neural Information Processing Systems 35. proceedings.neurips.cc/...40d5-Abstract-Conference
10. ^Shazeer, N. (2019). "Fast Transformer Decoding: One Write-Head Is All You Need." arXiv:1911.02150. arxiv.org/...1911.02150
11. ^Ainslie, J., Lee-Thorp, J., de Jong, M., et al. (2023). "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints." Proceedings of EMNLP, 4895-4901. aclanthology.org/2023.emnlp-main.298
12. ^Child, R., Gray, S., Radford, A., and Sutskever, I. (2019). "Generating Long Sequences with Sparse Transformers." arXiv:1904.10509. arxiv.org/...1904.10509
13. ^Beltagy, I., Peters, M. E., and Cohan, A. (2020). "Longformer: The Long-Document Transformer." arXiv:2004.05150. arxiv.org/...2004.05150
14. ^Zaheer, M., Guruganesh, G., Dubey, A., et al. (2020). "Big Bird: Transformers for Longer Sequences." Advances in Neural Information Processing Systems 33. proceedings.neurips.cc/...5f31a9a7a361ab9-Abstract
15. ^Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. (2020). "Transformers Are RNNs: Fast Autoregressive Transformers with Linear Attention." Proceedings of ICML, PMLR 119:5156-5165. proceedings.mlr.press/...katharopoulos20a
16. ^Choromanski, K., Likhosherstov, V., Dohan, D., et al. (2021). "Rethinking Attention with Performers." ICLR 2021. openreview.net/forum
17. ^Liu, H., Zaharia, M., and Abbeel, P. (2024). "RingAttention with Blockwise Transformers for Near-Infinite Context." ICLR 2024. proceedings.iclr.cc/...68c4935-Abstract-Conference
18. ^Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2021). "An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale." ICLR 2021. openreview.net/forum
19. ^Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. (2020). "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations." Advances in Neural Information Processing Systems 33. proceedings.neurips.cc/...3227870bb6d7f07-Abstract
20. ^Lee, J., Lee, Y., Kim, J., et al. (2019). "Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks." Proceedings of ICML, PMLR 97:3744-3753. proceedings.mlr.press/...lee19d
21. ^Jumper, J., Evans, R., Pritzel, A., et al. (2021). "Highly Accurate Protein Structure Prediction with AlphaFold." Nature 596, 583-589. nature.com/...s41586-021-03819-2
22. ^Chen, L., Lu, K., Rajeswaran, A., et al. (2021). "Decision Transformer: Reinforcement Learning via Sequence Modeling." Advances in Neural Information Processing Systems 34. proceedings.neurips.cc/...72b5c31057f0663-Abstract
23. ^Clark, K., Khandelwal, U., Levy, O., and Manning, C. D. (2019). "What Does BERT Look At? An Analysis of BERT's Attention." Proceedings of BlackboxNLP, 276-286. aclanthology.org/W19-4828
24. ^Jain, S., and Wallace, B. C. (2019). "Attention Is Not Explanation." Proceedings of NAACL-HLT, 3543-3556. aclanthology.org/N19-1357
25. ^Wiegreffe, S., and Pinter, Y. (2019). "Attention Is Not Not Explanation." Proceedings of EMNLP-IJCNLP, 11-20. aclanthology.org/D19-1002
26. ^Abnar, S., and Zuidema, W. (2020). "Quantifying Attention Flow in Transformers." Proceedings of ACL, 4190-4197. aclanthology.org/2020.acl-main.385
27. ^Liu, N. F., Lin, K., Hewitt, J., et al. (2024). "Lost in the Middle: How Language Models Use Long Contexts." Transactions of the Association for Computational Linguistics 12, 157-173. aclanthology.org/2024.tacl-1.9
28. ^Michel, P., Levy, O., and Neubig, G. (2019). "Are Sixteen Heads Really Better than One?" Advances in Neural Information Processing Systems 32. proceedings.neurips.cc/...282670cdd54f69f-Abstract
29. ^Hahn, M. (2020). "Theoretical Limitations of Self-Attention in Neural Sequence Models." Transactions of the Association for Computational Linguistics 8, 156-171. aclanthology.org/2020.tacl-1.11