Editing source: Differential Transformer (The two-map subtraction)

Markdown source editor. This section uses footnote, math or table markup that the visual editor cannot safely round-trip yet, so you are editing the Markdown directly. Keep the ## References section and inline markers like [3] intact. Your changes go live as soon as you save. Every edit is recorded in the page history and checked by editors, and edits that change the title, remove a large part of the article or add many links wait for an editor first. Cite a reliable source for each fact you add and keep the References section intact. You are not signed in, so the page history will credit your edit to an anonymous ID. Sign in to be credited by name.

Editing the The two-map subtraction section. The rest of the article stays as it is.Edit the whole article

Edit summary *

Section (Markdown)

### The two-map subtraction

In a standard multi-head attention block, each head computes a single attention map by applying softmax to scaled query-key inner products. The Differential Transformer replaces this with two separate softmax maps per head, subtracted with a learnable scalar coefficient lambda.[^1]

Given an input sequence X with N tokens and model dimension $$d_{\mathrm{model}}$$, each head projects X into queries, keys, and values. The query and key projections are doubled in width and then **split into two halves**: $$Q_1, Q_2$$ (each in $$\mathbb{R}^{N \times d}$$) and $$K_1, K_2$$ (each in $$\mathbb{R}^{N \times d}$$). The value projection $$V$$ remains a single tensor in $$\mathbb{R}^{N \times 2d}$$. The differential attention output for one head is:

$$
\mathrm{DiffAttn}(X) = \left(\mathrm{softmax}\left(\frac{Q_1 K_1^\top}{\sqrt{d}}\right) - \lambda \, \mathrm{softmax}\left(\frac{Q_2 K_2^\top}{\sqrt{d}}\right)\right) V
$$

The first softmax acts like a conventional attention map; the second softmax is interpreted as a "noise" map that captures the common-mode attention distributed over irrelevant positions. The subtraction cancels what the two maps share and amplifies what they disagree on. Because each head consumes twice the query/key budget but the same value budget, the authors recommend using models with a sufficiently large head count so that halving heads has little practical effect.[^1][^3]

Version 7

Explore AI Wiki

Editing source: Differential Transformer (The two-map subtraction)

Add a source