Vision Transformer

The Vision Transformer (ViT) is a deep learning architecture that represents an image as a sequence of fixed-size patches and processes that sequence with a Transformer encoder. Alexey Dosovitskiy and colleagues introduced the named architecture in the 2020 paper "An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale," which was published at ICLR 2021. Their experiments showed that a largely standard Transformer, with comparatively little image-specific machinery, could compete with strong convolutional models after supervised pre-training on sufficiently large image collections ^[1].

ViT is both a specific model family and the prototype for a broader class of patch-based vision transformers. In the original design, an image is divided into non-overlapping square patches, each patch is linearly projected to an embedding, learned position embeddings are added, and a special classification token is prepended. A stack of self-attention and feed-forward blocks then produces a representation used for classification ^[1]. Later models changed the training recipe, tokenization, attention pattern, feature hierarchy, pre-training objective, or output head while retaining this basic image-as-tokens formulation.

The architecture helped make Transformers a general tool in computer vision, but it did not make convolution obsolete. ViT and convolutional networks impose different structural assumptions, and comparisons depend on model size, data, training recipe, input resolution, task, and compute budget. ViT's lasting contribution is therefore narrower and more precise than a claim that one architecture universally replaced another: it established patch sequences plus Transformer encoders as a practical, scalable foundation for visual representation learning ^{[1][5][11][33]}.

Definition and scope

The term "Vision Transformer" is used in two related ways. With capitalization and the abbreviation ViT, it commonly denotes the architecture and model configurations described by Dosovitskiy et al. With a lowercase description, "vision transformer" can refer more broadly to models that use Transformer components for images or video. The broader category includes systems that differ substantially from original ViT, including hierarchical models, window-attention models, masked image encoders, vision-language encoders, detection transformers, and diffusion transformers ^{[1][11][14][20][23][27]}.

This distinction matters because not every visual model containing attention is a ViT. A convolutional network with one non-local block is not ordinarily called a Vision Transformer. Likewise, DETR uses a Transformer encoder-decoder for set prediction but used a convolutional image backbone in its original form. Swin Transformer is a vision transformer, but its hierarchy and local shifted windows differ from the single-scale, global-attention design of plain ViT. Diffusion Transformer borrows ViT-style patch processing for denoising latent representations rather than for image classification ^[11][23][27].

Original ViT was designed for image classification. Its transferable encoder later became useful in dense prediction, retrieval, multimodal learning, video analysis, and generative modeling, usually after adding task-specific components. Calling ViT a "backbone" means that its token representations are reused by another head or decoder. It does not imply that an unmodified classification model directly performs every downstream task ^{[1][20][24][25][28][29]}.

Historical context

Before ViT, convolutional neural networks were the standard trainable architecture for large-scale visual recognition. Convolution gives a model strong spatial structure: a kernel is shared across positions, nearby pixels are processed together, and deeper layers gradually enlarge the receptive field. Those properties support parameter sharing and translation equivariance, although padding, stride, pooling, and boundary effects can prevent exact translation equivariance in an implemented network ^[1].

The Transformer was introduced for sequence transduction in 2017. Its central operation, multi-head self-attention, lets each token form a content-dependent weighted combination of other tokens. The original encoder alternates self-attention with position-wise feed-forward networks, residual connections, and normalization ^[2]. Because attention does not by itself encode token order, sequence models add positional information.

Researchers applied attention to visual data before ViT. Image Transformer used local self-attention for autoregressive image generation and super-resolution ^[3]. Non-local neural networks inserted operations that aggregate information over distant positions into convolutional systems for video and image tasks ^[4]. Other work combined attention with convolution or used Transformers over feature maps. These predecessors established that attention could model visual relationships, but most retained convolutional processing, targeted generation, or used task-specific tokenization.

Dosovitskiy et al. deliberately tested a simpler question: could a standard Transformer encoder operate directly on a sequence made from raw image patches? The paper was first posted in October 2020 and later published at ICLR 2021. The design used only limited two-dimensional structure in patch extraction and in interpolation of positional embeddings during higher-resolution fine-tuning. The authors compared pure ViT models, convolutional baselines, and hybrid models whose tokens came from a convolutional feature map ^[1].

The experiment's important qualification was data scale. In the authors' study, larger ViT models were less favorable than comparable residual networks when pre-trained on ImageNet-1K, became more competitive on ImageNet-21K, and benefited more clearly from the much larger private JFT-300M dataset. This was evidence about the models and recipes tested, not a universal theorem that every ViT needs hundreds of millions of labeled images. Soon afterward, DeiT showed that a strong recipe and optional distillation could train competitive ViT models on ImageNet-1K alone ^[5]. Later systematic work showed that augmentation, regularization, compute, and model size interact strongly, so "data hunger" cannot be separated from the training protocol ^[6].

Core architecture

Patchification

Let an input image have height $H$, width $W$, and $C$ channels. For a square patch width $P$, original ViT divides the image into non-overlapping $P \times P$ patches. When $H$ and $W$ are divisible by $P$, the number of image patches is

$$N = \frac{H W}{P^2}.$$

Each patch contains $P^2 C$ scalar values. ViT flattens each patch and multiplies it by a learned projection matrix to produce a token of hidden width $D$. If $x_p^i$ is the flattened vector for patch $i$ and $E$ is the projection, the patch token is $x_p^i E$. The original paper calls these outputs patch embeddings ^[1].

For a 224 x 224 RGB image and 16 x 16 patches, there are $14 \times 14 = 196$ image tokens. Each raw patch has $16 \times 16 \times 3 = 768$ values, but the model hidden width does not have to equal 768. The projection maps the flattened patch into whatever width the selected model uses. In ViT-B/16, both happen to be 768, which can obscure that distinction.

Patch extraction plus linear projection can be implemented as a two-dimensional convolution whose kernel and stride both equal the patch size. The outputs are mathematically equivalent when the convolution uses the same learned weights and no overlapping padding. This implementation detail does not turn the encoder blocks into convolutional layers; it is a convenient way to perform the input projection ^[1].

Non-overlapping patchification compresses many pixels into one token before contextual processing begins. A smaller patch preserves finer spatial granularity and creates more tokens. A larger patch reduces sequence length and computation but may discard or entangle fine detail earlier. Patch size is therefore part of the model definition, which is why names such as ViT-B/16 and ViT-B/32 include it ^[1][13].

Classification token

Original ViT prepends one learned vector, usually called the class or CLS token, to the patch sequence. This vector does not correspond to a patch. It participates in every encoder layer and can attend to the patch tokens. After the final layer, its hidden state is treated as the representation of the whole image and sent to the classification head ^[1].

The initial class token is the same learned parameter for every image, but its output becomes image-dependent through attention and residual updates. The design follows the use of a classification token in BERT. It is a readout mechanism, not an explicit object token and not a guarantee that a single attention map explains the prediction ^[1].

Mean pooling is a common alternative: the model averages the final patch tokens and classifies the result. Some later architectures use pooling attention, multiple special tokens, or task-specific queries. These choices alter how spatial information is aggregated, but they do not change the central idea that the encoder maps a patch sequence to contextual token representations ^[1].

Position information

Self-attention without position information is equivariant to a permutation of its input tokens. An image model must therefore expose where patches came from. Original ViT adds a learned one-dimensional positional encoding of shape $(N+1) \times D$ to the projected patches and class token. Although the table is one-dimensional, each index consistently corresponds to a location in the two-dimensional patch grid ^[1].

The original study compared its learned one-dimensional embedding with alternatives that more explicitly represented two-dimensional structure and did not observe a significant improvement from those alternatives in that experimental setting ^[1]. This result should not be generalized to every vision task. Later architectures successfully used relative position biases, conditional encodings, continuous biases, and other position mechanisms, especially for variable resolutions and dense prediction.

A learned absolute table is tied to the token grid seen during training. When fine-tuning at a different resolution while keeping patch size fixed, original ViT reshapes the patch portion of the table to a two-dimensional grid, interpolates it to the new grid size, and then flattens it again. The class-token position is handled separately ^[1]. This procedure lets the encoder accept a longer sequence, but it is an adaptation rather than proof that a checkpoint will extrapolate perfectly to arbitrary resolutions.

Encoder input

Using x_class for the learned class token, x_p[i] for flattened patches, E for the patch projection, and E_pos for position embeddings, the initial sequence can be written as

$$z_0 = [x_class; x_p[1] E; x_p[2] E; ...; x_p[N] E] + E_pos.$$

The sequence has $N+1$ tokens for image classification. Each token has width $D$. Batching adds a leading batch dimension but does not change the model definition. Dropout may be applied after embeddings or within blocks according to the training configuration ^[1].

The patch tokens occupy an embedding space learned jointly with the rest of the network. A token is not a symbolic label for its patch. It is a continuous vector whose coordinates are optimized for the training objective. After several layers, each patch token can contain information drawn from many or all other patches.

Multi-head self-attention

In each attention layer, learned linear maps turn the token matrix $X$ into queries $Q$, keys $K$, and values $V$. Scaled dot-product attention is

$$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_h}}\right)V,$$

where $d_h$ is the query and key width for one head. Multi-head attention applies this operation in parallel subspaces, concatenates the head outputs, and projects them back to hidden width $D$ ^[2].

For plain ViT, attention is global: every token can directly exchange information with every other token in one layer. "Global" describes the connectivity of the operation. It does not mean that learned attention weights are uniform or that each head uses distant context equally. Heads can learn local, global, or mixed patterns ^[1][35].

Multi-head attention makes interactions content-dependent. The weight from token $i$ to token $j$ changes with the image because it depends on their query and key vectors. In contrast, a conventional convolution applies the same learned kernel pattern at each location. This flexibility reduces built-in spatial assumptions but shifts more of the burden to learned data and optimization ^[1][2].

Feed-forward network, normalization, and residual paths

Each original ViT encoder block has two main sublayers: multi-head self-attention and a position-wise multilayer perceptron. The MLP contains two linear layers with a GELU nonlinearity. Its intermediate width is larger than the model hidden width, four times larger in the original Base, Large, and Huge configurations ^[1].

ViT uses pre-normalization. Layer normalization is applied before attention and before the MLP, while residual connections add each sublayer's output back to its input. For layer $\ell$,

$$z'_\ell = \operatorname{MSA}(\operatorname{LN}(z_{\ell-1})) + z_{\ell-1},$$ $$z_\ell = \operatorname{MLP}(\operatorname{LN}(z'_\ell)) + z'_\ell.$$

A final normalization is applied to the class-token state before the head. Residual paths preserve and combine information across depth, while normalization stabilizes the scale of activations. Later vision transformers changed normalization placement, residual scaling, activation functions, or the ordering of attention and MLP computations, so those details should not automatically be attributed to original ViT ^[1][8][12].

Classification head

During the original supervised pre-training experiments, the authors used an MLP head with one hidden layer. For downstream fine-tuning, they replaced the pre-training head with a zero-initialized linear layer whose output width equaled the number of target classes ^[1]. The transferred component was the encoder, not the old class vocabulary.

The head produces logits. A softmax can turn these into normalized class probabilities for a single-label task, and cross-entropy can train the network. Other tasks attach different heads: per-patch decoders for segmentation, feature pyramids and region heads for detection, projection layers for contrastive image-text learning, or denoising outputs for diffusion.

Model notation and original configurations

The naming pattern ViT-{size}/{patch} combines an encoder scale with the patch width in pixels. ViT-B/16 means the Base encoder with 16 x 16 patches. The original paper defined Base, Large, and Huge configurations based partly on BERT's scale conventions ^[1].

Scroll sideways for more →

Parameter totals vary slightly across implementations because the classifier width, representation head, biases, or pooling choice can differ. Patch size has a modest effect on the patch-projection and position-embedding parameters, but its larger effect is computational: it changes the number of tokens processed by every encoder layer.

Names introduced by later projects are not governed by one universal standard. "Tiny," "Small," "giant," and capitalized variants can refer to different depths and widths in different codebases. A model should therefore be identified by a configuration table or checkpoint specification, not by a size letter alone.

Computation and memory

Let $S=N+1$ be the sequence length including the class token. Forming the attention score matrices costs on the order of $S^2D$, and storing them naively requires on the order of $S^2$ values per layer across heads. The token-wise projections and MLP cost on the order of $SD^2$. Which term dominates in measured runtime depends on hidden width, sequence length, batch size, numeric precision, kernel implementation, and hardware.

At fixed patch size, doubling both image dimensions multiplies the number of patch tokens by four. The quadratic attention term then grows by roughly sixteen, while token-wise layers grow by roughly four. This is why high-resolution use can become expensive even when the parameter count stays unchanged.

Patch size creates the same trade-off in the other direction. At 224 x 224 resolution:

Scroll sideways for more →

The table counts image tokens only. A class token adds one. The 14-pixel case divides 224 exactly, but arbitrary combinations of resolution and patch size may require resizing, cropping, or padding.

Smaller patches can improve spatial detail, but they are not guaranteed to improve every task or checkpoint. Training at one patch size and evaluating at another changes both the input projection and the positional grid. FlexiViT addressed this by sampling patch sizes during training and resizing patch and position parameters, producing one set of weights intended to operate across several patch sizes ^[13].

Compute comparisons require care. Floating-point operation counts do not fully predict latency, and token pruning does not guarantee proportional wall-clock speedup if irregular operations are poorly supported. Memory-efficient attention can reduce the materialization cost of attention matrices without changing the mathematical all-pairs operation. Windowing changes the connectivity itself. These are distinct optimization strategies.

Pre-training, fine-tuning, and data

Results in the original study

Original ViT used supervised pre-training on three image collections: ImageNet-1K with about 1.3 million images, ImageNet-21K with about 14 million, and JFT with roughly 303 million images and a large label space. The authors then transferred the encoders to classification benchmarks ^[1].

The paper's best reported ImageNet result was 88.55 percent top-1 for ViT-H/14 pre-trained on JFT-300M and fine-tuned at higher resolution. ViT-L/16 trained on the same source reported 87.76 percent. The comparison table also reported 87.54 percent for a BiT ResNet152x4 trained on JFT and 88.4 or 88.5 percent for Noisy Student EfficientNet-L2. The reported pre-training costs were 2,500, 680, 9,900, and 12,300 TPUv3-core-days, respectively, for ViT-H/14, ViT-L/16, BiT-L, and Noisy Student ^[1].

Scroll sideways for more →

These numbers are historical results under the paper's evaluation protocol. They are not a current leaderboard and should not be combined with later values obtained using different data, resolutions, label sets, or evaluation variants. The comparison supported the authors' claim that ViT could use large-scale pre-training effectively and, in their setup, reach strong transfer accuracy with less reported pre-training compute than the selected convolutional baselines.

What "data hungry" means

Plain ViT has less built-in locality and translation structure than a conventional convolutional network. The original paper found that this weaker image-specific inductive bias hurt the tested ViT models at smaller data scale and became less limiting as pre-training data grew ^[1]. That observation is the source of the common description of ViT as data hungry.

The description has limits. It does not specify a fixed minimum number of images, and it does not mean a ViT cannot be trained on ImageNet-1K. DeiT trained an 86-million-parameter model on ImageNet-1K without external images and reported 83.1 percent top-1. Its distilled version reported 85.2 percent. The paper attributed the result to a carefully engineered recipe and a Transformer-specific distillation design ^[5].

A broader empirical study of more than 50,000 ViT and hybrid training runs examined data size, augmentation, regularization, model size, and compute. It found that augmentation and regularization could substantially improve results on smaller data and that additional compute could partly compensate for less pre-training data ^[6]. The defensible conclusion is conditional: plain ViT benefits strongly from scale, while training recipe and supervision determine how much labeled data is needed for a given target.

Optimization and regularization

Successful ViT training recipes commonly combine stochastic image augmentation, weight decay, learning-rate warmup and decay, dropout or stochastic depth, and label smoothing. The exact combination is not universal. Stronger regularization can help a large model on a limited dataset but can underfit a smaller model or a short schedule. Hyperparameters that work for convolutional networks cannot always be transferred unchanged ^[5][6].

DeiT used AdamW, repeated augmentation, RandAugment, Mixup, CutMix, random erasing, stochastic depth, and label smoothing in its recipe, along with an optional teacher ^[5]. The importance of the work was not that every later ViT must use every component. It demonstrated that the original JFT-scale result was not the only viable path to competitive classification.

Large-scale studies also found that model capacity, data quantity, and compute should be considered jointly. A model too large for its data and regularization can overfit or optimize poorly, while a larger model can become more compute-efficient once the data regime supports it ^[7]. Reporting only parameter count therefore gives an incomplete picture.

Knowledge distillation

Knowledge distillation trains a student using predictions or representations from a teacher. DeiT introduced a learned distillation token alongside the class token. A separate head on that token was trained against a teacher's output, commonly from a convolutional RegNet. At inference, the class and distillation heads could be combined ^[5].

The extra token did not simply copy a static convolutional filter. Through self-attention it interacted with image patches and the class token, while its loss supplied a second supervisory signal. DeiT's experiments found convolutional teachers especially effective for this setup, which the authors related to different inductive biases. Other distillation methods use soft labels, intermediate features, attention maps, or teacher-free self-distillation and should not all be called DeiT-style distillation ^[5].

Transfer to a new resolution or task

Standard fine-tuning replaces the pre-training classifier, adapts positional embeddings when necessary, and updates some or all encoder weights. A linear probe instead freezes the encoder and trains only a shallow classifier. Few-shot evaluation limits the number of labeled target examples. Zero-shot image-text evaluation selects classes through text embeddings rather than fitting a target classifier. These protocols measure different properties and their accuracy values should not be compared as if interchangeable.

Higher-resolution fine-tuning keeps patch width fixed, which increases sequence length. Interpolating position embeddings supplies a compatible parameter shape, but fine-tuning still matters because the distribution and amount of visible detail change. Input normalization, crop policy, class mapping, and label preprocessing must also match the checkpoint or be deliberately adapted.

Architectural families

Plain global-attention ViTs

Plain ViT keeps a constant token-grid resolution and hidden width through the encoder. Every self-attention layer is global, and downsampling happens only at initial patchification. This regular structure is easy to scale and reuse, but dense prediction systems must recover multiscale features or process long high-resolution sequences ^[1][24].

Scaling studies extended this family far beyond the original 632-million-parameter Huge model. A 2022 study trained a two-billion-parameter ViT and analyzed scaling across data, parameters, and compute ^[7]. ViT-22B later used 22 billion parameters with architectural changes for stable and efficient training, including parallel attention and MLP branches, query-key normalization, and omitted biases in selected components ^[8]. These models demonstrate that plain or nearly plain ViT encoders can scale; they do not imply that parameter count alone guarantees better results on every downstream task.

Adding locality or deeper optimization

Several variants reintroduced spatial bias without returning to a fully convolutional backbone. ConViT initialized gated positional self-attention to behave locally and allowed the learned gates to move away from that bias ^[9]. CaiT separated patch self-attention from later class-attention layers and introduced LayerScale, which helped optimize deeper image Transformers ^[10].

These designs show that "Transformer versus convolution" is not a clean binary. Local attention, overlapping patch stems, depthwise convolutions, learned relative biases, and hierarchical token merging can place a model between the endpoints. Whether such a hybrid is preferable depends on the data regime, target task, and hardware ^{[9][10][11][33]}.

Hierarchical and windowed models

Swin Transformer changes both attention scope and feature geometry. It starts from small image patches, computes attention within local windows, alternates regular and shifted window partitions to connect neighboring windows, and merges patches between stages. The resulting hierarchy produces multiple spatial resolutions like a feature pyramid ^[11].

For a fixed window size, local window attention grows linearly with total image area rather than quadratically in the number of all image patches. The trade-off is that distant regions cannot directly interact in every layer. Shifted partitions and successive stages propagate information beyond one window ^[11].

Swin V2 addressed stability and transfer to larger image and window resolutions with residual post-normalization, scaled cosine attention, and a log-spaced continuous relative position bias. The authors reported scaling to three billion parameters and inputs up to 1,536 x 1,536 pixels ^[12]. Those techniques belong to Swin V2, not to original ViT.

Hierarchical models are especially convenient for detection and segmentation systems that expect feature maps at several scales. Plain ViT can also serve those tasks, but often uses window attention in most layers, a small number of global blocks, or a simple feature pyramid during fine-tuning ^[11][24].

Self-supervised visual representation learning

Self-supervised learning constructs training targets from the data instead of requiring a human class label for each image. ViT is well suited to these objectives because visible patches, masked positions, multiple views, or teacher predictions can all be represented as token sequences. The major methods differ in what the model predicts.

Masked autoencoders

The masked autoencoder (MAE) removes a high proportion of randomly selected patches and feeds only visible patches to the encoder. A lightweight decoder receives the encoded visible tokens plus mask tokens and reconstructs the missing pixel patches. The decoder is discarded after pre-training ^[14].

MAE's asymmetric design saves encoder computation because masked tokens do not pass through the large encoder. The paper used a 75 percent masking ratio as its standard setting and reported that a ViT-H model reached 87.8 percent ImageNet top-1 after self-supervised pre-training and fine-tuning using ImageNet-1K only. The result belongs to a specific model, schedule, and fine-tuning protocol, but it established masked pixel reconstruction as a scalable ViT pre-training objective ^[14].

BEiT also performs masked image modeling, but it predicts discrete visual token identifiers rather than raw patch pixels. Its tokenizer was learned separately, and the encoder was trained to infer tokenized targets at masked positions ^[15]. MAE and BEiT are therefore related by masking but differ in target representation, decoder design, and loss.

EVA used another target: image-text-aligned visual features from a frozen teacher. The masked student reconstructed teacher features conditioned on visible patches, and the work scaled a plain ViT to one billion parameters using publicly accessible data ^[18]. These approaches illustrate a spectrum from low-level pixel targets to discrete codes and semantic teacher features.

Self-distillation

DINO trains student and teacher networks on different augmented views of the same image. The student matches the teacher's output distribution, while the teacher parameters follow an exponential moving average of the student. Centering and sharpening help prevent collapse. DINO's ViT experiments also showed that class-token attention maps could delineate foreground objects without segmentation labels ^[16].

DINOv2 combined self-distillation, masked-image objectives, curated data, and scaling to learn general visual features. The project built a curated 142-million-image training set from a larger pool, trained a roughly one-billion-parameter ViT, and distilled smaller models. Its evaluation covered image-level and pixel-level tasks using frozen features as well as adapted models ^[17].

Neither DINO nor DINOv2 turns an attention map into a guaranteed explanation. Their findings concern emergent spatial structure and transfer quality under particular probes. A downstream user still needs to validate performance, calibration, and failure modes for the intended domain.

Register tokens

Work on register tokens identified high-norm patch tokens in low-information image regions in several supervised and self-supervised ViTs. The authors argued that the networks repurposed those spatial tokens for internal global computation. Adding extra non-spatial register tokens during training gave the model a place for that computation and produced smoother patch feature maps in their experiments ^[19].

Register tokens are additional learned inputs, not image patches and not output classes. They are related to the class token in being non-spatial, but they are not necessarily used as the final readout. The finding also does not establish that every checkpoint has the same artifact or that adding registers after training will always repair it.

Vision-language learning

Contrastive image-text encoders

CLIP jointly trained an image encoder and a text encoder to assign matching image-text pairs higher similarity than mismatched pairs. Its image side included both residual-network and ViT variants. The paper trained on 400 million image-text pairs and evaluated zero-shot transfer by comparing an image embedding with text embeddings generated from class prompts ^[20].

The ViT in CLIP is an image encoder inside a larger contrastive system. Text does not pass through the vision encoder, and the image encoder alone does not generate language. The shared output space enables retrieval and prompt-based classification because the two towers are optimized to align matching concepts ^[20].

Locked-image Tuning, or LiT, starts with a pre-trained image encoder, keeps it fixed, and trains a text tower to align with its representations. The method worked with ViT, convolutional, and MLP-Mixer image encoders, showing that language alignment can be added without relearning the visual backbone ^[21].

SigLIP changed the contrastive objective. Instead of a softmax normalized across a batch, it applied a pairwise sigmoid loss to image-text pairs. This removed the objective's need for a global normalization over all pair similarities and reduced communication requirements in distributed training ^[22]. SigLIP commonly uses ViT image towers, but the sigmoid loss, not patch tokenization, is its defining contribution.

Connection to multimodal language models

A vision-language model may use a pre-trained ViT or ViT-derived encoder to turn an image into visual features consumed by a language model. Published systems illustrate different connectors. LLaVA used a trainable linear projection from a CLIP ViT-L/14 encoder into a language model's embedding space. BLIP-2 used a Querying Transformer with learned queries and cross-attention between a frozen image encoder and a frozen language model ^[37][38].

This composition is not architecturally trivial. The image resolution determines visual token count; the connector determines how much spatial information is retained; the language model must learn how visual tokens relate to text; and instruction data determines behavior. Public papers or model cards are needed before attributing a particular vision encoder to a proprietary system. An undisclosed product architecture should not be inferred solely from its ability to process images ^[37][38].

ViT helped this design pattern because it already emits token-shaped features and scales with Transformer tooling. It is still one component of a multimodal system rather than an explanation for the system's full reasoning, language generation, or safety behavior.

Detection and segmentation

Detection transformers and ViT backbones

DETR predates the publication of ViT and should not be described as an original ViT application. Its initial architecture used a convolutional backbone to produce image features, then a Transformer encoder-decoder and learned object queries to predict a set of boxes and labels. Bipartite matching supplied a one-to-one training assignment, removing the need for anchor generation and non-maximum suppression in the final prediction pipeline ^[23].

Later object detection systems used ViT-style backbones. ViTDet showed that a plain, non-hierarchical ViT pre-trained with MAE could be adapted using window attention and a simple feature pyramid. Its experiments demonstrated that a redesigned hierarchical pre-training backbone was not strictly necessary for competitive detection ^[24].

The distinction between backbone and detection head remains important. A ViT encoder produces visual features. The detector still needs a mechanism for multiscale representation, localization, class prediction, matching, or region decoding. Different heads can share a similar backbone and still behave very differently ^[23][24].

Semantic and promptable segmentation

Segmenter extended plain ViT to semantic segmentation by decoding patch embeddings into per-pixel class predictions. It evaluated both a point-wise linear decoder and a mask Transformer decoder and used classification-pretrained ViT encoders ^[25]. Because patch tokens are lower resolution than pixels, the output must be reshaped and upsampled or otherwise decoded to the image grid.

The Segment Anything Model and Dataset uses a ViT image encoder, a prompt encoder, and a lightweight mask decoder. The project trained a promptable segmentation model and built SA-1B, reported as more than one billion masks from 11 million licensed and privacy-respecting images ^[26]. Points and boxes are prompts to the decoder; the ViT image encoder does not by itself define the prompt interface.

Dense tasks stress plain ViT's spatial cost because fine boundaries favor small patches or high resolution. Window attention, feature pyramids, multiscale decoders, and masked pre-training are common responses. Reported segmentation quality also depends on the decoder, training data, test resolution, and whether evaluation is zero-shot, fine-tuned, or prompt-guided ^[25][26].

Generative models and video

Diffusion transformers

A Diffusion Transformer (DiT) replaces the U-Net commonly used in latent diffusion with a Transformer over patches of a noisy latent representation. It embeds diffusion timesteps and class conditions and predicts a denoising target. The DiT paper found that increasing forward-pass compute through model size or token count correlated with lower FID in its experiments. DiT-XL/2 with classifier-free guidance at scale 1.50 reported FID-50K of 2.27 on class-conditional ImageNet at 256 x 256 resolution ^[27].

DiT is ViT-like in its patchification and Transformer blocks, but it is not an image classifier. Its input is a noised latent tensor produced within a diffusion pipeline, and its output has spatial structure needed for denoising. A variational autoencoder and diffusion sampling procedure are separate components ^[27]. Later generative systems may use related Transformer backbones, but the architecture of a proprietary generator should be cited to an official technical disclosure rather than assumed.

Video transformers

Video greatly increases token count because the model must represent time as well as space. A naive global attention layer over every patch in every frame has quadratic cost in the full spatiotemporal sequence. Video transformers therefore commonly factorize attention or use tube-shaped tokens ^[28][29][30].

TimeSformer applies a Transformer directly to frame-level patches. Its divided-attention variant performs temporal attention and spatial attention separately inside each block, reducing the cost relative to one joint spatiotemporal attention operation ^[28]. ViViT explored several factorization strategies, including separate spatial and temporal encoders, and showed how image-pretrained ViT weights could initialize video models ^[29].

VideoMAE adapts masked autoencoding to video with tube masking. The paper reported that masking 90 to 95 percent of video tokens could still work well, attributing the high useful ratio partly to temporal redundancy ^[30]. It used a vanilla ViT backbone and self-supervised reconstruction before downstream evaluation.

These models belong to the wider family of video classification models. Their temporal sampling, frame rate, clip length, augmentation, and evaluation protocol are as important as the spatial backbone. An image ViT applied independently to frames does not model motion unless another component combines time.

ViT and convolutional networks

Inductive bias

Convolution hard-codes local connectivity and weight sharing across spatial positions. Plain ViT hard-codes a patch grid at its input but allows global content-dependent mixing in each attention layer. Original ViT therefore has less image-specific inductive bias, not no inductive bias. Patch size, positional embeddings, token order, augmentation, and the classification objective all shape what it can learn ^[1][6].

The architectures also build representations differently. Analyses of matched ViT and convolutional models found that ViT representations were more uniform across depth, that global information was aggregated early through self-attention, and that residual connections preserved lower-layer features into deeper layers ^[35]. Such findings describe measured representation similarities; they do not make attention weights a complete causal explanation.

Data efficiency and optimization

At the recipe and scale tested in the original paper, residual networks performed better in smaller pre-training regimes, while ViT benefited more from larger data ^[1]. DeiT, augmentation and regularization studies, and self-supervised pre-training narrowed that difference ^[5][6][14]. It is thus misleading to list "CNN: data efficient" and "ViT: data inefficient" without specifying training method.

Convolutional models also absorbed lessons from ViT training. ConvNeXt modernized a residual network with design and optimization choices associated with vision transformers and reported performance competitive with Swin models on classification and dense tasks ^[33]. This supports a cautious interpretation: some gains attributed to an architecture can actually come from its training recipe, normalization, stage design, or scaling choices.

Receptive field and spatial hierarchy

A plain ViT token can attend globally in the first layer, while a small convolutional kernel initially sees only a local neighborhood. A deep convolutional network nevertheless obtains a large effective receptive field, and many ViT heads learn strongly local patterns. "Global versus local" describes available connectivity, not the only behavior the trained model can express ^[1][35].

Conventional residual networks build a hierarchy by reducing spatial resolution and increasing channel width across stages. Original ViT keeps one token resolution. Swin and other hierarchical transformers reintroduce stages, while ViTDet shows that a plain backbone can construct a pyramid during detection fine-tuning ^[11][24].

Robustness

Robustness is not one scalar property. It includes common corruptions, natural distribution shifts, occlusion, adversarial perturbations, calibration, and changes to model parameters. Results can reverse when training data, model scale, attack strength, or evaluation set changes.

One broad study found that sufficiently pre-trained ViTs were at least as robust as its matched ResNet counterparts across several input and model perturbations ^[36]. Other studies have found different advantages under particular attacks or shifts. The safe summary is that ViT and convolutional networks have different failure patterns; an architecture name alone is not evidence of robustness for a deployed checkpoint.

Efficiency

For high-resolution inputs, global attention's quadratic token interaction is a clear disadvantage. Convolution and fixed-window attention scale more directly with image area. At modest sequence lengths, however, token-wise projections and MLPs can dominate, and dense matrix multiplication may use accelerators efficiently.

Parameter count, operations, peak memory, throughput, latency, energy, and batch-size sensitivity are different metrics. A model with fewer operations can be slower if its kernels, memory access, or dynamic control flow are inefficient on the target device. Fair comparison requires the same input resolution, precision, batch size, hardware, software stack, and measurement method.

Efficiency methods

Windowing and hierarchy

Window attention limits each token to a local group, reducing the all-pairs term. Shifted windows, occasional global blocks, pooling, or cross-window tokens then carry information between groups. This changes the receptive pattern and may require a task-specific balance between locality and global context ^[11].

Hierarchical pooling reduces token count as depth increases. It is particularly helpful for dense tasks because early layers preserve high spatial resolution while later layers operate on coarser semantic features. The cost is added architectural complexity and less uniformity than plain ViT ^[11][24].

Token pruning and merging

DynamicViT learns scores that progressively prune less informative tokens. Its paper reported pruning about 66 percent of input tokens, reducing 31 to 37 percent of floating-point operations and improving measured throughput by more than 40 percent with less than a 0.5 percentage-point accuracy drop in the tested settings ^[31]. Those values are experimental, not guarantees for arbitrary hardware or tasks.

Token Merging, or ToMe, instead combines similar tokens so that their information is aggregated rather than simply discarded. The method can be applied to pre-trained models without retraining and can also be used during training. Its paper reported roughly doubled throughput for selected high-resolution image models with a 0.2 to 0.3 percentage-point accuracy decrease ^[32].

Pruning and merging can interfere with tasks that need small objects, boundaries, or correspondence. A token considered unimportant for classification may matter for segmentation. Efficiency methods should therefore be validated on the downstream task, not only on an upstream ImageNet checkpoint.

Numeric and implementation optimization

Mixed precision, quantization, operator fusion, activation checkpointing, and memory-efficient attention can reduce resource use without changing the high-level architecture. Distillation can transfer behavior into a smaller encoder. Compilation and layout choices can make the patch projection, attention, and MLP operations more efficient on particular accelerators.

These methods have distinct error sources. Low-precision weights can alter logits and calibration. Activation checkpointing saves memory by recomputing values and can increase runtime. Memory-efficient exact attention preserves the mathematical result up to numeric differences, while sparse or linear attention approximations change the operation.

Evaluation and reproducibility

What an accuracy number includes

An ImageNet top-1 result is inseparable from the training and evaluation protocol. Relevant fields include:

• checkpoint and exact encoder configuration;
• patch size and evaluation resolution;
• pre-training data and whether labels, text, or self-supervision were used;
• fine-tuning, linear-probe, few-shot, or zero-shot protocol;
• crop and augmentation policy;
• label mapping and dataset version;
• single-crop versus multi-crop evaluation;
• use of distillation, ensembles, or extra data.

For example, 84 percent on ImageNet-V2 is not the same metric as 84 percent on the original ImageNet validation set. A zero-shot CLIP score is not directly comparable with a fully fine-tuned classifier. A model pre-trained on private web-scale data is not an equal-data comparison with a model trained only on ImageNet-1K.

Reproducing a checkpoint

The original authors released JAX code and checkpoints in the Google Research vision_transformer repository, including models pre-trained on ImageNet-21K and fine-tuning examples ^[34]. Independent libraries also implement ViT, but defaults can differ in class-token handling, interpolation, normalization epsilon, classifier heads, stochastic depth, or input preprocessing.

A reproducibility report should record the source commit or package version, checkpoint hash, input transform, numeric precision, random seeds, optimizer, schedule, and hardware. If a published checkpoint is evaluated rather than retrained, that distinction should be explicit. Small preprocessing differences can change accuracy enough to invalidate close comparisons.

Interpreting attention and features

Attention matrices show how a layer mixes value vectors under a particular head and input. They can reveal spatial patterns, but a high attention weight is not by itself a causal attribution of the final decision. Residual paths, MLPs, earlier layers, and value-vector content all contribute.

Class-token maps from DINO and smoother patch maps from register-token experiments are useful empirical observations ^[16][19]. They should be described as properties of tested models and training regimes. Claims about "understanding," object discovery, or explainability require a stated probe and evaluation, not only an appealing visualization.

Limitations and failure modes

Resolution cost

Global attention makes fine patch grids expensive. Medical slides, satellite scenes, documents, and long video can contain far more spatial tokens than a standard 224 x 224 image. Cropping into tiles reduces sequence length but can lose cross-tile context. Windowing restores efficiency but delays or limits long-range interaction.

Patch boundaries and small structures

Patchification combines pixels before the main encoder. If an object or boundary is much smaller than a patch, its evidence shares a token with surrounding content. Smaller patches, overlapping stems, high-resolution fine-tuning, or dense decoders can help, but each increases compute or changes the architecture.

The rectangular grid also creates sensitivity to crop and alignment. A small translation can assign pixels to different patches even if the semantic content is nearly unchanged. Augmentation and training scale can improve stability, but the model is not inherently invariant to every shift.

Dependence on data and labels

Large-scale supervised pre-training can inherit label errors, dataset imbalance, and selection bias. Web image-text pre-training adds caption noise, cultural bias, duplicates, and uncertain licensing or provenance. Self-supervision removes the need for class annotations but does not remove bias from which images were collected.

Transfer quality is domain-dependent. Features learned from natural photographs may not preserve clinically relevant signals, remote-sensing bands, or industrial defects. Validation on a target population and task is necessary even when a backbone performs well on broad academic benchmarks.

Position and resolution transfer

Interpolating position embeddings produces a tensor with the right shape but may create a distribution shift. Very different aspect ratios or much longer sequences can exceed what the checkpoint learned. Relative position systems and patch-flexible training reduce some constraints, but they introduce their own assumptions.

Calibration, uncertainty, and adversarial behavior

High classification accuracy does not imply calibrated probabilities, adversarial security, or reliable uncertainty estimates. Robustness findings are conditional on the threat model and training. A checkpoint should be tested against the corruptions, shifts, or attacks relevant to its application.

Environmental and access costs

The regular structure of ViT enables scaling, but the largest models require substantial computation, memory, and data infrastructure. Published TPU-core-days or floating-point counts are incomplete environmental measures because hardware generation, utilization, energy source, and failed experiments matter. Private training datasets and unreleased checkpoints can also limit independent replication.

Practical model selection

A plain ViT checkpoint is a reasonable starting point when transferable representations, simple architecture, or compatibility with an existing Transformer stack matters. A hierarchical or windowed model may be preferable for high-resolution detection and segmentation. A convolutional or hybrid model may offer better latency or data efficiency under a small deployment budget.

The selection process should begin with constraints rather than a model name:

1. Define the task and evaluation distribution.
2. Set latency, memory, throughput, and energy limits on the target hardware.
3. Decide whether labeled, unlabeled, or image-text pre-training is acceptable.
4. Compare checkpoints under identical preprocessing and resolution.
5. Measure calibration and relevant failure modes, not only aggregate accuracy.
6. Verify dataset provenance, license terms, and model-card limitations.

Patch size should be treated as a deployment parameter only if the checkpoint was trained or adapted for that flexibility. Otherwise it is part of the learned model. Likewise, raising resolution without fine-tuning can increase cost without reliably improving accuracy.

For feature extraction, users should decide whether to take the class token, mean-pooled patch tokens, selected intermediate layers, or dense patch features. The best choice depends on whether the downstream task needs global semantics or spatial detail. DINO-style encoders may provide useful dense features, while a supervised classification checkpoint can prioritize its final global readout.

Significance

ViT established that a general Transformer encoder could learn competitive visual representations from patch sequences at scale. That result changed the design space of visual models: researchers could transfer advances in attention, self-supervision, scaling, distillation, and multimodal alignment between language and vision with fewer architecture-specific barriers.

Its influence is visible in several distinct lines of work. DeiT made ImageNet-only training practical; Swin introduced a hierarchical windowed backbone; MAE and BEiT developed masked image modeling; DINO explored self-distilled dense features; CLIP and SigLIP aligned ViT image encoders with text; ViTDet and Segmenter adapted patch tokens to dense prediction; and DiT applied patch-based Transformers to diffusion.

The most accurate historical conclusion is not that ViT ended convolution. Instead, it made patch-based Transformer encoders a durable alternative and encouraged convergence between previously separate design traditions. Modern visual systems freely combine global attention, local windows, convolutional stems, multiscale hierarchies, and task-specific decoders. The original ViT remains the clearest reference point for understanding that broader family ^{[1][5][11][14][20][24][27][33]}.

See also

• Artificial Intelligence
• Attention
• Image Recognition
• Large Language Model

References

1. ^Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. "An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale." ICLR 2021. arxiv.org/...2010.11929
2. ^Vaswani, A., Shazeer, N., Parmar, N., et al. "Attention Is All You Need." NeurIPS 2017. papers.nips.cc/...47dee91fbd053c1c4a845aa-Abstract
3. ^Parmar, N., Vaswani, A., Uszkoreit, J., et al. "Image Transformer." ICML 2018. proceedings.mlr.press/...parmar18a
4. ^Wang, X., Girshick, R., Gupta, A., and He, K. "Non-Local Neural Networks." CVPR 2018. openaccess.thecvf.com/..._Networks_CVPR_2018_paper
5. ^Touvron, H., Cord, M., Douze, M., et al. "Training Data-Efficient Image Transformers and Distillation Through Attention." ICML 2021. proceedings.mlr.press/...touvron21a
6. ^Steiner, A., Kolesnikov, A., Zhai, X., et al. "How to Train Your ViT? Data, Augmentation, and Regularization in Vision Transformers." Transactions on Machine Learning Research, 2022. arxiv.org/...2106.10270
7. ^Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. "Scaling Vision Transformers." CVPR 2022. openaccess.thecvf.com/...nsformers_CVPR_2022_paper
8. ^Dehghani, M., Djolonga, J., Mustafa, B., et al. "Scaling Vision Transformers to 22 Billion Parameters." ICML 2023. proceedings.mlr.press/...dehghani23a
9. ^D'Ascoli, S., Touvron, H., Leavitt, M. L., et al. "ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases." ICML 2021. proceedings.mlr.press/...d-ascoli21a
10. ^Touvron, H., Cord, M., Sablayrolles, A., et al. "Going Deeper with Image Transformers." ICCV 2021. openaccess.thecvf.com/...nsformers_ICCV_2021_paper
11. ^Liu, Z., Lin, Y., Cao, Y., et al. "Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows." ICCV 2021. openaccess.thecvf.com/...d_Windows_ICCV_2021_paper
12. ^Liu, Z., Hu, H., Lin, Y., et al. "Swin Transformer V2: Scaling Up Capacity and Resolution." CVPR 2022. openaccess.thecvf.com/...esolution_CVPR_2022_paper
13. ^Beyer, L., Izmailov, P., Kolesnikov, A., et al. "FlexiViT: One Model for All Patch Sizes." CVPR 2023. openaccess.thecvf.com/...tch_Sizes_CVPR_2023_paper
14. ^He, K., Chen, X., Xie, S., et al. "Masked Autoencoders Are Scalable Vision Learners." CVPR 2022. openaccess.thecvf.com/..._Learners_CVPR_2022_paper
15. ^Bao, H., Dong, L., Piao, S., and Wei, F. "BEiT: BERT Pre-Training of Image Transformers." ICLR 2022. arxiv.org/...2106.08254
16. ^Caron, M., Touvron, H., Misra, I., et al. "Emerging Properties in Self-Supervised Vision Transformers." ICCV 2021. openaccess.thecvf.com/...nsformers_ICCV_2021_paper
17. ^Oquab, M., Darcet, T., Moutakanni, T., et al. "DINOv2: Learning Robust Visual Features Without Supervision." Transactions on Machine Learning Research, 2024. arxiv.org/...2304.07193
18. ^Fang, Y., Wang, W., Xie, B., et al. "EVA: Exploring the Limits of Masked Visual Representation Learning at Scale." CVPR 2023. openaccess.thecvf.com/...arning_at_CVPR_2023_paper
19. ^Darcet, T., Oquab, M., Mairal, J., and Bojanowski, P. "Vision Transformers Need Registers." ICLR 2024. proceedings.iclr.cc/...57e531a-Abstract-Conference
20. ^Radford, A., Kim, J. W., Hallacy, C., et al. "Learning Transferable Visual Models From Natural Language Supervision." ICML 2021. proceedings.mlr.press/...radford21a
21. ^Zhai, X., Wang, X., Mustafa, B., et al. "LiT: Zero-Shot Transfer with Locked-Image Text Tuning." CVPR 2022. openaccess.thecvf.com/...xt_Tuning_CVPR_2022_paper
22. ^Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. "Sigmoid Loss for Language Image Pre-Training." ICCV 2023. openaccess.thecvf.com/...-Training_ICCV_2023_paper
23. ^Carion, N., Massa, F., Synnaeve, G., et al. "End-to-End Object Detection with Transformers." ECCV 2020. ecva.net/...832_ECCV_2020_paper
24. ^Li, Y., Mao, H., Girshick, R., and He, K. "Exploring Plain Vision Transformer Backbones for Object Detection." ECCV 2022. ecva.net/...2151_ECCV_2022_paper
25. ^Strudel, R., Garcia, R., Laptev, I., and Schmid, C. "Segmenter: Transformer for Semantic Segmentation." ICCV 2021. openaccess.thecvf.com/...mentation_ICCV_2021_paper
26. ^Kirillov, A., Mintun, E., Ravi, N., et al. "Segment Anything." ICCV 2023. openaccess.thecvf.com/..._Anything_ICCV_2023_paper
27. ^Peebles, W., and Xie, S. "Scalable Diffusion Models with Transformers." ICCV 2023. openaccess.thecvf.com/...nsformers_ICCV_2023_paper
28. ^Bertasius, G., Wang, H., and Torresani, L. "Is Space-Time Attention All You Need for Video Understanding?" ICML 2021. proceedings.mlr.press/...bertasius21a
29. ^Arnab, A., Dehghani, M., Heigold, G., et al. "ViViT: A Video Vision Transformer." ICCV 2021. openaccess.thecvf.com/...ansformer_ICCV_2021_paper
30. ^Tong, Z., Song, Y., Wang, J., and Wang, L. "VideoMAE: Masked Autoencoders Are Data-Efficient Learners for Self-Supervised Video Pre-Training." NeurIPS 2022. proceedings.neurips.cc/...354a-Abstract-Conference
31. ^Rao, Y., Zhao, W., Liu, B., et al. "DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification." NeurIPS 2021. proceedings.neurips.cc/...7fbb873e8b2f9f2-Abstract
32. ^Bolya, D., Fu, C. Y., Dai, X., et al. "Token Merging: Your ViT But Faster." ICLR 2023. arxiv.org/...2210.09461
33. ^Liu, Z., Mao, H., Wu, C. Y., et al. "A ConvNet for the 2020s." CVPR 2022. openaccess.thecvf.com/...the_2020s_CVPR_2022_paper
34. ^Google Research. "Vision Transformer and MLP-Mixer Architectures." Source code and model repository. github.com/...vision_transformer
35. ^Raghu, M., Unterthiner, T., Kornblith, S., et al. "Do Vision Transformers See Like Convolutional Neural Networks?" NeurIPS 2021. proceedings.neurips.cc/...302ba2b8b7f51e0-Abstract
36. ^Bhojanapalli, S., Chakrabarti, A., Glasner, D., et al. "Understanding Robustness of Transformers for Image Classification." ICCV 2021. openaccess.thecvf.com/...ification_ICCV_2021_paper
37. ^Liu, H., Li, C., Wu, Q., and Lee, Y. J. "Visual Instruction Tuning." NeurIPS 2023. proceedings.neurips.cc/...6de0-Abstract-Conference
38. ^Li, J., Li, D., Savarese, S., and Hoi, S. "BLIP-2: Bootstrapping Language-Image Pre-Training with Frozen Image Encoders and Large Language Models." ICML 2023. proceedings.mlr.press/...li23q