Vision Encoder

A vision encoder is the neural network component that turns an image into a sequence of numerical vectors that other models can consume. In a modern vision-language model (VLM), the encoder converts pixels into visual tokens, a small adapter maps those tokens into the input space of a large language model, and the language model then reasons over image and text together. The encoder is what decides which visual information survives this handoff, so its architecture, pretraining recipe, and input resolution set a ceiling on everything the downstream system can do with images.

Most production vision encoders today are vision transformers (ViTs) pretrained on image-text pairs with a contrastive objective, following the pattern set by OpenAI's CLIP in 2021 and refined by Google's SigLIP family ^[3][4]. Open VLMs such as LLaVA popularized a simple assembly: take a frozen or lightly tuned pretrained encoder, bolt it to an LLM through a small projection network, and train on instruction data ^[13][14]. Since then the main lines of work have been scaling encoders up, feeding them higher and more flexible resolutions, replacing contrastive training with self-supervised or hybrid objectives, and, in a smaller counter-trend, removing the separate encoder entirely.

From CNN features to transformer patches

Before transformers, multimodal systems built on convolutional neural network features. A common recipe in the late 2010s used an object detection model to propose salient image regions and passed each region's feature vector to the language side: the "bottom-up and top-down attention" method of 2017 extracted region features with Faster R-CNN and won that year's VQA Challenge ^[1]. Whole-image features from classifiers such as ResNet served the same role in captioning pipelines.

The Vision Transformer changed the interface. The 2020 paper "An Image is Worth 16x16 Words" showed that a pure transformer applied to a sequence of 16x16 pixel patches could match or beat convolutional networks on ImageNet-scale classification when pretrained on enough data, concluding that "reliance on CNNs is not necessary" ^[2]. This patchification step, cutting an image into a grid of patches and linearly projecting each one into an embedding, is what makes ViTs natural companions for language models: the encoder's output is already a token sequence, structurally similar to embedded text. Nearly every encoder discussed below is a ViT variant, though convolution-influenced designs such as the Swin Transformer also serve as backbones in vision systems.

Contrastive pretraining: CLIP and SigLIP

The dominant pretraining recipe for VLM encoders is contrastive learning on web image-text pairs. CLIP (Contrastive Language-Image Pre-training, paper published February 2021) trained an image encoder and a text encoder on 400 million pairs so that matching image-caption pairs land close together in a shared embedding space and mismatched pairs land far apart. The pretraining task, predicting which caption goes with which image, produced representations that transfer zero-shot: CLIP matched the ImageNet accuracy of the original ResNet-50 without using any of ImageNet's 1.28 million training examples ^[3]. CLIP's public ViT checkpoints, especially ViT-L/14, became the default eyes of early open VLMs.

SigLIP (Sigmoid Loss for Language Image Pre-training, March 2023) replaced CLIP's softmax contrastive loss with a pairwise sigmoid loss that operates on individual image-text pairs and needs no global normalization across the batch. That decoupling made training cheaper and revealed that enormous batches were unnecessary; the authors found a batch size of 32k sufficient, and, combining the sigmoid loss with locked-image tuning (a frozen pretrained image tower), trained a SigLiT model to 84.5 percent zero-shot ImageNet accuracy in two days on four TPUv4 chips ^[4].

SigLIP 2 (February 2025) extended the recipe into a unified objective that combines the original image-text loss with captioning-based pretraining, self-distillation, masked prediction, and online data curation, and it added multilingual coverage. The family spans four sizes: ViT-B (86M parameters), L (303M), So400m (400M), and g (1B) ^[5]. The B and So400m sizes also ship in NaFlex variants that process images at multiple resolutions while preserving native aspect ratio, which matters for OCR and document tasks where squashing a page into a square destroys detail ^[6].

Beyond contrastive training

Contrastive image-text training is not the only route to a strong encoder. Self-supervised learning on images alone produces encoders with strong dense, spatially detailed features. Meta's DINOv2 (April 2023) trained a 1-billion-parameter ViT on an automatically curated image dataset with no text supervision at all, then distilled it into smaller models that beat OpenCLIP on a range of benchmarks ^[7]. Its successor DINOv3 (August 2025) scaled the approach further and introduced Gram anchoring, a technique that counters the degradation of dense feature maps over long training runs; Meta reports it outperforms specialized state-of-the-art models on various dense vision tasks without fine-tuning ^[8].

Meta's Perception Encoder (April 2025) took a third position: contrastive vision-language training alone can produce strong, general embeddings for classification, retrieval, question answering, and dense prediction, but the best embeddings sit in intermediate layers rather than at the network's output. The paper introduces language alignment and spatial alignment methods to pull those hidden representations out for multimodal language modeling and dense prediction respectively ^[9].

Scale is its own axis. InternVL (December 2023) scaled a vision encoder to 6 billion parameters (InternViT-6B) and progressively aligned it with an LLM on web-scale image-text data, positioning the result as an alternative to Google's ViT-22B ^[10].

Scroll sideways for more →

Plugging an encoder into a language model

Architectures differ mainly in the connector between the encoder and the LLM. DeepMind's Flamingo (April 2022) bridged pretrained vision-only and language-only models with new interface layers and set few-shot state of the art across captioning and visual question answering benchmarks ^[11]. BLIP-2 (January 2023) made frugality the point: it froze both the image encoder and the LLM and trained only a lightweight Querying Transformer (Q-Former) between them, in two stages (representation learning against the frozen encoder, then generative learning against the frozen LLM). It outperformed the 80B-parameter Flamingo on zero-shot VQAv2 by 8.7 percent with 54 times fewer trainable parameters ^[12].

LLaVA (April 2023) simplified the connector to nearly nothing: a single projection matrix mapping the output of the CLIP ViT-L/14 encoder into the token space of the Vicuna LLM. Training happens in two stages, first updating only the projection to align modalities, then fine-tuning the projection and the language model on instruction data ^[13][14]. LLaVA-1.5 (October 2023) swapped in a higher-resolution CLIP-ViT-L-336px encoder and replaced the linear layer with a two-layer MLP projection, which, together with better data, produced much stronger baselines ^[15]. This encoder-projector-LLM pattern became the template for most open VLMs. Google's PaliGemma (2024), for instance, pairs the SigLIP-So400m encoder with the Gemma-2B language model ^[16], and the Cambrian-1 project (June 2024) used the same interface to benchmark more than 20 different vision encoders (self-supervised, strongly supervised, and combinations), introducing a Spatial Vision Aggregator connector that fuses high-resolution features while reducing token count ^[17].

The number of visual tokens the connector emits is a recurring tension. More tokens preserve more detail but consume LLM context and compute; resampling connectors like the Q-Former compress aggressively, while projection-only designs like LLaVA's pass everything through.

Resolution strategies

Classic CLIP-style encoders operate at a fixed, low resolution (224 or 336 pixels square), which is hopeless for reading dense documents or spotting small objects. Three approaches emerged.

Tiling splits a large image into a grid of encoder-sized crops. LLaVA-NeXT (January 2024) implemented this as "AnyRes," encoding grid configurations of {2x2, 1x{2,3,4}, {2,3,4}x1} tiles alongside a downscaled overview, supporting effective resolutions such as 672x672, 336x1344, and 1344x336 ^[18]. Many open models adopted variants of the idea.

Native dynamic resolution redesigns the encoder itself. Alibaba's Qwen2-VL (August 2024) introduced "Naive Dynamic Resolution": one ViT of 675 million parameters, shared across all model sizes and across image and video input, maps images of arbitrary resolution to a variable number of visual tokens, paired with a multimodal rotary position embedding (M-RoPE) that factors position into temporal, height, and width components ^[19][20]. Qwen2.5-VL (technical report February 2025) trained its native dynamic-resolution ViT from scratch and added window attention to keep compute manageable at high resolution ^[21]. SigLIP 2's NaFlex variant brings the same flexibility to an off-the-shelf pretrained encoder ^[6].

Encoder-free models

A minority line of work removes the separate encoder entirely. Fuyu-8B, released by Adept in October 2023, is a plain decoder-only transformer with no image encoder: image patches are linearly projected straight into the first layer of the transformer and processed in raster order, with a special image-newline character marking row breaks, so the model handles arbitrary resolutions without a separate vision stage ^[22]. The EVE paper ("Unveiling Encoder-Free Vision-Language Models," June 2024, a NeurIPS 2024 spotlight) showed that with the right training strategy an encoder-free model trained on 35 million publicly accessible examples could match similarly sized encoder-based rivals and outperform Fuyu-8B ^[23]. Encoder-free designs simplify the pipeline and dodge the fixed-resolution problem, but as of 2026 the encoder-plus-LLM pattern still dominates open production systems.

Why the choice of encoder matters

Cambrian-1's controlled comparison found that encoder choice materially shifts downstream ability: language-supervised encoders like CLIP outperformed self-supervised ones overall, but the gap narrowed on vision-centric benchmarks, where a well-trained self-supervised model like DINOv2 was competitive, and combining several encoders brought consistent gains, with even OCR benchmarks benefiting from the addition of DINOv2 ^[17]. That result explains a visible trend in open VLMs toward hybrid vision stacks and toward encoders like SigLIP 2 and Perception Encoder that fold captioning, self-distillation, or alignment objectives into contrastive training so a single tower serves both needs ^[5][9]. It also explains why OCR-heavy applications, from document parsing to GUI agents, pushed the field toward the native-resolution encoders of the Qwen-VL line ^[19][21].

See also

• Vision transformer
• CLIP
• SigLIP
• Vision-language model
• LLaVA
• Vision-language-action model

References

1. ^"Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering." arXiv, July 2017. arxiv.org/...1707.07998
2. ^Dosovitskiy, A. et al. "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale." arXiv, October 2020. arxiv.org/...2010.11929
3. ^OpenAI. "Learning Transferable Visual Models From Natural Language Supervision." arXiv, February 2021. arxiv.org/...2103.00020
4. ^"Sigmoid Loss for Language Image Pre-Training." arXiv, March 2023. arxiv.org/...2303.15343
5. ^"SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features." arXiv, February 2025. arxiv.org/...2502.14786
6. ^Hugging Face. "SigLIP 2: A better multilingual vision language encoder." huggingface.co/...siglip2
7. ^"DINOv2: Learning Robust Visual Features without Supervision." arXiv, April 2023. arxiv.org/...2304.07193
8. ^"DINOv3." arXiv, August 2025. arxiv.org/...2508.10104
9. ^"Perception Encoder: The best visual embeddings are not at the output of the network." arXiv, April 2025. arxiv.org/...2504.13181
10. ^"InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks." arXiv, December 2023. arxiv.org/...2312.14238
11. ^"Flamingo: a Visual Language Model for Few-Shot Learning." arXiv, April 2022. arxiv.org/...2204.14198
12. ^"BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models." arXiv, January 2023. arxiv.org/...2301.12597
13. ^"Visual Instruction Tuning." arXiv, April 2023. arxiv.org/...2304.08485
14. ^LLaVA project page. "Visual Instruction Tuning." llava-vl.github.io
15. ^"Improved Baselines with Visual Instruction Tuning." arXiv, October 2023. arxiv.org/...2310.03744
16. ^"PaliGemma: A versatile 3B VLM for transfer." arXiv, July 2024. arxiv.org/...2407.07726
17. ^"Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs." arXiv, June 2024. arxiv.org/...2406.16860
18. ^LLaVA team. "LLaVA-NeXT: Improved reasoning, OCR, and world knowledge." January 30, 2024. llava-vl.github.io/...2024-01-30-llava-next
19. ^"Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution." arXiv, September 2024. arxiv.org/...2409.12191
20. ^Qwen team. "Qwen2-VL: To See the World More Clearly." Qwen blog, August 29, 2024. qwenlm.github.io/...qwen2-vl
21. ^Qwen team. "Qwen2.5-VL Technical Report." arXiv, February 2025. arxiv.org/...2502.13923
22. ^Adept. "fuyu-8b model card." Hugging Face. huggingface.co/...fuyu-8b
23. ^"Unveiling Encoder-Free Vision-Language Models." arXiv, June 2024. arxiv.org/...2406.11832