CLIP (Contrastive Language-Image Pre-training)

CLIP, short for Contrastive Language-Image Pre-training, is a family of neural networks developed by OpenAI researchers to learn a shared representation of images and natural-language descriptions. Instead of training an image classifier on a fixed set of labeled categories, CLIP learns which text belongs with which image. At inference time, the resulting image and text encoders can score arbitrary image-text pairs, support retrieval, or turn written class descriptions into a zero-shot classifier. The original study trained on 400 million image-text pairs gathered from the web and reported results across more than 30 datasets.^[1]

The name CLIP can refer to the training method, the paired image-and-text architecture, or OpenAI's released checkpoints. The original family combined either a modified ResNet or a Vision Transformer with a Transformer text encoder. Model dimensions differ across the family: the text tower has 12 layers in every reported variant, but its width and attention-head count increase for larger models.^[2] This distinction matters because a single text-encoder size does not describe every CLIP checkpoint.

CLIP was presented as a research system rather than a model cleared for unrestricted deployment. Its model card says that deployed use was outside the intended scope without careful, task-specific evaluation, and that surveillance and facial-recognition uses were always out of scope. It also limits intended language use to English because the model was not deliberately trained or evaluated for other languages.^[3]

At a glance

Scroll sideways for more →

The official repository exposes the checkpoints RN50, RN101, RN50x4, RN50x16, RN50x64, ViT-B/32, ViT-B/16, ViT-L/14, and ViT-L/14@336px. Its tokenizer API uses a context length of 77 tokens. The repository license applies to the released reference software; it should not be read as a license for the unreleased training data.^[4]

Training data

The researchers built a private dataset called WebImageText, usually abbreviated WIT. They began with a query list containing words that appeared at least 100 times in English Wikipedia, high pointwise-mutual-information bigrams, Wikipedia article titles above a search-volume threshold, and WordNet synsets not already represented. The final list contained 500,000 queries. Up to 20,000 image-text pairs were collected per query in an effort to keep the query distribution approximately balanced. The resulting corpus contained 400 million pairs.^[1]

The pairs came from publicly available internet sources and were gathered in what the model card calls a mostly non-interventionist manner, rather than being manually authored for a fixed classification taxonomy. CLIP therefore learns from natural-language supervision at a much larger scale than conventional labeled datasets, but the supervision is less controlled. The paper does not publish a site-by-site inventory or the examples in WIT. The model card says the sources included internet crawling and existing public datasets such as YFCC100M, and it states that WIT would not be released.^[3]

The lack of a released training set has practical consequences. Independent researchers cannot reproduce the original data filtering, fully audit which people or copyrighted works appear, or directly measure all kinds of train-test contamination. The paper did perform a duplicate analysis against 35 evaluation datasets. Nine had no detected overlap; among the rest, median detected overlap was 2.2 percent and the average was 3.2 percent. The authors estimated that overlap changed overall accuracy by less than 0.1 percentage points in most cases, with the largest detected increase being 0.6 points on Birdsnap. They also noted that approximate duplicate detection cannot establish recall over all 400 million examples.^[1]

Architecture

CLIP has two encoders. The image encoder maps an image to a feature vector, while the text encoder maps a token sequence to another vector in the same embedding space. Learned linear projections put the two outputs into a shared dimensionality. Both outputs are normalized, so their dot product is a cosine similarity. During training, matching pairs are pulled together in the shared space and nonmatching examples in the same batch are pushed apart.^[1]

Image encoders

The ResNet branch modifies the standard architecture in three main ways. It replaces the initial stem with three convolutional layers, uses anti-aliased strided convolutions, and replaces global average pooling with a single attention-pooling layer. The attention query is based on the global average-pooled representation, while keys and values come from spatial features.^[1]

The alternative branch adapts the Vision Transformer. An image is divided into patches, embedded as a token sequence, and processed by self-attention blocks. The CLIP version adds layer normalization to the combined patch and position embeddings before the Transformer. ViT-B/32 uses 32-pixel patches, ViT-B/16 uses 16-pixel patches, and ViT-L/14 uses 14-pixel patches.^[1]

Text encoder

The text tower tokenizes lowercased text with a byte-pair encoding vocabulary of 49,408 entries. It uses masked self-attention, and the representation at the end-of-text token becomes the sequence feature before projection into the shared embedding space. The masking preserves the option of adding language modeling, although the reported CLIP models were optimized with the contrastive objective.^[1]

The text tower is not fixed at one universal width. Depth remains 12 layers, but width rises from 512 in RN50, RN101, ViT-B/32, and ViT-B/16 to 640 in RN50x4, 768 in RN50x16 and ViT-L/14, and 1,024 in RN50x64. Attention heads rise with width from 8 to 10, 12, or 16. The official tokenizer packs inputs into a 77-token context, padding shorter sequences and rejecting over-length inputs unless truncation is explicitly enabled.^[2][4]

Reported model configurations

Scroll sideways for more →

These are the dimensions reported in the paper supplement, not parameter counts. ViT-L/14@336px is a higher-resolution fine-tuning of ViT-L/14 rather than a new tower shape.^[2]

Training objective and procedure

For a minibatch of N image-text pairs, CLIP computes all N by N similarities between normalized image and text features. The correct pairs lie on the diagonal. One cross-entropy loss asks each image to select its matching text from the batch, and a second asks each text to select its matching image. The final loss is the mean of those two directions. A learned temperature rescales the similarities before the softmax.^[1]

This design turns every batch into many classification alternatives without requiring a manually defined label vocabulary. With the reported batch size of 32,768, each example is contrasted against 32,767 mismatched examples in each direction. The models were trained from scratch for 32 epochs with Adam, weight decay, learning-rate warmup, and cosine decay. The only image augmentation reported for CLIP pre-training was a random square crop.^[2]

The largest ResNet, RN50x64, took 18 days on 592 Nvidia V100 GPUs. ViT-L/14 took 12 days on 256 V100 GPUs. The researchers then fine-tuned ViT-L/14 for one additional epoch at 336-pixel resolution to produce ViT-L/14@336px.^[1] These are the paper's reported training runs, not current hardware requirements for inference or third-party reimplementations.

Zero-shot classification

CLIP does not need a learned classification head to score a new set of classes. For each candidate label, a user writes one or more prompts, such as a photo of a dog. The text encoder converts those prompts into class vectors. The image encoder converts the input image into its vector, and similarity scores between the image and every class vector become the logits for a zero-shot classifier.^[1]

Prompt wording is part of the classifier specification. A bare label such as boxer can refer to a dog breed or a person, while a phrase such as a photo of a boxer, a type of dog resolves the intended sense. The original experiments used dataset-specific templates and averaged embeddings from multiple prompts. On ImageNet, changing from bare class names to the phrase a photo of a {label} improved accuracy by 1.3 percentage points. Ensembling 80 prompts added another 3.5 points over the default prompt. Across the paper's 36-dataset prompt study, prompt engineering and ensembling together improved average zero-shot performance by almost 5 points.^[2]

This procedure is called zero-shot because no labeled examples from the target classification dataset are used to fit the classifier. It does not mean that the concepts were absent from WIT. The image, words, or closely related material may have appeared in pre-training, and the unreleased corpus prevents a complete concept-level audit.^[1][3]

Evaluation

The strongest reported model, ViT-L/14@336px, reached 76.2 percent zero-shot top-1 accuracy on ImageNet. The paper compared this with 76.2 percent for the original supervised ResNet-50 while emphasizing that CLIP used none of ImageNet's 1.28 million labeled training examples. Across 27 datasets, zero-shot CLIP outperformed a linear classifier trained on features from a supervised ResNet-50 on 16 tasks. Its relative results were weaker on specialized tasks such as satellite imagery, lymph-node tumor detection, counting rendered objects, traffic-sign recognition, and estimating vehicle distance.^[1]

The paper also tested frozen CLIP features with linear probes. On a 12-dataset evaluation suite used in earlier transfer-learning work, its best linear-probe model improved average score by 2.6 points over the best prior result reported by the authors. On their broader 27-dataset suite, the improvement was about 5 points. Those results assess representation quality after fitting a supervised linear classifier and should not be confused with the zero-shot figures.^[1]

Class design and deployment evaluation

CLIP's flexible class vocabulary can expose behavior that a standard fixed-head classifier would not show. OpenAI's follow-up evaluation found that changing the names and membership of candidate classes could substantially change both performance and the distribution of harmful outputs. The study also found disparities in exploratory demographic probes and argued that aggregate benchmark accuracy was insufficient to establish deployment safety.^[5]

These findings make the evaluation context inseparable from the model score. A result depends on the checkpoint, image preprocessing, prompt templates, candidate classes, and dataset. A responsible report should preserve all of those details and should test the exact domain and class taxonomy rather than transferring a headline ImageNet number to a different task.^[3][5]

Distribution shift

The original study compared ImageNet accuracy with results on ImageNetV2, ImageNet Sketch, ImageNet-A, and ImageNet-R. It reported that zero-shot CLIP reduced the difference between expected and observed accuracy under these natural distribution shifts by as much as 75 percent relative to the paper's supervised baseline trend.^[1] This is a specific regression-based measure of effective robustness, not a claim that error falls by 75 percent on every shifted dataset.

A later controlled study trained CLIP-like models on several datasets and objectives. It found that the diversity of the training distribution was the main factor explaining the observed distribution-shift robustness, while language supervision, contrastive training, and other tested factors made little or no independent contribution in that setup.^[6] That result refines the causal explanation for CLIP's robustness without negating the original measured comparisons.

Limitations and risks

Prompt and task sensitivity

Natural-language labels are useful because they let users define a task without retraining, but the same flexibility makes results sensitive to wording. Prompt templates can resolve ambiguity and improve performance, while a poorly chosen or incomplete class set can produce misleading probabilities. Similarities are normalized only across the candidate texts supplied at inference time, so the output is not an open-ended statement of everything that might be present in an image.^[1][5]

CLIP is also weak on tasks that demand fine spatial detail, exact counting, or highly specialized visual expertise. The original paper highlighted large gaps on EuroSAT, RESISC45, PatchCamelyon, CLEVR Counts, GTSRB, and KITTI Distance. It suggested that scaling alone was unlikely to close every gap: a simple extrapolation estimated roughly a 1,000-fold increase in compute would be needed for zero-shot CLIP to reach the paper's overall state-of-the-art average.^[1] That extrapolation is an illustration of the observed scaling trend, not a reliable forecast of later systems.

Typographic and compositional failures

CLIP can respond strongly to words rendered inside an image. Researchers analyzing multimodal neurons demonstrated a typographic attack in which adding the word iPod to an apple could move the model toward the text-induced concept.^[7] Such examples show that a high image-text similarity can reflect visible text rather than the depicted object. They do not imply that every CLIP prediction is dominated by text.

Compositional tests reveal a different weakness. Winoground holds caption words constant while changing their order and pairs those captions with two images; the benchmark found that leading vision-language models, including CLIP variants, performed poorly on the combined image-and-text matching criterion.^[8] ARO similarly tested object attributes, relations, and word order, finding that common vision-language models could rely heavily on bag-of-words-like cues.^[9]

Benchmark construction can itself create artifacts. SugarCrepe showed that rule-generated negative captions in several compositional benchmarks were often implausible or ungrammatical, allowing models that never saw an image to perform unexpectedly well. Its adversarially filtered benchmark still found substantial compositional limitations across 17 pretrained CLIP models.^[10] Together, these studies support a narrower conclusion: strong retrieval or classification performance does not establish reliable understanding of relations, attribute binding, or word order.

Data, bias, and privacy

WIT was assembled from public web sources at scale. The model card cautions that internet data overrepresents people and societies with greater internet access and can carry social biases. Because the dataset is unavailable, independent auditors cannot enumerate its contents or reproduce the original collection.^[3]

The original paper and follow-up evaluation tested sensitive classifications to expose possible harms, not to endorse those uses. They found demographic disparities and showed that adding a category such as child could sharply change how images of younger people were distributed among other labels.^[1][5] OpenAI's model card therefore excludes surveillance and facial recognition and recommends thorough in-domain testing even for constrained research uses.^[3]

The model can also associate names with faces from web supervision. The paper reported nontrivial zero-shot celebrity identification even though it was not trained on a dedicated face-identification dataset.^[1] This capability increases privacy and misuse concerns. It should not be interpreted as evidence that CLIP is accurate, appropriate, or authorized for identifying people.

Language and calibration

The released model was not purposefully trained or evaluated outside English, so the model card limits its intended use to English.^[3] Performance for other languages, dialects, transliterated text, or culture-specific concepts cannot be inferred from English benchmarks.

CLIP similarities are relative scores in an embedding space. Applying softmax to a selected class list produces probabilities that sum to one, but those values are not automatically calibrated confidence estimates for real-world deployment. If every candidate label is wrong, the procedure still ranks one of them first. Out-of-domain evaluation, abstention rules, and calibration must be designed for the application rather than assumed from the base model.^[3][5]

Uses in later systems

Retrieval and evaluation

Because images and text share an embedding space, CLIP supports text-to-image and image-to-text retrieval without a separate cross-encoder. A system can precompute features, compare them with cosine similarity, and rank the nearest candidates. This is an efficient use of the representation, but retrieval quality still depends on the domain, the checkpoint, and the text used as a query.^[1][3]

CLIPScore repurposed CLIP similarity as a reference-free metric for image captioning. In the authors' experiments, it correlated with human judgments better than the reference-based CIDEr and SPICE metrics across the studied corpora. Combining it with reference captions produced RefCLIPScore and improved correlation further. The study also identified a weaker setting: news captions that require context beyond what appears in the image.^[11] CLIPScore should therefore be treated as one evaluation signal, not a universal measure of caption quality.

Image generation

DALL-E 2, described in the unCLIP paper, used a prior to generate a CLIP image embedding from a caption and a diffusion decoder to generate an image from that embedding. Its CLIP model remained frozen while the prior and decoder were trained. The paper's appendix specifies a separately trained ViT-H/16 image encoder at 256-pixel resolution, so the system should not be described as simply using one of OpenAI's nine public CLIP checkpoints.^[12]

Version 1 of Stable Diffusion used a fixed pretrained CLIP ViT-L/14 text encoder to condition a latent diffusion model. The official CompVis documentation describes an 860-million-parameter U-Net and a 123-million-parameter text encoder, with prompts represented by non-pooled CLIP text embeddings.^[13] This is a use of CLIP's text tower for conditioning; the diffusion model does not use CLIP's image encoder to synthesize pixels.

Multimodal assistants

The original LLaVA architecture used a pretrained CLIP ViT-L/14 visual encoder, a trainable linear projection, and a large language model. The first training stage froze both the visual encoder and language model and optimized only the projection. The second stage kept the CLIP encoder frozen while updating the projection and language model.^[14] This illustrates how a CLIP image tower can serve as a reusable visual tokenizer, but later LLaVA releases use different encoders and training recipes.

Adaptation and related methods

Prompt learning

Context Optimization, or CoOp, replaces hand-written templates with continuous prompt vectors learned from labeled examples while keeping the pretrained CLIP model fixed. The paper reported that learned contexts improved average few-shot classification across its 11 datasets, especially with more training examples. A later base-to-new-class evaluation found that CoOp could overfit the base classes.^[15][16]

Conditional Context Optimization, or CoCoOp, extends that idea by generating an input-dependent token and appending it to the learned context. Its study reported better generalization from base classes to unseen classes than CoOp, with a tradeoff in base-class performance.^[16] Both methods adapt the classifier interface around a frozen CLIP representation; neither changes what the original WIT-trained model learned.

Open data replications

OpenCLIP is an independent open-source implementation used to train CLIP-like models on public LAION datasets. A 2023 scaling study trained models on datasets containing up to two billion image-text pairs and found power-law relationships among performance, compute, and data. It also found that OpenAI CLIP and OpenCLIP followed different scaling trends, which the authors attributed to differences in training distributions.^[17]

MetaCLIP examined data curation rather than introducing a new two-tower architecture. It used metadata derived from WordNet concepts and Wikipedia entries to balance web data. In its controlled 400-million-pair comparison, a ViT-B/16 model reached 70.8 percent ImageNet zero-shot accuracy, compared with 68.3 percent for the paper's evaluation of OpenAI's ViT-B/16.^[18] These figures belong to that paper's setup and do not show that any metadata filter will produce the same gain.

DataComp created a benchmark for selecting training data from a common pool while holding model and training choices fixed at several compute scales. Its DataComp-1B dataset contained 1.4 billion pairs selected from a 12.8-billion-pair pool. A ViT-L/14 trained under the benchmark reached 79.2 percent ImageNet accuracy, 3.7 points above the OpenAI ViT-L/14 comparison in the paper under matched training procedure and compute.^[19] The result reinforces that data selection can materially change a CLIP-like model even when the architecture and loss are held constant.

Alternative loss

SigLIP replaces CLIP's batch-normalized softmax contrastive loss with an independent sigmoid loss over image-text pairs. Because each pair is scored independently, training does not require a global normalization over every example in a distributed batch. The authors reported stronger results at smaller batch sizes and diminishing returns beyond 32,000 examples per batch in their ablations.^[20] SigLIP is related to CLIP, but its objective and trained checkpoints are distinct and should not be labeled as original OpenAI CLIP models.

Availability and reproducibility

OpenAI's repository provides model-loading code, preprocessing, tokenization, zero-shot classification examples, linear-probe examples, and the nine named checkpoint downloads. The reference implementation is licensed under the MIT License.^[4] The model card documents staged checkpoint releases from the initial RN50 and ViT-B/32 through ViT-L/14@336px.^[3]

Full reproduction of the original models remains unavailable because WIT and its collection pipeline were not released. Public efforts such as OpenCLIP, MetaCLIP, and DataComp reproduce the broad two-tower approach with different data. Their results help test which findings transfer, but they do not recreate the exact corpus, weights, or training history of OpenAI's checkpoints.^[17][18][19]

See also

• Contrastive learning
• Computer vision
• Natural language processing
• Prompt engineering
• Diffusion model

References

1. ^Radford, A., et al. "Learning Transferable Visual Models From Natural Language Supervision." Proceedings of the 38th International Conference on Machine Learning, 2021. proceedings.mlr.press/...radford21a
2. ^Radford, A., et al. "Learning Transferable Visual Models From Natural Language Supervision: Supplementary Material." Proceedings of the 38th International Conference on Machine Learning, 2021. proceedings.mlr.press/...radford21a-supp.pdf
3. ^OpenAI. "Model Card: CLIP." GitHub. github.com/...model-card.md
4. ^OpenAI. "CLIP: Connecting Text and Images." GitHub repository. github.com/...CLIP
5. ^Agarwal, S., et al. "Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications." arXiv, 2021. arxiv.org/...2108.02818
6. ^Fang, A., et al. "Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)." Proceedings of the 39th International Conference on Machine Learning, 2022. proceedings.mlr.press/...fang22a
7. ^Goh, G., et al. "Multimodal Neurons in Artificial Neural Networks." Distill, 2021. distill.pub/...multimodal-neurons
8. ^Thrush, T., et al. "Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. openaccess.thecvf.com/...tionality_CVPR_2022_paper
9. ^Yuksekgonul, M., et al. "When and Why Vision-Language Models Behave Like Bags-of-Words, and What to Do About It?" International Conference on Learning Representations, 2023. arxiv.org/...2210.01936
10. ^Hsieh, C.-Y., et al. "SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality." Advances in Neural Information Processing Systems, 2023. proceedings.neurips.cc/...-Datasets_and_Benchmarks
11. ^Hessel, J., et al. "CLIPScore: A Reference-free Evaluation Metric for Image Captioning." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021. aclanthology.org/2021.emnlp-main.595
12. ^Ramesh, A., et al. "Hierarchical Text-Conditional Image Generation with CLIP Latents." arXiv, 2022. arxiv.org/...2204.06125
13. ^CompVis. "Stable Diffusion." GitHub repository. github.com/...stable-diffusion
14. ^Liu, H., et al. "Visual Instruction Tuning." Advances in Neural Information Processing Systems, 2023. proceedings.neurips.cc/...6de0-Abstract-Conference
15. ^Zhou, K., et al. "Learning to Prompt for Vision-Language Models." International Journal of Computer Vision, 2022. arxiv.org/...2109.01134
16. ^Zhou, K., et al. "Conditional Prompt Learning for Vision-Language Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. openaccess.thecvf.com/...ge_Models_CVPR_2022_paper
17. ^Cherti, M., et al. "Reproducible Scaling Laws for Contrastive Language-Image Learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. openaccess.thecvf.com/..._Learning_CVPR_2023_paper
18. ^Xu, H., et al. "Demystifying CLIP Data." International Conference on Learning Representations, 2024. arxiv.org/...2309.16671
19. ^Gadre, S. Y., et al. "DataComp: In Search of the Next Generation of Multimodal Datasets." Advances in Neural Information Processing Systems, 2023. proceedings.neurips.cc/...-Datasets_and_Benchmarks
20. ^Zhai, X., et al. "Sigmoid Loss for Language Image Pre-Training." Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. openaccess.thecvf.com/...-Training_ICCV_2023_paper