# Image captioning

> Source: https://aiwiki.ai/wiki/image_captioning
> Updated: 2026-11-18
> Fact-checked: 2026-11-18
> Categories: Computer Vision, Multimodal AI, Natural Language Processing
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Image captioning." aiwiki.ai, 18 Nov 2026. https://aiwiki.ai/wiki/image_captioning
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Image captioning** is the task of automatically generating a natural-language description of the content of an image. It connects [computer vision](https://aiwiki.ai/wiki/computer_vision) and [natural language processing](https://aiwiki.ai/wiki/natural_language_processing): a system has to work out what a picture shows and then say it in a sentence that is both grammatical and accurate.[1][2] Since 2015 the task has usually been handled by a visual encoder that turns the image into features and a [language model](https://aiwiki.ai/wiki/language_model) that generates text from them, and both components have changed substantially since then.[2]

Modern image captioning took shape in late 2014 and 2015, when several groups showed that a [convolutional neural network](https://aiwiki.ai/wiki/convolutional_neural_network) paired with a [recurrent neural network](https://aiwiki.ai/wiki/recurrent_neural_network) could be trained end to end on image and caption pairs. Among them was Google's "Show and Tell" model, followed in 2015 by "Show, Attend and Tell", which added visual [attention](https://aiwiki.ai/wiki/attention).[1][3] The Microsoft [COCO](https://aiwiki.ai/wiki/coco_dataset) Captions dataset and its evaluation server, released in 2015, gave the field a shared benchmark, and a version of the [CIDEr](https://aiwiki.ai/wiki/cider_metric) metric became one of its official scores.[4][5] From around 2020, captioning was increasingly handled by large models pretrained on image-text pairs, and later by general [vision-language models](https://aiwiki.ai/wiki/vision_language_model).[6][7] Captioning models are now also used to write training data for [text-to-image models](https://aiwiki.ai/wiki/text-to-image_models) and for vision-language models themselves.[8][9][10]

## Task definition

Given an image $I$, a captioning model produces a sequence of words $S$ describing it. Neural captioning models are usually trained as conditional language models. Show and Tell, for example, chose the parameters $\theta$ that maximize the log probability of the correct description over the training set, $\theta^\star = \arg\max_\theta \sum_{(I,S)} \log p(S \mid I; \theta)$, and decomposed that probability word by word with the chain rule:[11]

$$\log p(S \mid I) = \sum_{t=0}^{N} \log p(S_t \mid I, S_0, \dots, S_{t-1})$$

At inference time the model generates one word at a time until it emits an end-of-sentence token. Show and Tell compared sampling with [beam search](https://aiwiki.ai/wiki/beam_search) and used beam search with a beam of 20 in its main experiments; greedy decoding (a beam of 1) lowered its results by 2 BLEU points on average.[11]

The standard task asks for one sentence describing a whole image, but several variants exist:

| Variant | What the system must produce | Example source |
|---|---|---|
| Sentence-level captioning | One short description of the whole image | MS COCO Captions[4] |
| Dense captioning | Descriptions for many localized regions of an image | DenseCap (2015)[12] |
| Novel object captioning | Captions that mention objects absent from caption training data | nocaps (2018)[13] |
| Text-aware captioning | Captions that recognize text in the image and relate it to its visual context | TextCaps (2020)[14] |
| Detailed or dense paragraph captioning | Long descriptions covering background, text, style and spatial relations | DOCCI, PixMo-Cap[15][10] |
| Detailed localized captioning | Long descriptions of a specified region in an image or video | Describe Anything (2025)[16] |

The DenseCap authors described dense captioning as a generalization of two older tasks: it becomes [object detection](https://aiwiki.ai/wiki/object_detection) when each description is a single word and becomes ordinary image captioning when one region covers the whole image.[12]

<figure class="embedvideo">
<iframe src="https://www.youtube-nocookie.com/embed/yk6XDFm3J2c" title="Andrej Karpathy - Automated Image Captioning with ConvNets and Recurrent Nets"></iframe>
<figcaption>"Andrej Karpathy - Automated Image Captioning with ConvNets and Recurrent Nets", video from MITCBMM, published October 5, 2015. <a href="https://www.youtube.com/watch?v=yk6XDFm3J2c">Watch on YouTube</a>.</figcaption>
</figure>

## History

### Before neural captioning

Generating language from visual data was studied long before deep learning, but mostly for video. The Show and Tell authors describe these early systems as combinations of hand-built visual recognizers and formal languages, such as And-Or graphs or logic systems, whose output was turned into text by rules; such systems were "heavily hand-designed" and worked only in narrow domains such as traffic scenes or sports.[11]

For still images, early work used object and attribute detectors to fill in templates. Farhadi and colleagues inferred a triplet of scene elements from detections and converted it to text with templates, Kulkarni and colleagues used a larger graph of detections with template-based generation, and other systems pieced together phrases about detected objects and relationships.[11] A second line of work treated description as retrieval: images and sentences were embedded in a shared vector space and the system returned existing sentences that lay close to the query image.[11] Micah Hodosh, Peter Young and Julia Hockenmaier framed sentence-based image annotation as ranking a pool of captions and introduced a benchmark of 8,000 images, each paired with five captions; Karpathy and Fei-Fei, among others, later used it under the name Flickr8K.[17][18] The Show and Tell authors noted that retrieval methods cannot describe new combinations of objects that never appeared together in the training data.[11]

### Encoder-decoder models (2014-2015)

Neural caption generators were inspired directly by [machine translation](https://aiwiki.ai/wiki/machine_translation). Sequence-to-sequence translation systems used one recurrent network to encode a source sentence into a fixed-length vector and a second to decode it into the target language; captioning replaced the source sentence with an image.[11][3] Kelvin Xu and colleagues credit Ryan Kiros and colleagues (2014) with the first neural caption generator, a multimodal log-bilinear model conditioned on image features.[3]

Several closely related models appeared within weeks of each other in late 2014:

| Model | First arXiv version | Authors (lead) | Approach |
|---|---|---|---|
| m-RNN | 2014-10-04 | Junhua Mao et al. | A recurrent sentence network and a convolutional image network joined in a multimodal layer[19] |
| Show and Tell (NIC) | 2014-11-17 | [Oriol Vinyals](https://aiwiki.ai/wiki/oriol_vinyals) et al. (Google) | CNN image encoder feeding an [LSTM](https://aiwiki.ai/wiki/lstm) decoder[1] |
| LRCN | 2014-11-17 | Jeff Donahue et al. | Recurrent convolutional networks for activity recognition, image description and video description[20] |
| Deep visual-semantic alignments | 2014-12-07 | [Andrej Karpathy](https://aiwiki.ai/wiki/andrej_karpathy), [Fei-Fei Li](https://aiwiki.ai/wiki/fei_fei_li) | Aligns sentence fragments with image regions, then generates region and image descriptions[18] |
| Show, Attend and Tell | 2015-02-10 | Kelvin Xu et al. | LSTM decoder with soft or hard attention over convolutional feature maps[3] |

Show and Tell, by Oriol Vinyals, Alexander Toshev, Samy Bengio and Dumitru Erhan, was presented at CVPR 2015 together with the Karpathy and Fei-Fei paper and LRCN.[1][18][20] Its Neural Image Caption (NIC) model used a CNN pretrained for image classification as the encoder and an LSTM as the decoder; the image was fed to the LSTM only once, at the first step, because feeding it at every step gave worse results in the authors' experiments.[11] The paper reported a BLEU-1 score of 59 on the PASCAL dataset against a previous best of 25 (human performance was about 69), improvements on Flickr30k (56 to 66) and SBU (19 to 28), and a BLEU-4 of 27.7 on COCO.[1] Karpathy and Fei-Fei used a CNN over image regions and a bidirectional RNN over sentences, aligned them through a multimodal embedding, and trained a multimodal RNN on the inferred alignments to describe both full images and image regions.[18]

### Visual attention

Show, Attend and Tell, by Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, [Aaron Courville](https://aiwiki.ai/wiki/aaron_courville), [Ruslan Salakhutdinov](https://aiwiki.ai/wiki/ruslan_salakhutdinov), Richard Zemel and [Yoshua Bengio](https://aiwiki.ai/wiki/yoshua_bengio), appeared at ICML 2015.[3] Instead of compressing the image into one vector, it kept a grid of features from a lower convolutional layer: the 14 x 14 x 512 feature map of an ImageNet-pretrained VGG network, flattened into 196 "annotation vectors".[3] At each step the decoder computed weights $\alpha_{t,i}$ over these vectors $a_i$, conditioned on its previous hidden state. In the deterministic "soft" version the context vector is the weighted average $\hat{z}_t = \sum_{i=1}^{L} \alpha_{t,i} a_i$, trainable by ordinary backpropagation; the stochastic "hard" version samples one location and is trained by maximizing a variational lower bound, which the authors note is equivalent to REINFORCE.[3] Visualizing the weights showed the model fixing its gaze on salient objects as it generated the corresponding words, and the model reached state-of-the-art results on Flickr8k, Flickr30k and MS COCO.[3]

In 2017 Peter Anderson and colleagues changed what attention operates over. Their bottom-up and top-down model used a [Faster R-CNN](https://aiwiki.ai/wiki/faster_r_cnn) detector to propose objects and other salient regions, each with its own feature vector, and a top-down mechanism to weight them. On the MS COCO test server it reached CIDEr, SPICE and BLEU-4 scores of 117.9, 21.5 and 36.9, a new state of the art, and the same features won the 2017 [visual question answering](https://aiwiki.ai/wiki/visual_question_answering) challenge.[21] In 2021 the VinVL authors still described the bottom-up and top-down model as the "most widely used" source of object-centric visual features for vision-language models.[22]

### The 2015 COCO captioning challenge

Microsoft described the COCO Captions dataset and its evaluation server in an April 2015 paper, and a captioning competition was held on the new dataset the same year.[4][11] Teams could make up to five submissions to the server on a large set of unseen test images, and the most promising 15 entries, together with a human baseline, were also judged by human raters. The two human metrics used to decide the winner were the percentage of captions rated as good as or better than a human caption (M1) and the percentage that passed a Turing test (M2).[11] Google's Show and Tell team and a team from [Microsoft Research](https://aiwiki.ai/wiki/microsoft_research) tied for first place.[11]

| Entry | CIDEr-D | BLEU-4 | Rank on automatic metrics | M1 | M2 | Rank on human judgment |
|---|---|---|---|---|---|---|
| Google (NIC) | 0.943 | 0.309 | 1st | 0.273 | 0.317 | 1st |
| MSR Captivator | 0.931 | 0.308 | 2nd | 0.250 | 0.301 | 3rd |
| m-RNN | 0.917 | 0.299 | 3rd | not in top 5 | not in top 5 | not in top 5 |
| MSR | 0.912 | 0.291 | 4th | 0.268 | 0.322 | 1st |
| Human captions | 0.854 | 0.217 | 8th | 0.638 | 0.675 | 1st |

*Source: Vinyals et al., Tables 9 and 10, using five reference captions per image for the automatic scores.[11]*

The results showed a gap between automatic metrics and people: human-written captions ranked eighth on the automatic leaderboard but first in the human evaluation, far ahead of every machine.[11] Using the 40-reference version of the test set, the Show and Tell team found that humans ranked 13th of 16 entries under BLEU-4, 6th under CIDEr and 3rd under METEOR.[11] The team attributed its own gains after the CVPR paper to a better image model with [batch normalization](https://aiwiki.ai/wiki/batch_normalization) (about 2 BLEU-4 points), a smaller beam (about 2), fine-tuning the CNN (about 1), scheduled sampling (about 1.5) and an [ensemble](https://aiwiki.ai/wiki/ensemble_learning) of models (about 1.5), and released an open-source TensorFlow implementation called im2txt.[11]

### Training on the metric

Captioning models are usually trained with a word-level cross-entropy loss but evaluated with sentence-level metrics that cannot be differentiated.[2][23] Self-critical sequence training (SCST), by Steven Rennie and colleagues at IBM, applied the REINFORCE [policy gradient](https://aiwiki.ai/wiki/policy_gradient) method to optimize the CIDEr score directly. Instead of learning a separate baseline to reduce variance, it used the score of the model's own greedy-decoded caption as the baseline. The paper reported that this approach raised the best CIDEr result on the MS COCO evaluation server from 104.9 to 114.7.[23] A 2021 survey of the field described using the CIDEr score as the reward, the strategy SCST introduced, as the most widely adopted reinforcement learning approach in captioning.[2]

### Vision-language pretraining (2018-2022)

Captioning models trained only on COCO saw a limited range of objects. In 2018 Google researchers released Conceptual Captions, about 3.3 million image and description pairs harvested from the alt-text HTML attribute of web images and passed through an automatic filtering pipeline. The pipeline replaced proper names with general terms, so a caption naming two film stars at a premiere became "actors attend the premiere at festival."[24] In the same year the nocaps benchmark tested whether models could describe objects they had not seen in caption training data. Built from Open Images, it contains 166,100 human captions for 15,100 images, and nearly 400 of the object classes in its test images have few or no training captions.[13]

By 2020, large-scale pretraining on image-text pairs had become a popular approach to vision-language tasks, including captioning, with models fine-tuned on each task after pretraining; a 2021 survey lists [BERT](https://aiwiki.ai/wiki/bert)-like early-fusion strategies among the main developments in captioning architectures.[6][2] Microsoft's Oscar (2020) used object tags detected in images as anchor points to align image regions with words, pretrained on 6.5 million image-text pairs, and set new results on six vision-language tasks including captioning.[6] VinVL (2021) kept the Oscar fusion model but replaced its visual features with those of a larger object detector trained on several combined detection datasets, improving results on seven benchmarks.[22] On 2020-10-14 Microsoft said that a captioning system built on VIVO, a pretraining method that learns from large datasets of image and tag pairs, had surpassed human CIDEr scores on nocaps for the first time.[25] The company said the model was being incorporated into its Seeing AI talking camera app and would roll out in Word, Outlook and PowerPoint, and that on another benchmark it was two times better than the captioning model used in Microsoft products since 2015.[26]

Later models moved away from external object detectors and toward larger, noisier web data:[27][28][29]

| Model | Year (arXiv) | Developer | Notable feature for captioning |
|---|---|---|---|
| SimVLM | 2021 | Google | Single prefix language modeling objective on weakly labeled web data; reported an average +10.1% CIDEr gain on captioning tasks[28] |
| BLIP | 2022 | Salesforce | A captioner writes synthetic captions for web images and a filter removes noisy ones; +2.8% CIDEr over previous best[29] |
| [Flamingo](https://aiwiki.ai/wiki/flamingo) | 2022 | DeepMind | Bridges pretrained vision-only and language-only models; adapts to captioning and question answering from a few examples in the prompt[30] |
| CoCa | 2022 | Google | One encoder-decoder trained with both a contrastive loss and a captioning loss[31] |
| GIT | 2022 | Microsoft | One image encoder and one text decoder under a single language modeling task; 138.2 CIDEr on TextCaps against a human score of 125.5[27] |
| PaLI | 2022 | Google | Joint scaling of a 4-billion-parameter ViT-e and a language model, pretrained on 10 billion images and texts in over 100 languages[32] |
| BLIP-2 | 2023 | Salesforce | A lightweight Querying Transformer connects a frozen image encoder to a frozen [large language model](https://aiwiki.ai/wiki/large_language_model)[7] |

BLIP-2 also showed what its authors called emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.[7]

### Large vision-language models and detailed captions (2023-2026)

General-purpose multimodal models brought image description into conversational assistants. In March 2023 Be My Eyes announced a "Virtual Volunteer" (later Be My AI), powered by OpenAI's [GPT-4](https://aiwiki.ai/wiki/gpt-4), to which blind and low-vision users could send images and ask questions; it was then in closed beta with a small group of users.[33]

Captioning also became a way to make training data. OpenAI's report on [DALL-E 3](https://aiwiki.ai/wiki/dall_e_3) argued that text-to-image models often ignore or confuse words in prompts because the captions in their training data are noisy and inaccurate. The team trained a dedicated image captioner, fine-tuned it to produce "descriptive synthetic captions" covering the main subject, surroundings, background, visible text, style and color, recaptioned its whole training set, and found that training on these captions reliably improved prompt following; its experiments used blends of up to 95% synthetic captions.[8] For vision-language models, ShareGPT4V (2023) collected 100,000 detailed captions from GPT-4 Vision, trained a captioner on them to produce 1.2 million, and showed that swapping some of the GPT-4 Vision captions into fine-tuning data improved models such as LLaVA-1.5.[9] The Molmo team at the [Allen Institute for AI](https://aiwiki.ai/wiki/allen_institute_for_ai) took the opposite route with PixMo-Cap (2024), avoiding captions from other VLMs: annotators described 712,000 images aloud for at least 60 seconds (later at least 90 seconds), the audio was transcribed, and a language-only model cleaned up the transcripts. The resulting captions average 196 words, compared with 11 words for COCO captions.[10]

Other recent work has pushed captions toward more detail and finer control. Microsoft's [Florence-2](https://aiwiki.ai/wiki/florence_2) (2023) treats captioning, detection, grounding and segmentation as text-prompted tasks, trained on FLD-5B, a set of 5.4 billion annotations on 126 million images.[34] Google's DOCCI dataset (2024) provides human descriptions of 15,000 images that average 136 words, and a 5-billion-parameter PaLI model fine-tuned on it matched or beat larger models such as LLaVA-1.5 7B and InstructBLIP 7B.[15] NVIDIA's Describe Anything Model (2025) produces detailed descriptions of a user-specified region in an image or video.[16]

## Datasets

| Dataset | Year | Size | Caption source and notes |
|---|---|---|---|
| Flickr8k | 2013 | 8,000 images, 5 captions each | Flickr photos captioned on Mechanical Turk[17][4] |
| Flickr30k | 2014 | About 30,000 images, 150,000 captions | Flickr photos captioned on Mechanical Turk[35][4] |
| MS COCO Captions | 2015 | 413,915 training captions (82,783 images), 202,520 validation (40,504), 379,249 test (40,775) | Five captions per image from [Amazon Mechanical Turk](https://aiwiki.ai/wiki/amazon_mechanical_turk); c40 subset of 5,000 test images with 40 captions each[4] |
| Visual Genome (as used by DenseCap) | 2015 (DenseCap) | 94,000 images, 4.1 million region captions | Region-grounded phrases for dense captioning[12] |
| Conceptual Captions | 2018 | About 3.3 million pairs | Filtered and hypernymized web alt-text[24] |
| nocaps | 2018 | 15,100 images, 166,100 captions | Open Images validation and test images; nearly 400 classes rare or absent in COCO captions[13] |
| VizWiz-Captions | 2020 | Over 39,000 images, 5 captions each | Photos taken by blind people[36] |
| ShareGPT4V | 2023 | 1.2 million captions | 100,000 from GPT-4 Vision, the rest from a captioner trained on them[9] |
| DOCCI | 2024 | 15,000 images | Human descriptions averaging 136 words[15] |
| PixMo-Cap | 2024 | 712,000 images, 1.3 million transcripts and captions | Spoken descriptions, transcribed and cleaned; average 196 words[10] |

COCO's annotators were told to describe all the important parts of the scene, not to start sentences with "There is", not to describe unimportant details, things that might have happened in the past or future, or what a person might say, not to give people proper names, and to write at least eight words.[4] The reference captions for the COCO test set are kept private to reduce overfitting, so test results come from the evaluation server.[4] Karpathy and Fei-Fei held out 5,000 COCO images for validation and 5,000 for testing in their experiments, and 1,000 each for Flickr8k and Flickr30k.[18]

The VizWiz-Captions authors argued that few public captioning datasets reflect the needs of real users. Their images were taken by blind people who, the authors note, had relied on human-based captioning services for nearly a decade; the authors compared the content with eight popular vision datasets and analyzed what makes the new dataset challenging for modern captioning algorithms.[36]

## Evaluation

### Reference-based metrics

Most captioning benchmarks score a generated caption against several human references. The COCO evaluation server reports BLEU-1 to BLEU-4, [METEOR](https://aiwiki.ai/wiki/meteor_metric), [ROUGE-L](https://aiwiki.ai/wiki/rouge_score) and CIDEr-D on both the five-reference and the 40-reference test sets, after tokenizing all captions with the Stanford PTBTokenizer and removing punctuation.[4] [BLEU](https://aiwiki.ai/wiki/bleu_bilingual_evaluation_understudy) and METEOR came from [machine translation](https://aiwiki.ai/wiki/machine_translation) evaluation and ROUGE from text summarization.[4]

| Metric | Origin | What it measures |
|---|---|---|
| BLEU | Machine translation | [N-gram](https://aiwiki.ai/wiki/n-gram) precision against references, with a brevity penalty[4] |
| METEOR | Machine translation | Alignment with references by exact tokens, WordNet synonyms, stems and paraphrases[4] |
| ROUGE-L | Summarization | Longest common subsequence with references[4] |
| CIDEr / CIDEr-D | Image description (2015) | Consensus with many references, using [TF-IDF](https://aiwiki.ai/wiki/tf_idf) weighted n-grams[5][4] |
| SPICE | Image captioning (2016) | Overlap of objects, attributes and relations in scene graphs[37] |
| CLIPScore | Image captioning (2021) | Image-caption compatibility measured by [CLIP](https://aiwiki.ai/wiki/clip), with no references needed[38] |
| CHAIR | Image captioning (2018) | Share of mentioned objects that are not in the image, or of captions that contain one[39] |

CIDEr was introduced by Ramakrishna Vedantam, C. Lawrence Zitnick and Devi Parikh to measure how well a caption matches the consensus of human descriptions, and a version named CIDEr-D runs on the COCO server.[5] SPICE, from Peter Anderson and colleagues, argued that n-gram overlap is "neither necessary nor sufficient" for simulating human judgment and compared captions through scene graphs instead. Its authors reported a system-level correlation with human judgments on COCO of 0.88, against 0.43 for CIDEr and 0.53 for METEOR.[37]

### Reference-free and model-based metrics

[CLIP Score](https://aiwiki.ai/wiki/clip_score), proposed by Jack Hessel and colleagues in 2021, uses CLIP, pretrained on 400 million web image-caption pairs, to judge how well a caption fits an image without any human references. The authors found it correlated with human judgments better than CIDEr and SPICE on several corpora, and a variant that also uses references, RefCLIPScore, did better still; it was weaker on news captions that require contextual knowledge.[38]

Evaluating long, detailed captions is harder; the authors of DeCapBench (2025) called it underexplored because of outdated metrics and coarse annotations.[40] CapArena (2025) ran more than 6,000 pairwise caption comparisons with human preference votes and found that while some metrics such as METEOR agree with people reasonably well on individual captions, their systematic biases lead to inconsistent model rankings; a [VLM used as a judge](https://aiwiki.ai/wiki/llm_as_a_judge) ranked models more reliably.[41] DeCapBench and its DCScore metric (ICLR 2025) break a detailed caption into "primitive information units" and check each for hallucination and coverage.[40]

### Human evaluation

Automatic metrics are judged by how well they agree with human ratings.[37][38][41] In the 2015 COCO challenge, human raters scored captions on whether they were as good as a human caption, whether they could pass for human-written, correctness, level of detail, and similarity to a human description.[11] The CapArena study concluded that leading models such as [GPT-4o](https://aiwiki.ai/wiki/gpt_4o) matched or exceeded human performance on detailed captioning in its arena, while most open-source models lagged behind.[41]

## Applications

- **Accessibility.** Facebook introduced automatic alternative text in April 2016, starting with iOS screen readers set to English. It used object recognition to tell screen-reader users what a photo may contain, in the form "Image may contain three people, smiling, outdoors", where they previously heard only the poster's name and the word "photo".[42] Microsoft added its 2020 captioning model to the Seeing AI talking camera app, and Be My Eyes launched Be My AI on GPT-4 to answer blind and low-vision users' questions about their photos.[26][33]
- **Training data for generative models.** Synthetic captions are used to recaption image collections for text-to-image models such as DALL-E 3 and to build pretraining and fine-tuning data for vision-language models.[8][9][10] BLIP used its own captioner and filter to clean noisy web data during pretraining.[29]
- **Retrieval and search.** Early captioning research was closely tied to sentence-based image search, and several early models were evaluated on both caption generation and image-sentence retrieval.[17][18][19]
- **Building block for multimodal models.** Several pretrained vision-language models use captioning as a training objective: SimVLM and GIT train with a single language modeling objective, and CoCa combines a captioning loss with a contrastive loss.[28][27][31]

## Limitations and open problems

**Object hallucination.** Anna Rohrbach, Lisa Anne Hendricks and colleagues showed in 2018 that captioning models often mention objects that are not in the scene. They proposed the CHAIR metric, which counts the fraction of mentioned COCO objects (CHAIRi) or of sentences (CHAIRs) that contain a hallucinated object, and found that the models scoring best on standard sentence metrics do not always hallucinate less, and that models which hallucinate more tend to make errors driven by language priors.[39] Hallucination remains a concern for detailed captions: DCScore was designed partly to measure it, and its authors reported that preference optimization based on the metric significantly reduced hallucinations.[40]

**Metric reliability.** Automatic scores have repeatedly disagreed with people, from human captions ranking eighth on the 2015 COCO leaderboard while winning the human evaluation, to the model-ranking inconsistencies CapArena found for traditional metrics on long captions.[11][41]

**Coverage of the visual world.** The nocaps authors argued that models trained on datasets with limited visual concepts would need to learn a much larger variety of concepts to work in the wild, and Conceptual Captions was built to offer a wider variety of images and caption styles than COCO.[13][24] The VizWiz-Captions authors noted that few public captioning datasets address the interests of real users, and found their collection of photos taken by blind people challenging for modern captioning algorithms.[36]

**Data provenance.** Many strong open models rely on synthetic captions from proprietary systems. The Molmo authors described this as "effectively distilling these closed VLMs into open ones" and built PixMo-Cap from human speech to avoid it.[10]

## See also

- [CIDEr](https://aiwiki.ai/wiki/cider_metric)
- [CLIP Score](https://aiwiki.ai/wiki/clip_score)
- [Visual question answering](https://aiwiki.ai/wiki/visual_question_answering)
- [Vision language model](https://aiwiki.ai/wiki/vision_language_model)
- [Multimodal model](https://aiwiki.ai/wiki/multimodal_model)
- [COCO](https://aiwiki.ai/wiki/coco_dataset)
- [Text-to-image models](https://aiwiki.ai/wiki/text-to-image_models)

## References

1. Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan. "Show and Tell: A Neural Image Caption Generator". arXiv (CVPR 2015), 2014-11-17. https://arxiv.org/abs/1411.4555
2. Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, Rita Cucchiara. "From Show to Tell: A Survey on Deep Learning-based Image Captioning". arXiv, 2021-07-14. https://arxiv.org/abs/2107.06912
3. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio. "Show, Attend and Tell: Neural Image Caption Generation with Visual Attention". arXiv (ICML 2015), 2015-02-10. https://arxiv.org/abs/1502.03044
4. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, C. Lawrence Zitnick. "Microsoft COCO Captions: Data Collection and Evaluation Server". arXiv, 2015-04-01. https://arxiv.org/abs/1504.00325
5. Ramakrishna Vedantam, C. Lawrence Zitnick, Devi Parikh. "CIDEr: Consensus-based Image Description Evaluation". arXiv (CVPR 2015), 2014-11-20. https://arxiv.org/abs/1411.5726
6. Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, et al. "Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks". arXiv (ECCV 2020), 2020-04-13. https://arxiv.org/abs/2004.06165
7. Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi. "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models". arXiv, 2023-01-30. https://arxiv.org/abs/2301.12597
8. James Betker, Gabriel Goh, Li Jing, Tim Brooks, et al. "Improving Image Generation with Better Captions". OpenAI, 2023. https://cdn.openai.com/papers/dall-e-3.pdf
9. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, et al. "ShareGPT4V: Improving Large Multi-Modal Models with Better Captions". arXiv, 2023-11-21. https://arxiv.org/abs/2311.12793
10. Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, et al. "Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models". arXiv, 2024-09-25. https://arxiv.org/abs/2409.17146
11. Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan. "Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge". arXiv (IEEE Transactions on Pattern Analysis and Machine Intelligence), 2016-09-21. https://arxiv.org/abs/1609.06647
12. Justin Johnson, Andrej Karpathy, Li Fei-Fei. "DenseCap: Fully Convolutional Localization Networks for Dense Captioning". arXiv, 2015-11-24. https://arxiv.org/abs/1511.07571
13. Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, et al. "nocaps: novel object captioning at scale". arXiv, 2018-12-20. https://arxiv.org/abs/1812.08658
14. Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, Amanpreet Singh. "TextCaps: a Dataset for Image Captioning with Reading Comprehension". arXiv, 2020-03-24. https://arxiv.org/abs/2003.12462
15. Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, et al. "DOCCI: Descriptions of Connected and Contrasting Images". arXiv, 2024-04-30. https://arxiv.org/abs/2404.19753
16. Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, et al. "Describe Anything: Detailed Localized Image and Video Captioning". arXiv, 2025-04-22. https://arxiv.org/abs/2504.16072
17. Micah Hodosh, Peter Young, Julia Hockenmaier. "Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics". Journal of Artificial Intelligence Research, vol. 47, 2013. https://doi.org/10.1613/jair.3994
18. Andrej Karpathy, Li Fei-Fei. "Deep Visual-Semantic Alignments for Generating Image Descriptions". arXiv (CVPR 2015), 2014-12-07. https://arxiv.org/abs/1412.2306
19. Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Alan L. Yuille. "Explain Images with Multimodal Recurrent Neural Networks". arXiv, 2014-10-04. https://arxiv.org/abs/1410.1090
20. Jeff Donahue, Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, Trevor Darrell. "Long-term Recurrent Convolutional Networks for Visual Recognition and Description". arXiv (CVPR 2015), 2014-11-17. https://arxiv.org/abs/1411.4389
21. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang. "Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering". arXiv (CVPR 2018), 2017-07-25. https://arxiv.org/abs/1707.07998
22. Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, et al. "VinVL: Revisiting Visual Representations in Vision-Language Models". arXiv, 2021-01-02. https://arxiv.org/abs/2101.00529
23. Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, Vaibhava Goel. "Self-critical Sequence Training for Image Captioning". arXiv (CVPR 2017), 2016-12-02. https://arxiv.org/abs/1612.00563
24. Piyush Sharma, Nan Ding, Sebastian Goodman, Radu Soricut. "Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning". Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, July 2018. https://aclanthology.org/P18-1238/
25. Kevin Lin, Xiaowei Hu, Lijuan Wang. "Novel object captioning surpasses human performance on benchmarks". Microsoft Research Blog, 2020-10-14. https://www.microsoft.com/en-us/research/blog/novel-object-captioning-surpasses-human-performance-on-benchmarks
26. John Roach. "What's that? Microsoft's latest breakthrough, now in Azure AI, describes images as well as people do". Microsoft Source, 2020-10-14. https://news.microsoft.com/source/features/ai/azure-image-captioning/
27. Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, et al. "GIT: A Generative Image-to-text Transformer for Vision and Language". arXiv, 2022-05-27. https://arxiv.org/abs/2205.14100
28. Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, Yuan Cao. "SimVLM: Simple Visual Language Model Pretraining with Weak Supervision". arXiv (ICLR 2022), 2021-08-24. https://arxiv.org/abs/2108.10904
29. Junnan Li, Dongxu Li, Caiming Xiong, Steven Hoi. "BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation". arXiv, 2022-01-28. https://arxiv.org/abs/2201.12086
30. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, et al. "Flamingo: a Visual Language Model for Few-Shot Learning". arXiv (NeurIPS 2022), 2022-04-29. https://arxiv.org/abs/2204.14198
31. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, Yonghui Wu. "CoCa: Contrastive Captioners are Image-Text Foundation Models". arXiv, 2022-05-04. https://arxiv.org/abs/2205.01917
32. Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, et al. "PaLI: A Jointly-Scaled Multilingual Language-Image Model". arXiv (ICLR 2023), 2022-09-14. https://arxiv.org/abs/2209.06794
33. Be My Eyes. "Introducing Be My AI (formerly Virtual Volunteer) for People who are Blind or Have Low Vision, Powered by OpenAI's GPT-4". Be My Eyes, 2023-03-14. https://www.bemyeyes.com/news/introducing-be-my-ai-formerly-virtual-volunteer-for-people-who-are-blind-or-have-low-vision-powered-by-openais-gpt-4/
34. Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, et al. "Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks". arXiv, 2023-11-10. https://arxiv.org/abs/2311.06242
35. Peter Young, Alice Lai, Micah Hodosh, Julia Hockenmaier. "From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions". Transactions of the Association for Computational Linguistics, vol. 2, 2014. https://aclanthology.org/Q14-1006/
36. Danna Gurari, Yinan Zhao, Meng Zhang, Nilavra Bhattacharya. "Captioning Images Taken by People Who Are Blind". arXiv, 2020-02-20. https://arxiv.org/abs/2002.08565
37. Peter Anderson, Basura Fernando, Mark Johnson, Stephen Gould. "SPICE: Semantic Propositional Image Caption Evaluation". arXiv (ECCV 2016), 2016-07-29. https://arxiv.org/abs/1607.08822
38. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, Yejin Choi. "CLIPScore: A Reference-free Evaluation Metric for Image Captioning". arXiv, 2021-04-18. https://arxiv.org/abs/2104.08718
39. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, Kate Saenko. "Object Hallucination in Image Captioning". arXiv (EMNLP 2018), 2018-09-06. https://arxiv.org/abs/1809.02156
40. Qinghao Ye, Xianhan Zeng, Fu Li, Chunyuan Li, Haoqi Fan. "Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning". arXiv (ICLR 2025), 2025-03-10. https://arxiv.org/abs/2503.07906
41. Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, et al. "CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era". arXiv, 2025-03-16. https://arxiv.org/abs/2503.12329
42. Shaomei Wu, Hermes Pique, Jeffrey Wieland. "Using Artificial Intelligence to Help Blind People 'See' Facebook". Meta Newsroom, 2016-04-04. https://about.fb.com/news/2016/04/using-artificial-intelligence-to-help-blind-people-see-facebook/

