Visual question answering

Visual question answering (VQA) is the task of producing a natural-language answer to a natural-language question about an image. A system receives a picture and a free-form question such as "What color is the umbrella?" or "Is the person wearing glasses?" and must return an answer, usually a single word or short phrase. The task was proposed in its modern open-ended form by Stanislaw Antol and colleagues at ICCV 2015, who argued that answering arbitrary questions demands a more detailed understanding of an image than captioning does, while still allowing automatic scoring ^[1].

From 2015 through the early 2020s, VQA was pursued as a standalone research problem with purpose-built architectures. That changed once general vision-language models became capable enough to answer image questions as one behavior among many. VQA datasets did not disappear; they were repurposed as evaluation suites for multimodal models, and the research effort moved toward harder variants that require reading text, retrieving outside knowledge, or reasoning over college-level diagrams.

The task also has a direct practical motivation. The original VQA paper named assistance for blind and low-vision users as a target application ^[1], and that use case has driven several of the field's most-used datasets as well as the first large consumer deployment of an image-question system.

How the task is scored

The standard VQA setup collects ten independent human answers per question and scores a candidate answer against them. The official metric is

accuracy = min(number of humans who gave that answer / 3, 1)

so an answer earns full credit once at least three of ten annotators produced it, and partial credit below that ^[1]. To keep machine scores comparable to human scores, accuracy is averaged over all ten-choose-nine subsets of the annotators ^[3]. Before matching, both predicted and reference answers are normalized: lowercased, stripped of articles and most punctuation, with number words converted to digits and missing apostrophes restored ^[3]. The original benchmark also offered a multiple-choice variant alongside the open-ended one ^[1].

Later datasets replaced this metric where it did not fit. DocVQA scores with Average Normalized Levenshtein Similarity (ANLS) so that near-miss strings caused by OCR errors are not penalized as hard failures ^[13]. ChartQA uses a relaxed accuracy that tolerates small numeric deviations ^[15]. GQA added five diagnostic measures beyond accuracy: consistency across logically entailed questions, validity (is the answer the right type?), plausibility (is it reasonable at all?), grounding (did attention land on relevant regions?), and distribution (does the predicted answer distribution match the reference?) ^[9].

Exact-match scoring sits awkwardly on generative models, which tend to answer in full sentences. The usual workaround is a response-format instruction appended to the question. The LLaVA-1.5 authors found that adding a short response-formatting prompt, such as a request to answer using a single word or phrase, was enough for an instruction-tuned model to score properly on short-answer VQA benchmarks ^[29].

Origins

The first widely cited attempt at the task predates the VQA name. In 2014 Mateusz Malinowski and Mario Fritz introduced DAQUAR along with a multi-world Bayesian method that represented uncertainty about the perceived scene, describing the benchmark as "a modern attempt at a visual turing test" ^[4].

The 2015 VQA dataset set the template that followed. It drew 123,287 training and validation images plus 81,434 test images from MS COCO, added 50,000 synthetic abstract scenes, collected three questions per image from separate crowd workers, and gathered ten answers per question, producing roughly 760,000 questions and 10 million answers ^[1]. Humans scored 83.30% on the open-ended real-image split. The paper's best model, a deeper LSTM over the question combined with normalized image features, reached 58.16% on the open-ended test-standard split and 63.09% in multiple-choice mode ^[1]. That gap defined the research agenda for the next several years.

Language priors and the balancing problem

The early benchmark had a structural flaw: a large share of its questions could be answered without looking at the image. Answer distributions were skewed enough that a model memorizing question-type priors did well. In the VQA v2 paper, Yash Goyal and colleagues quantified it: "tennis" is correct for 41% of questions beginning "What sport is", "2" answers 39% of questions beginning "How many", and simply replying "yes" to every question starting "Do you see a" scores 87% ^[5].

Their fix was to rebalance by construction. For each question, annotators found a second, visually similar image for which the same question has a different answer, so every question is tied to a complementary image pair. The result, VQA v2, holds about 1.1 million question-image pairs over 204,721 COCO images, roughly double the original ^[5][2]. Models scored several points lower on the balanced data than on the original: MCB fell from 60.36% to 54.22%, HieCoAtt from 57.09% to 50.31%, and a deeper-LSTM baseline from 54.40% to 47.56% ^[5]. VQA v2 became the default training and evaluation set and remains in use.

Two complementary probes followed. Aishwarya Agrawal and colleagues built VQA-CP (Changing Priors), which reshuffles the splits so that the answer distribution for each question type differs between train and test, exposing models that lean on priors; their GVQA model separated visual concept recognition from answer-space prediction and held up far better under that shift ^[6]. Justin Johnson and colleagues took the opposite approach with CLEVR, a fully synthetic diagnostic set of rendered scenes with minimal bias and per-question annotations of the reasoning required. CLEVR contains 100,000 rendered images and 999,968 questions in total, of which 70,000 images and 699,989 questions are training data and 15,000 images and 149,991 questions are validation data ^[7][8].

The problem was reduced rather than solved. In 2024 the MMStar authors found that Gemini Pro scored 42.9% on MMMU with no visual input at all, and that Sphinx-X-MoE reached 43.6% on the same benchmark without images, beating its own language backbone by 17.9 points, which points to data contamination as well as language priors ^[33].

Datasets

Scroll sideways for more →

Knowledge, text, and documents

OK-VQA, introduced by Kenneth Marino and colleagues in 2019, keeps only questions whose answers require information not present in the picture, sorted into ten knowledge categories such as cooking and food, science and technology, and weather and climate ^[10][11]. The strongest VQA models of the day landed in the mid-20s in accuracy, a collapse relative to their VQA v2 scores ^[10].

TextVQA came from the observation that a dominant class of questions asked by visually impaired users involves reading text in the image, something the VQA models of 2019 simply could not do ^[12]. Amanpreet Singh and colleagues released 45,336 questions on 28,408 images and a model called LoRRA (Look, Read, Reason and Answer) that fed OCR tokens into the answer space. The human-machine gap was far wider than on VQA v2: humans scored 85.01% on the TextVQA validation split against 26.56% for LoRRA ^[12].

DocVQA moved the task to scanned business documents from the UCSF Industry Documents Library, with roughly 50,000 questions over more than 12,000 document images and answers always extractable as a span of text from the page ^[13][14]. Humans scored 94.36% on the test set against 55.77% for the paper's best baseline, a BERT-large model fine-tuned on SQuAD and DocVQA ^[13]. ChartQA extended the same idea to data graphics, pairing 9,600 human-authored questions with 23,100 questions derived from human-written chart summaries and emphasizing questions that need several logical or arithmetic steps ^[15].

College-level and diagnostic benchmarks

MMMU, released in late 2023 by Xiang Yue and colleagues, assembles 11,550 college-level questions across six disciplines, 30 subjects, and 183 subfields, drawn from exams, quizzes, and textbooks and annotated by over 50 university students. Its images span 30 heterogeneous types including charts, maps, music sheets, and chemical structures ^[17][18]. At release, GPT-4V scored 56.8% on the validation split and 55.7% on test, and Gemini Ultra reached 59.4% on validation ^[17].

MMMU-Pro, published in September 2024, hardens the benchmark in three steps: it removes questions that text-only models can answer, expands the candidate answer options, and adds a vision-only setting in which the question itself is embedded inside the image. Model scores dropped by 16.8 to 26.9 points relative to MMMU ^[19].

Architectures

Joint embeddings and attention

The first generation encoded the image with a convolutional network and the question with an LSTM, fused the two vectors, and classified over a fixed answer vocabulary ^[1]. Zichao Yang and colleagues added Stacked Attention Networks in 2015, using the question representation as a query to select image regions and stacking several attention layers so the model could refine its focus over multiple steps of reasoning ^[20].

Peter Anderson and colleagues then changed what attention operates over. Instead of a uniform grid of CNN features, their bottom-up and top-down model used a Faster R-CNN object detector with a ResNet-101 backbone to propose salient regions, and applied question-conditioned weighting across those object-level proposals. An ensemble of 30 such models won first place in the 2017 VQA Challenge with 70.34% on the VQA v2 test-standard server, and a four-model ensemble of the same approach reached 117.9 CIDEr on the MSCOCO captioning test server ^[21]. Region features from this pipeline became the standard visual input for years afterward.

Vision-language pretraining

In 2019 the BERT recipe crossed over to vision and language. ViLBERT, from Jiasen Lu and colleagues, processed image regions and text in two separate streams that exchanged information through co-attentional transformer layers, pretrained on Conceptual Captions with proxy objectives, then fine-tuned on visual question answering, visual commonsense reasoning, referring expressions, and caption-based retrieval ^[22]. LXMERT, from Hao Tan and Mohit Bansal, used three encoders (an object relationship encoder, a language encoder, and a cross-modality encoder built from bi-directional cross-attention sub-layers) and five pretraining tasks including image question answering itself. Pretrained on 9.18 million image-and-sentence pairs over 180,000 distinct images, it reached 72.5% on VQA v2 test-standard, 60.3% on GQA test-standard, and 76.2% on NLVR2 ^[23].

Scaling this line further produced the high-water mark for task-specific VQA systems. BEiT-3, a Multiway Transformer pretrained with masked modeling over images, text, and image-text pairs, reported 84.19% on VQA v2 test-dev and 84.03% on test-standard, ahead of CoCa at 82.30%, Flamingo at 82.10%, and OFA at 82.00% ^[24].

Unified vision-language models

The current generation treats VQA as one prompt format for a general model rather than a task to train for. BLIP (2022) bootstrapped noisy web captions with a captioner-and-filter loop and improved VQA score by 1.6 points over its predecessors ^[26]. Flamingo, also 2022, bridged a frozen vision encoder and a frozen language model, handled arbitrarily interleaved images and text, and answered open-ended visual questions from a handful of in-context examples, outperforming systems fine-tuned on thousands of times more task data ^[25]. BLIP-2 (2023) made that bridge cheap with a lightweight Querying Transformer between a frozen vision encoder and a frozen LLM, beating Flamingo-80B by 8.7 points on zero-shot VQAv2 with 54 times fewer trainable parameters ^[27].

LLaVA closed the loop with instruction tuning. Haotian Liu and colleagues used language-only GPT-4 to generate multimodal instruction-following data, then trained a linear projection connecting a CLIP vision encoder to an LLM. The result scored 85.1% relative to GPT-4 on a synthetic multimodal instruction set and, combined with GPT-4, reached 92.53% on ScienceQA ^[28]. LLaVA-1.5 swapped in a CLIP ViT-L at 336 pixels with an MLP projection and added academic VQA data plus response-format prompts, taking the lead on 11 benchmarks with 1.2 million public samples and about one day of training on a single eight-GPU A100 node ^[29]. That recipe, a vision transformer plus a projector plus an instruction-tuned LLM, underpins most open multimodal models since, including InternVL, Qwen3-VL, PaliGemma, and Molmo.

Proprietary systems followed the same trajectory. OpenAI published the GPT-4V(ision) system card on 25 September 2023, describing a model first trained to predict the next word in a document over a large dataset of text and image data, then fine-tuned with reinforcement learning from human feedback ^[30]. GPT-4o, Gemini 3 Pro, and their contemporaries answer image questions as ordinary chat turns.

Applications

Accessibility remains the field's clearest application. VizWiz, released in 2018 by Danna Gurari and colleagues, was the first goal-oriented VQA dataset from a natural setting: over 31,000 visual questions in which blind people photographed something with a phone and recorded a spoken question about it, each paired with ten crowdsourced answers. Its images are often poorly framed or blurred, its questions are conversational, and a substantial share of them are unanswerable from the photo ^[16].

The first large deployment came from Be My Eyes, an organization that builds tools for visually impaired users. Beginning in March 2023 it worked with OpenAI to build Be My AI, which incorporated GPT-4V into the existing Be My Eyes platform. According to OpenAI's system card, the pilot ran from March to early August 2023 with nearly 200 blind and low-vision beta testers; by September the beta group had grown to 16,000 users requesting a daily average of 25,000 descriptions, inside a product with 500,000 blind and low-vision users ^[30]. Be My Eyes warns its users not to rely on the system for safety and health questions such as reading prescriptions, checking ingredient lists for allergens, or crossing the street, and lets them leave the AI session to connect with a human volunteer instead ^[30].

Beyond accessibility, document and chart question answering supports enterprise document processing ^[13][15]. Consumer exposure brought its own problems. OpenAI analyzed a sample of GPT-4V alpha traffic to understand how people were using the model for medical advice, person identification, and CAPTCHA solving. It found that 20% of sampled prompts were general requests to explain or describe an image, and it named medical advice as one of six areas prioritized for external red teaming ^[30].

Limitations and open problems

Hallucination is the most persistent failure. Yifan Li and colleagues introduced POPE, a polling-based query method that asks a model whether specific objects are present, and found that large vision-language models frequently assert objects that are not in the image, with errors concentrated on objects that are frequent in training data or that commonly co-occur with what is actually shown ^[31]. The same behavior appeared in deployment. One Be My AI beta tester quoted in OpenAI's system card said of the product: "It very confidently told me there was an item on a menu that was in fact not there." ^[30] This is the multimodal form of the hallucination problem in language models.

Perception itself is weaker than benchmark scores suggest. Shengbang Tong and colleagues identified CLIP-blind pairs, images that a CLIP encoder maps to nearly identical embeddings despite obvious visual differences, and built the MMVP benchmark from them. Models including GPT-4V failed straightforward questions across nine basic visual patterns and produced hallucinated explanations for their errors, and the visual patterns that trouble CLIP models correlated with those that trouble the multimodal systems built on them ^[32]. Penghao Wu and Saining Xie made a related point with V*, arguing that multimodal models lack an active visual search mechanism for locating small details in high-resolution images, and released V*Bench to measure it ^[34].

Evaluation is itself contested. Beyond the leakage and image-independence findings behind MMStar ^[33], exact-match accuracy rewards terse answers and penalizes correct paraphrases, which is why generative systems need format prompts to score fairly ^[29]. Knowledge-intensive VQA adds another layer: a July 2026 diagnostic study by Hanseok Oh and colleagues annotated a benchmark stage by stage to separate visual grounding, object identification, and knowledge retrieval, and found retrieval and reasoning to be the primary bottleneck, though models also struggled to identify the target object and image retrievers struggled to use textual cues. Adding a grounding module that crops the target before image retrieval raised accuracy by 13.3 points for GPT-5 and 8.5 points for Qwen ^[37].

State of the art in 2025 and 2026

Frontier model announcements now lead with MMMU-Pro rather than VQA v2. Google announced Gemini 3 on 18 November 2025 with 81% on MMMU-Pro and 87.6% on Video-MMMU ^[35]. The comparison table on Google DeepMind's Gemini Pro model page puts Gemini 3.1 Pro Thinking at 80.5% on MMMU-Pro, GPT-5.2 Thinking at 79.5%, Claude Sonnet 4.6 Thinking at 74.5%, and Claude Opus 4.6 Thinking at 73.9% ^[36]. Those figures come from a vendor page and should be read as such; on that table the leading proprietary models sit within about seven points of each other on college-level multimodal reasoning.

Active research has moved past the plain image-question format toward knowledge-intensive and retrieval-augmented question answering, and toward stage-wise diagnostics that localize where a pipeline fails rather than reporting end-task accuracy alone ^[37][33]. The task itself has become an interface rather than a research target. Most claims about what a multimodal model can perceive are now settled by asking it questions about pictures.

See also

• Vision language model
• Multimodal model
• Vision encoder
• Computer vision
• MMMU
• LLaVA

References

1. ^Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., Parikh, D. "VQA: Visual Question Answering." Proceedings of the IEEE International Conference on Computer Vision (ICCV) 2015, pp. 2425-2433. arXiv preprint: arxiv.org/...1505.00468
2. ^VQA dataset download page, Virginia Tech / Georgia Tech. visualqa.org/download
3. ^VQA evaluation page (accuracy metric and answer normalization). visualqa.org/evaluation
4. ^Malinowski, M., Fritz, M. "A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input." NIPS 2014. arxiv.org/...1410.0210
5. ^Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D. "Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering." CVPR 2017. arxiv.org/...1612.00837
6. ^Agrawal, A., Batra, D., Parikh, D., Kembhavi, A. "Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering." CVPR 2018. arxiv.org/...1712.00377
7. ^Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., Girshick, R. "CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning." 2016. arxiv.org/...1612.06890
8. ^CLEVR dataset page, Stanford University. cs.stanford.edu/...clevr
9. ^Hudson, D. A., Manning, C. D. "GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering." CVPR 2019. arxiv.org/...1902.09506
10. ^Marino, K., Rastegari, M., Farhadi, A., Mottaghi, R. "OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge." CVPR 2019. arxiv.org/...1906.00067
11. ^OK-VQA dataset site (question count, answers per question, and the ten knowledge categories). okvqa.allenai.org
12. ^Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., Rohrbach, M. "Towards VQA Models That Can Read." CVPR 2019. arxiv.org/...1904.08920
13. ^Mathew, M., Karatzas, D., Jawahar, C. V. "DocVQA: A Dataset for VQA on Document Images." 2020. arxiv.org/...2007.00398
14. ^DocVQA dataset page. docvqa.org/...docvqa
15. ^Masry, A., Long, D. X., Tan, J. Q., Joty, S., Hoque, E. "ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning." 2022. arxiv.org/...2203.10244
16. ^Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., Bigham, J. P. "VizWiz Grand Challenge: Answering Visual Questions from Blind People." CVPR 2018. arxiv.org/...1802.08218
17. ^Yue, X., Ni, Y., Zhang, K., Zheng, T., et al. "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI." 2023. arxiv.org/...2311.16502
18. ^MMMU benchmark site and leaderboard. mmmu-benchmark.github.io
19. ^Yue, X., Zheng, T., Ni, Y., Wang, Y., et al. "MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark." 2024. arxiv.org/...2409.02813
20. ^Yang, Z., He, X., Gao, J., Deng, L., Smola, A. "Stacked Attention Networks for Image Question Answering." 2015. arxiv.org/...1511.02274
21. ^Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L. "Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering." 2017. arxiv.org/...1707.07998
22. ^Lu, J., Batra, D., Parikh, D., Lee, S. "ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks." 2019. arxiv.org/...1908.02265
23. ^Tan, H., Bansal, M. "LXMERT: Learning Cross-Modality Encoder Representations from Transformers." EMNLP 2019. arxiv.org/...1908.07490
24. ^Wang, W., Bao, H., Dong, L., Bjorck, J., et al. "Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks." 2022. arxiv.org/...2208.10442
25. ^Alayrac, J.-B., et al. "Flamingo: a Visual Language Model for Few-Shot Learning." 2022. arxiv.org/...2204.14198
26. ^Li, J., Li, D., Xiong, C., Hoi, S. "BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation." 2022. arxiv.org/...2201.12086
27. ^Li, J., Li, D., Savarese, S., Hoi, S. "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models." 2023. arxiv.org/...2301.12597
28. ^Liu, H., Li, C., Wu, Q., Lee, Y. J. "Visual Instruction Tuning." 2023. arxiv.org/...2304.08485
29. ^Liu, H., Li, C., Li, Y., Lee, Y. J. "Improved Baselines with Visual Instruction Tuning." 2023. arxiv.org/...2310.03744
30. ^OpenAI. "GPT-4V(ision) System Card." 25 September 2023. cdn.openai.com/...GPTV_System_Card.pdf
31. ^Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., Wen, J.-R. "Evaluating Object Hallucination in Large Vision-Language Models." EMNLP 2023. arxiv.org/...2305.10355
32. ^Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., Xie, S. "Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs." 2024. arxiv.org/...2401.06209
33. ^Chen, L., Li, J., Dong, X., Zhang, P., et al. "Are We on the Right Way for Evaluating Large Vision-Language Models?" 2024. arxiv.org/...2403.20330
34. ^Wu, P., Xie, S. "V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs." 2023. arxiv.org/...2312.14135
35. ^Google. "Gemini 3." Official blog announcement, 18 November 2025. blog.google/...gemini-3
36. ^Google DeepMind. Gemini Pro model page (benchmark comparison table). deepmind.google/...pro
37. ^Oh, H., BehnamGhader, P., Krojer, B., Lee, H., Liang, P., Reddy, S., Dankers, V. "CRAG-MM-Diagnostics: Enabling Stage-Wise Analysis of Knowledge-Intensive VQA." 23 July 2026. arxiv.org/...2607.21155