# LingoQA

> Source: https://aiwiki.ai/wiki/lingoqa
> Updated: 2026-08-04
> Fact-checked: 2026-08-04
> Categories: AI Benchmarks, Autonomous Vehicles
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "LingoQA." aiwiki.ai, 4 Aug 2026. https://aiwiki.ai/wiki/lingoqa
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**LingoQA** is a video question answering benchmark and dataset for [autonomous driving](https://aiwiki.ai/wiki/autonomous_driving), developed by researchers at the London-based self-driving company [Wayve](https://aiwiki.ai/wiki/wayve)[8] and published as the paper "LingoQA: Visual Question Answering for Autonomous Driving" (arXiv 2312.14115, first version December 21, 2023), which was accepted to ECCV 2024.[1] The benchmark pairs short driving video clips with free-form questions and answers covering driving behavior and scene perception, and introduces Lingo-Judge, a learned text classifier that scores model answers in close agreement with human raters.[1] By 2026 LingoQA had become a standard external evaluation for driving-focused [vision-language-action models](https://aiwiki.ai/wiki/vision_language_action_model): on August 4, 2026, [NVIDIA](https://aiwiki.ai/wiki/nvidia) reported that its open [Alpamayo 2 Super](https://aiwiki.ai/wiki/nvidia_alpamayo_2_super) model ranked first on LingoQA among nearly 40 evaluated models.[4]

## Key facts

| Item | Detail |
|---|---|
| Developer | [Wayve](https://aiwiki.ai/wiki/wayve) (London, UK) |
| Paper | "LingoQA: Visual Question Answering for Autonomous Driving," arXiv 2312.14115, v1 December 21, 2023, v4 September 26, 2024 [1] |
| Venue | ECCV 2024 [1] |
| Task | Free-form [visual question answering](https://aiwiki.ai/wiki/visual_question_answering) over short driving videos [1] |
| Scale | About 28,000 unique video scenarios and 419,000 annotations [1] |
| Training splits | Action: 24,500 videos, about 267,800 QA pairs; Scenery: 3,500 videos, about 152,500 QA pairs [1] |
| Evaluation split | 100 videos, 500 questions with two reference answers each, 1,000 QA pairs [1] |
| Metric | Lingo-Judge, a fine-tuned DeBERTa-V3 truthfulness classifier; 0.950 Spearman correlation with human ratings [1] |
| Human baseline | 96.6 percent (five-frame video input) [1] |
| Code and data | github.com/wayveai/LingoQA; Lingo-Judge weights on Hugging Face [2][3] |

## Dataset

LingoQA was built from Wayve's recorded driving footage, captured with a single front-facing camera and cut into 4-second clips.[1] The dataset contains roughly 28,000 unique short video scenarios with about 419,000 question-answer annotations, organized into two training subsets and an evaluation suite:[1][2]

- **Action** (24,500 videos, about 267,800 QA pairs, roughly 11 per video): scenarios centered on notable driving events such as accelerations, lane changes, and turns. Human operators labeled the events with high-level descriptions, and question-answer pairs were generated from those descriptions with GPT-3.5 using prompt templates.[1][2]
- **Scenery** (3,500 videos, about 152,500 QA pairs, roughly 44 per video): perception-focused scenarios drawn from three 30-minute driving sessions that were manually annotated in the ELAN video annotation tool, with diverse questions generated using GPT-4.[1][2]
- **Evaluation** (100 videos, 500 questions, five per scenario): a held-out test suite in which each question carries two distinct correct reference answers, for 1,000 reference QA pairs in total.[1][2] The dataset table in the project's GitHub README transposes the Action and Scenery QA-pair counts; the figures above follow the paper.[1]

The questions span competencies including driving action, justification, attention, object identification, localization, scene description, counting, anticipation, and counterfactual reasoning, which the authors position as a step beyond benchmarks that test perception alone.[1]

## Lingo-Judge metric

The paper's second contribution is Lingo-Judge, a learned evaluation metric built because classical text-overlap metrics correlate poorly with human judgments of answer quality. On the LingoQA evaluation suite, Lingo-Judge reaches a 0.950 Spearman and 0.993 Pearson correlation with human ratings, compared with 0.835 Spearman for BLEU, 0.853 for CIDEr, 0.876 for METEOR, 0.941 for GPT-4 used as a judge, and 0.932 for GPT-4 with chain-of-thought prompting.[1]

Architecturally, Lingo-Judge is a DeBERTa-V3 language model fine-tuned with LoRA and a linear classification head. It takes the question, a reference answer, and the model's predicted answer as text input and outputs the probability that the prediction is truthful; a prediction is scored against both reference answers and the maximum probability is kept.[1] Unlike GPT-4-based judging, the small classifier is cheap enough to run during training: the authors report evaluating the full suite in 10.5 seconds versus 3,016 seconds for GPT-4 with chain-of-thought.[1] The classifier weights are published on Hugging Face as wayveai/Lingo-Judge and are invoked through a standard text-classification pipeline in the benchmark's evaluation script.[2][3]

The authors note that Lingo-Judge is tailored to LingoQA rather than a general-purpose metric: it is optimized for the response style of the dataset's annotators, can struggle with long-form incorrect answers, and measures truthfulness rather than which of two correct answers a human would prefer.[1]

## Baseline results

Evaluating models available in late 2023, the paper found vision-language models well below human performance on LingoQA. GPT-4V answered 59.6 percent of questions truthfully by Lingo-Judge, against 96.6 percent for humans given five-frame video input; humans restricted to a single frame scored 81.8 percent, which the authors cite as evidence that temporal context matters.[1] The paper's own baseline, a Vicuna-1.5-7B language model with a CLIP vision encoder and Q-Former bridge processing five frames per 4-second clip, reached 60.8 percent after fine-tuning on the Action and Scenery subsets.[1]

## Origins and the LINGO model family

LingoQA shares its "Lingo" branding with Wayve's LINGO driving models. Wayve describes LINGO-1 as an open-loop driving commentator that leveraged vision-language inputs to perform visual question answering, while [LINGO-2](https://aiwiki.ai/wiki/lingo_2_wayve), announced April 17, 2024, integrated language directly with driving control as a closed-loop vision-language-action driving model tested on public roads.[8] The LingoQA author list includes Ana-Maria Marcu, Long Chen, Jan Hunermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, Elahe Arani, and Oleg Sinavski.[1]

## Adoption

Since its release, LingoQA has been used both as raw material for derived benchmarks and as an external evaluation for driving-focused models:

- **AutoDrive-QA** (Khalili and Smyth, arXiv 2503.15778, March 2025) converts the open-ended QA pairs of LingoQA, DriveLM, and NuScenes-QA into structured multiple-choice questions with error-based distractors for standardized evaluation.[7]
- **NVIDIA Cosmos Reason 2**: NVIDIA's Cosmos Cookbook uses LingoQA as "a standardized external benchmark for AV visual question answering" when post-training Cosmos Reason 2 for autonomous vehicle captioning and VQA, reporting that post-training lifted the 8B model's Lingo-Judge score from 0.632 zero-shot to 0.770.[6]
- **NVIDIA Alpamayo 2 Super**: announcing the model's open commercial release on August 4, 2026, NVIDIA stated that Alpamayo 2 Super ranked first on LingoQA among nearly 40 models evaluated, outperforming Qwen2.5-VL 72B (see [Qwen](https://aiwiki.ai/wiki/qwen)) by 17.0 points, [Gemini 2.5 Pro](https://aiwiki.ai/wiki/gemini_2_5) by 15.1 points, and [GPT-4o](https://aiwiki.ai/wiki/gpt_4o) by 23.2 points in NVIDIA's testing using the Lingo-Judge metric.[4] The model card reports a Lingo-Judge score of 79.2 for the 34-billion-parameter model.[5]

## Limitations

The authors describe several limitations of the [benchmark](https://aiwiki.ai/wiki/benchmark) in the paper. The videos are short segments sampled at few frames, limiting longer-horizon context, and come from a single front-facing camera with no lidar or other sensor modalities.[1] The benchmark tests question answering rather than driving decisions, so a high LingoQA score does not by itself demonstrate driving competence.[1] The Lingo-Judge metric's own limitations, including its specialization to LingoQA's answer style, are noted above.[1]

## See also

- [Wayve](https://aiwiki.ai/wiki/wayve)
- [LINGO-2](https://aiwiki.ai/wiki/lingo_2_wayve)
- [NVIDIA Alpamayo 2 Super](https://aiwiki.ai/wiki/nvidia_alpamayo_2_super)
- [Vision-language-action model](https://aiwiki.ai/wiki/vision_language_action_model)
- [Visual question answering](https://aiwiki.ai/wiki/visual_question_answering)
- [Autonomous driving](https://aiwiki.ai/wiki/autonomous_driving)

## References

1. Marcu, Ana-Maria; Chen, Long; Hunermann, Jan; Karnsund, Alice; Hanotte, Benoit; Chidananda, Prajwal; Nair, Saurabh; Badrinarayanan, Vijay; Kendall, Alex; Shotton, Jamie; Arani, Elahe; Sinavski, Oleg. "LingoQA: Visual Question Answering for Autonomous Driving." arXiv 2312.14115, v1 December 21, 2023, v4 September 26, 2024. https://arxiv.org/abs/2312.14115
2. Wayve. "wayveai/LingoQA" (official ECCV 2024 benchmark repository). GitHub. https://github.com/wayveai/LingoQA
3. Wayve. "wayveai/Lingo-Judge" (model card). Hugging Face. https://huggingface.co/wayveai/Lingo-Judge
4. NVIDIA. "NVIDIA Alpamayo 2 Super, the Frontier Open Model for Robotaxis and Autonomous Vehicles, Now Available for Commercial Use." August 4, 2026. https://blogs.nvidia.com/blog/alpamayo-2-super-open-model-now-available/
5. NVIDIA. "nvidia/Alpamayo2-Super" (model card). Hugging Face. August 4, 2026. https://huggingface.co/nvidia/Alpamayo2-Super
6. NVIDIA. "Post-train Cosmos Reason 2 for Autonomous Vehicle Video Captioning and VQA." Cosmos Cookbook. https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/post_training/reason2/video_caption_vqa/post_training.html
7. Khalili, Boshra; Smyth, Andrew W. "AutoDrive-QA: A Multiple-Choice Benchmark for Vision-Language Evaluation in Urban Autonomous Driving." arXiv 2503.15778, March 20, 2025. https://arxiv.org/abs/2503.15778
8. Wayve. "LINGO-2: Driving with Natural Language." April 17, 2024. https://wayve.ai/thinking/lingo-2-driving-with-language/