LingoQA

LingoQA is a video question answering benchmark and dataset for autonomous driving, developed by researchers at the London-based self-driving company Wayve^[8] and published as the paper "LingoQA: Visual Question Answering for Autonomous Driving" (arXiv 2312.14115, first version December 21, 2023), which was accepted to ECCV 2024.^[1] The benchmark pairs short driving video clips with free-form questions and answers covering driving behavior and scene perception, and introduces Lingo-Judge, a learned text classifier that scores model answers in close agreement with human raters.^[1] By 2026 LingoQA had become a standard external evaluation for driving-focused vision-language-action models: on August 4, 2026, NVIDIA reported that its open Alpamayo 2 Super model ranked first on LingoQA among nearly 40 evaluated models.^[4]

Key facts

Scroll sideways for more →

Dataset

LingoQA was built from Wayve's recorded driving footage, captured with a single front-facing camera and cut into 4-second clips.^[1] The dataset contains roughly 28,000 unique short video scenarios with about 419,000 question-answer annotations, organized into two training subsets and an evaluation suite:^[1][2]

• Action (24,500 videos, about 267,800 QA pairs, roughly 11 per video): scenarios centered on notable driving events such as accelerations, lane changes, and turns. Human operators labeled the events with high-level descriptions, and question-answer pairs were generated from those descriptions with GPT-3.5 using prompt templates.^[1][2]
• Scenery (3,500 videos, about 152,500 QA pairs, roughly 44 per video): perception-focused scenarios drawn from three 30-minute driving sessions that were manually annotated in the ELAN video annotation tool, with diverse questions generated using GPT-4.^[1][2]
• Evaluation (100 videos, 500 questions, five per scenario): a held-out test suite in which each question carries two distinct correct reference answers, for 1,000 reference QA pairs in total.^[1][2] The dataset table in the project's GitHub README transposes the Action and Scenery QA-pair counts; the figures above follow the paper.^[1]

The questions span competencies including driving action, justification, attention, object identification, localization, scene description, counting, anticipation, and counterfactual reasoning, which the authors position as a step beyond benchmarks that test perception alone.^[1]

Lingo-Judge metric

The paper's second contribution is Lingo-Judge, a learned evaluation metric built because classical text-overlap metrics correlate poorly with human judgments of answer quality. On the LingoQA evaluation suite, Lingo-Judge reaches a 0.950 Spearman and 0.993 Pearson correlation with human ratings, compared with 0.835 Spearman for BLEU, 0.853 for CIDEr, 0.876 for METEOR, 0.941 for GPT-4 used as a judge, and 0.932 for GPT-4 with chain-of-thought prompting.^[1]

Architecturally, Lingo-Judge is a DeBERTa-V3 language model fine-tuned with LoRA and a linear classification head. It takes the question, a reference answer, and the model's predicted answer as text input and outputs the probability that the prediction is truthful; a prediction is scored against both reference answers and the maximum probability is kept.^[1] Unlike GPT-4-based judging, the small classifier is cheap enough to run during training: the authors report evaluating the full suite in 10.5 seconds versus 3,016 seconds for GPT-4 with chain-of-thought.^[1] The classifier weights are published on Hugging Face as wayveai/Lingo-Judge and are invoked through a standard text-classification pipeline in the benchmark's evaluation script.^[2][3]

The authors note that Lingo-Judge is tailored to LingoQA rather than a general-purpose metric: it is optimized for the response style of the dataset's annotators, can struggle with long-form incorrect answers, and measures truthfulness rather than which of two correct answers a human would prefer.^[1]

Baseline results

Evaluating models available in late 2023, the paper found vision-language models well below human performance on LingoQA. GPT-4V answered 59.6 percent of questions truthfully by Lingo-Judge, against 96.6 percent for humans given five-frame video input; humans restricted to a single frame scored 81.8 percent, which the authors cite as evidence that temporal context matters.^[1] The paper's own baseline, a Vicuna-1.5-7B language model with a CLIP vision encoder and Q-Former bridge processing five frames per 4-second clip, reached 60.8 percent after fine-tuning on the Action and Scenery subsets.^[1]

Origins and the LINGO model family

LingoQA shares its "Lingo" branding with Wayve's LINGO driving models. Wayve describes LINGO-1 as an open-loop driving commentator that leveraged vision-language inputs to perform visual question answering, while LINGO-2, announced April 17, 2024, integrated language directly with driving control as a closed-loop vision-language-action driving model tested on public roads.^[8] The LingoQA author list includes Ana-Maria Marcu, Long Chen, Jan Hunermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, Elahe Arani, and Oleg Sinavski.^[1]

Adoption

Since its release, LingoQA has been used both as raw material for derived benchmarks and as an external evaluation for driving-focused models:

• AutoDrive-QA (Khalili and Smyth, arXiv 2503.15778, March 2025) converts the open-ended QA pairs of LingoQA, DriveLM, and NuScenes-QA into structured multiple-choice questions with error-based distractors for standardized evaluation.^[7]
• NVIDIA Cosmos Reason 2: NVIDIA's Cosmos Cookbook uses LingoQA as "a standardized external benchmark for AV visual question answering" when post-training Cosmos Reason 2 for autonomous vehicle captioning and VQA, reporting that post-training lifted the 8B model's Lingo-Judge score from 0.632 zero-shot to 0.770.^[6]
• NVIDIA Alpamayo 2 Super: announcing the model's open commercial release on August 4, 2026, NVIDIA stated that Alpamayo 2 Super ranked first on LingoQA among nearly 40 models evaluated, outperforming Qwen2.5-VL 72B (see Qwen) by 17.0 points, Gemini 2.5 Pro by 15.1 points, and GPT-4o by 23.2 points in NVIDIA's testing using the Lingo-Judge metric.^[4] The model card reports a Lingo-Judge score of 79.2 for the 34-billion-parameter model.^[5]

Limitations

The authors describe several limitations of the benchmark in the paper. The videos are short segments sampled at few frames, limiting longer-horizon context, and come from a single front-facing camera with no lidar or other sensor modalities.^[1] The benchmark tests question answering rather than driving decisions, so a high LingoQA score does not by itself demonstrate driving competence.^[1] The Lingo-Judge metric's own limitations, including its specialization to LingoQA's answer style, are noted above.^[1]

See also

• Wayve
• LINGO-2
• NVIDIA Alpamayo 2 Super
• Vision-language-action model
• Visual question answering
• Autonomous driving

References

1. ^Marcu, Ana-Maria; Chen, Long; Hunermann, Jan; Karnsund, Alice; Hanotte, Benoit; Chidananda, Prajwal; Nair, Saurabh; Badrinarayanan, Vijay; Kendall, Alex; Shotton, Jamie; Arani, Elahe; Sinavski, Oleg. "LingoQA: Visual Question Answering for Autonomous Driving." arXiv 2312.14115, v1 December 21, 2023, v4 September 26, 2024. arxiv.org/...2312.14115
2. ^Wayve. "wayveai/LingoQA" (official ECCV 2024 benchmark repository). GitHub. github.com/...LingoQA
3. ^Wayve. "wayveai/Lingo-Judge" (model card). Hugging Face. huggingface.co/...Lingo-Judge
4. ^NVIDIA. "NVIDIA Alpamayo 2 Super, the Frontier Open Model for Robotaxis and Autonomous Vehicles, Now Available for Commercial Use." August 4, 2026. blogs.nvidia.com/...super-open-model-now-available
5. ^NVIDIA. "nvidia/Alpamayo2-Super" (model card). Hugging Face. August 4, 2026. huggingface.co/...Alpamayo2-Super
6. ^NVIDIA. "Post-train Cosmos Reason 2 for Autonomous Vehicle Video Captioning and VQA." Cosmos Cookbook. nvidia-cosmos.github.io/...post_training
7. ^Khalili, Boshra; Smyth, Andrew W. "AutoDrive-QA: A Multiple-Choice Benchmark for Vision-Language Evaluation in Urban Autonomous Driving." arXiv 2503.15778, March 20, 2025. arxiv.org/...2503.15778
8. ^Wayve. "LINGO-2: Driving with Natural Language." April 17, 2024. wayve.ai/...lingo-2-driving-with-language