LingoQA

LingoQA is a video question answering benchmark and dataset for autonomous driving, developed by researchers at the London-based self-driving company Wayve^[8] and published as the paper "LingoQA: Visual Question Answering for Autonomous Driving" (arXiv 2312.14115, first version December 21, 2023), which was accepted to ECCV 2024.^[1] The benchmark pairs short driving video clips with free-form questions and answers covering driving behavior and scene perception, and introduces Lingo-Judge, a learned text classifier that scores model answers in close agreement with human raters.^[1] By 2026 LingoQA had become a standard external evaluation for driving-focused vision-language-action models: on August 4, 2026, NVIDIA reported that its open Alpamayo 2 Super model ranked first on LingoQA among nearly 40 evaluated models.^[4]

Key facts

Dataset

LingoQA was built from Wayve's recorded driving footage, captured with a single front-facing camera and cut into 4-second clips.^[1] The dataset contains roughly 28,000 unique short video scenarios with about 419,000 question-answer annotations, organized into two training subsets and an evaluation suite:^[1][2]

• Action (24,500 videos, about 267,800 QA pairs, roughly 11 per video): scenarios centered on notable driving events such as accelerations, lane changes, and turns. Human operators labeled the events with high-level descriptions, and question-answer pairs were generated from those descriptions with GPT-3.5 using prompt templates.^[1][2]
• Scenery (3,500 videos, about 152,500 QA pairs, roughly 44 per video): perception-focused scenarios drawn from three 30-minute driving sessions that were manually annotated in the ELAN video annotation tool, with diverse questions generated using GPT-4.^[1][2]
• Evaluation (100 videos, 500 questions, five per scenario): a held-out test suite in which each question carries two distinct correct reference answers, for 1,000 reference QA pairs in total.^[1][2] The dataset table in the project's GitHub README transposes the Action and Scenery QA-pair counts; the figures above follow the paper.^[1]

The questions span competencies including driving action, justification, attention, object identification, localization, scene description, counting, anticipation, and counterfactual reasoning, which the authors position as a step beyond benchmarks that test perception alone.^[1]

Lingo-Judge metric

The paper's second contribution is Lingo-Judge, a learned evaluation metric built because classical text-overlap metrics correlate poorly with human judgments of answer quality. On the LingoQA evaluation suite, Lingo-Judge reaches a 0.950 Spearman and 0.993 Pearson correlation with human ratings, compared with 0.835 Spearman for BLEU, 0.853 for CIDEr, 0.876 for METEOR, 0.941 for GPT-4 used as a judge, and 0.932 for GPT-4 with chain-of-thought prompting.^[1]

Architecturally, Lingo-Judge is a DeBERTa-V3 language model fine-tuned with LoRA and a linear classification head. It takes the question, a reference answer, and the model's predicted answer as text input and outputs the probability that the prediction is truthful; a prediction is scored against both reference answers and the maximum probability is kept.^[1] Unlike GPT-4-based judging, the small classifier is cheap enough to run during training: the authors report evaluating the full suite in 10.5 seconds versus 3,016 seconds for GPT-4 with chain-of-thought.^[1] The classifier weights are published on Hugging Face as wayveai/Lingo-Judge and are invoked through a standard text-classification pipeline in the benchmark's evaluation script.^[2][3]

The authors note that Lingo-Judge is tailored to LingoQA rather than a general-purpose metric: it is optimized for the response style of the dataset's annotators, can struggle with long-form incorrect answers, and measures truthfulness rather than which of two correct answers a human would prefer.^[1]

Baseline results

Evaluating models available in late 2023, the paper found vision-language models well below human performance on LingoQA. GPT-4V answered 59.6 percent of questions truthfully by Lingo-Judge, against 96.6 percent for humans given five-frame video input; humans restricted to a single frame scored 81.8 percent, which the authors cite as evidence that temporal context matters.^[1] The paper's own baseline, a Vicuna-1.5-7B language model with a CLIP vision encoder and Q-Former bridge processing five frames per 4-second clip, reached 60.8 percent after fine-tuning on the Action and Scenery subsets.^[1]

Origins and the LINGO model family

LingoQA shares its "Lingo" branding with Wayve's LINGO driving models. Wayve describes LINGO-1 as an open-loop driving commentator that leveraged vision-language inputs to perform visual question answering, while LINGO-2, announced April 17, 2024, integrated language directly with driving control as a closed-loop vision-language-action driving model tested on public roads.^[8] The LingoQA author list includes Ana-Maria Marcu, Long Chen, Jan Hunermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, Elahe Arani, and Oleg Sinavski.^[1]

Adoption

Since its release, LingoQA has been used both as raw material for derived benchmarks and as an external evaluation for driving-focused models:

• AutoDrive-QA (Khalili and Smyth, arXiv 2503.15778, March 2025) converts the open-ended QA pairs of LingoQA, DriveLM, and NuScenes-QA into structured multiple-choice questions with error-based distractors for standardized evaluation.^[7]
• NVIDIA Cosmos Reason 2: NVIDIA's Cosmos Cookbook uses LingoQA as "a standardized external benchmark for AV visual question answering" when post-training Cosmos Reason 2 for autonomous vehicle captioning and VQA, reporting that post-training lifted the 8B model's Lingo-Judge score from 0.632 zero-shot to 0.770.^[6]
• NVIDIA Alpamayo 2 Super: announcing the model's open commercial release on August 4, 2026, NVIDIA stated that Alpamayo 2 Super ranked first on LingoQA among nearly 40 models evaluated, outperforming Qwen2.5-VL 72B (see Qwen) by 17.0 points, Gemini 2.5 Pro by 15.1 points, and GPT-4o by 23.2 points in NVIDIA's testing using the Lingo-Judge metric.^[4] The model card reports a Lingo-Judge score of 79.2 for the 34-billion-parameter model.^[5]

Limitations

The authors describe several limitations of the benchmark in the paper. The videos are short segments sampled at few frames, limiting longer-horizon context, and come from a single front-facing camera with no lidar or other sensor modalities.^[1] The benchmark tests question answering rather than driving decisions, so a high LingoQA score does not by itself demonstrate driving competence.^[1] The Lingo-Judge metric's own limitations, including its specialization to LingoQA's answer style, are noted above.^[1]

Continue reading

• WayveWayve is a British artificial intelligence company, founded in Cambridge in 2017 by Alex Kendall and Amar Shah, that develops end to end "embodied AI" software for autonomous driving and robotics, and in May…Mentioned 5 times above
• VVisual question answeringVisual question answering (VQA) is the task of producing a natural-language answer to a natural-language question about an image.Mentioned 5 times above
• NVIDIA Alpamayo 2 SuperNVIDIA Alpamayo 2 Super is an open, 34-billion-parameter reasoning-based vision-language-action model (VLA) for safe, Level 4 robotaxi and autonomous-vehicle development from NVIDIA.Mentioned 3 times above
• AAutonomous drivingAutonomous driving is the sustained performance by a vehicle system of part or all of the dynamic driving task.Mentioned 3 times above
• LINGO-2 (Wayve)LINGO-2 is a closed-loop vision-language-action model for autonomous driving developed by the British self-driving company Wayve.Listed in See also
• VVision-language-action modelA vision-language-action model (VLA model, or VLA) is a machine-learning model for robotics that uses visual observations and a natural-language instruction to generate actions a robot can execute.Listed in See also

More to read

• Nvidia
• Benchmark (AI)
• Waymo
• Hyundai
• BYD
• Baidu AI

Topics:AI BenchmarksAutonomous Vehicles

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27Marcu, Ana-Maria; Chen, Long; Hunermann, Jan; Karnsund, Alice; Hanotte, Benoit; Chidananda, Prajwal; Nair, Saurabh; Badrinarayanan, Vijay; Kendall, Alex; Shotton, Jamie; Arani, Elahe; Sinavski, Oleg. "LingoQA: Visual Question Answering for Autonomous Driving." arXiv 2312.14115, v1 December 21, 2023, v4 September 26, 2024. arxiv.org/...2312.14115
2. ^1 ^2 ^3 ^4 ^5 ^6Wayve. "wayveai/LingoQA" (official ECCV 2024 benchmark repository). GitHub. github.com/...LingoQA
3. ^1 ^2Wayve. "wayveai/Lingo-Judge" (model card). Hugging Face. huggingface.co/...Lingo-Judge
4. ^1 ^2NVIDIA. "NVIDIA Alpamayo 2 Super, the Frontier Open Model for Robotaxis and Autonomous Vehicles, Now Available for Commercial Use." August 4, 2026. blogs.nvidia.com/...super-open-model-now-available
5. ^NVIDIA. "nvidia/Alpamayo2-Super" (model card). Hugging Face. August 4, 2026. huggingface.co/...Alpamayo2-Super
6. ^NVIDIA. "Post-train Cosmos Reason 2 for Autonomous Vehicle Video Captioning and VQA." Cosmos Cookbook. nvidia-cosmos.github.io/...post_training
7. ^Khalili, Boshra; Smyth, Andrew W. "AutoDrive-QA: A Multiple-Choice Benchmark for Vision-Language Evaluation in Urban Autonomous Driving." arXiv 2503.15778, March 20, 2025. arxiv.org/...2503.15778
8. ^1 ^2Wayve. "LINGO-2: Driving with Natural Language." April 17, 2024. wayve.ai/...lingo-2-driving-with-language