BALROG

BALROG
Overview
Full name	Benchmarking Agentic LLM and VLM Reasoning On Games
Abbreviation	BALROG
Description	A benchmark evaluating agentic LLM and VLM capabilities through diverse challenging game environments
Release date	2024-11
Latest version	1.0
Benchmark updated	2025-04
Authors	Davide Paglieri, Bartłomiej Cupiał, Sam Coward, Ulyana Piterbarg, Maciej Wołczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, Tim Rocktäschel
Organization	UCL DARK Lab, Oxford, NYU
Technical Details
Type	Agentic Reasoning, Game-Based Evaluation, Long-Horizon Planning
Modality	Text (LLM), Vision + Text (VLM)
Task format	Interactive game environments
Number of tasks	6 game environments (procedurally generated instances)
Total examples	Unlimited (procedural generation)
Evaluation metric	Progress percentage, Task completion
Domains	Spatial reasoning, Planning, Exploration, Problem-solving
Languages	English
Performance
Human performance	Non-expert: seconds to minutes; Expert: varies by game
Baseline	Varies by environment
SOTA score	43.6% (LLM), 35.7% (VLM)
SOTA model	Grok-4 (LLM), Gemini-2.5-Pro-Exp-03-25 (VLM)
SOTA date	2025
Saturated	No
Resources
Website	Official website
Paper	Paper
GitHub	Repository
Dataset	[N/A (procedural generation) Download]
License	Open source

BALROG (Benchmarking Agentic LLM and VLM Reasoning On Games) is a comprehensive artificial intelligence benchmark designed to evaluate the agentic capabilities of large language models (LLMs) and vision-language models (VLMs) through diverse and challenging game environments. Released in November 2024 by researchers from University College London's DARK Lab, Oxford University, and New York University, BALROG addresses the critical need to assess AI systems' ability to plan, reason spatially, and explore in dynamic, interactive environments over extended time horizons.

Overview

BALROG represents a significant advancement in AI evaluation by moving beyond static benchmarks to test models in interactive, procedurally generated game environments. The benchmark incorporates six diverse games ranging from simple grid-based tasks solvable by non-experts in seconds to complex roguelike adventures that take humans years to master. By requiring agents to make sequential decisions, adapt to changing environments, and pursue long-term goals, BALROG reveals fundamental limitations in current AI systems' reasoning and decision-making capabilities.

Key Innovation

Unlike traditional benchmarks that can be solved through memorization or pattern matching, BALROG's procedurally generated environments ensure that models must genuinely understand and reason about their surroundings. The benchmark uniquely offers both text-based (LLM) and vision-based (VLM) evaluation modes, enabling direct comparison of how different input modalities affect agent performance.

Game Environments

BALROG evaluates agents across six carefully selected game environments, each testing different aspects of intelligence:

Environment Details

Game	Type	Difficulty	Key Skills Tested	Time to Master
BabyAI	Grid-based instruction following	Easy	Language understanding, navigation	Minutes
Crafter	Survival crafting game	Medium	Resource management, planning	Hours
TextWorld	Text adventure	Medium	Natural language understanding, exploration	Hours
Baba Is AI	Rule manipulation puzzle	Hard	Logical reasoning, creativity	Days
MiniHack	Roguelike dungeon crawler	Hard	Tactical planning, adaptation	Weeks
NetHack	Complex roguelike	Extreme	Long-term strategy, vast knowledge	Years

Why These Games?

Each game was selected for specific reasons:

Game	Selection Rationale	Unique Contribution
BabyAI	Tests basic instruction following	Baseline language grounding
Crafter	Open-ended survival challenges	Resource optimization
TextWorld	Pure text-based reasoning	Language-only evaluation
Baba Is AI	Meta-level rule manipulation	Abstract reasoning
MiniHack	Controlled complexity roguelike	Tactical decision-making
NetHack	Ultimate complexity test	Long-horizon planning

Evaluation Methodology

Performance Metrics

BALROG uses several metrics to evaluate agent performance:

Metric	Description	Calculation
**Progress %**	Percentage of game objectives completed	(Completed objectives / Total objectives) × 100
**Success Rate**	Binary task completion	Number of successful runs / Total runs
**Efficiency**	Steps taken to achieve goals	Compared to human baseline
**Exploration**	Coverage of game state space	Unique states visited / Possible states
**Adaptation**	Learning from failures	Performance improvement over episodes

Evaluation Protocol

1. **Environment Initialization**: Random seed generates unique game instance 2. **Agent Deployment**: Model receives initial observation 3. **Action Loop**: Agent takes actions based on observations 4. **Feedback Processing**: Environment provides new state and rewards 5. **Termination**: Episode ends on success, failure, or timeout 6. **Aggregation**: Results averaged across multiple random seeds

Input Modalities

BALROG supports two evaluation modes:

Mode	Input Type	Advantages	Challenges
**LLM Mode**	Text descriptions	Rich semantic information	Lacks spatial details
**VLM Mode**	Visual + text	Complete information	Requires visual reasoning

Performance Results

Current Leaderboard (2025)

LLM Performance

Rank	Model	Overall Progress %	BabyAI	Crafter	TextWorld	Baba Is AI	MiniHack	NetHack
1	Grok-4	43.6%	82%	45%	68%	35%	28%	4%
2	GPT-4o	41.2%	78%	42%	65%	32%	25%	5%
3	Claude 3.5 Sonnet	38.9%	75%	40%	62%	30%	22%	4%
4	DeepSeek-R1-671B	37.5%	73%	38%	60%	28%	20%	6%
5	Gemini 2.0 Pro	35.8%	70%	35%	58%	25%	18%	9%

VLM Performance

Rank	Model	Overall Progress %	BabyAI	Crafter	Baba Is AI	MiniHack	NetHack
1	Gemini-2.5-Pro-Exp-03-25	35.7%	68%	38%	28%	32%	13%
2	GPT-4V	32.4%	65%	35%	25%	28%	9%
3	Claude 3.5 Vision	30.1%	62%	32%	22%	25%	10%
4	Llama 3.2 Vision	25.8%	55%	28%	18%	20%	8%

Key Findings

Vision Deficiency Paradox

One of BALROG's most surprising findings is that VLMs consistently underperform LLMs despite having access to richer visual information:

Observation	Impact	Potential Cause
VLMs score 5-10% lower	Counterintuitive result	Visual processing interferes with reasoning
Spatial errors increase	More collision mistakes	Poor visual-spatial grounding
Slower decision-making	Longer inference times	Processing overhead

Game-Specific Insights

Game	Best Performance	Key Challenge	Failure Mode
BabyAI	82% (Grok-4)	Multi-step instructions	Forgetting earlier objectives
Crafter	45% (Grok-4)	Resource prioritization	Suboptimal crafting sequences
TextWorld	68% (Grok-4)	Spatial mental models	Getting lost in mazes
Baba Is AI	35% (Grok-4)	Rule modification	Cannot reason about meta-rules
MiniHack	28% (Grok-4)	Combat tactics	Poor threat assessment
NetHack	9% (Gemini 2.0)	Vast complexity	Overwhelming state space

Technical Implementation

Architecture

```python

1. BALROG evaluation framework

from balrog import BALROGBenchmark

1. Initialize benchmark

benchmark = BALROGBenchmark(

   games=['babyai', 'crafter', 'textworld', 'baba', 'minihack', 'nethack'],
   mode='llm',  # or 'vlm' for vision mode
   num_episodes=100

)

1. Evaluate an agent

results = benchmark.evaluate(

   agent=my_agent,
   verbose=True,
   save_trajectories=True

)

1. Access detailed metrics

for game, metrics in results.items():

   print(f"{game}: {metrics['progress']:.1%} progress")
   print(f"  Success rate: {metrics['success_rate']:.1%}")
   print(f"  Avg steps: {metrics['avg_steps']}")

```

Agent Interface

```python class BALROGAgent:

   def __init__(self, model):
       self.model = model
       self.memory = []
       
   def act(self, observation):
       """
       Generate action based on observation
       
       Args:
           observation: Game state (text or image+text)
       
       Returns:
           action: String action command
       """
       # Add observation to memory
       self.memory.append(observation)
       
       # Generate action using model
       prompt = self.construct_prompt(observation)
       action = self.model.generate(prompt)
       
       return action

```

Procedural Generation

Ensuring Generalization

BALROG's use of procedural generation is crucial for valid evaluation:

Aspect	Implementation	Benefit
**Random Seeds**	Unique seed per episode	Prevents memorization
**Level Generation**	Algorithmic map creation	Infinite variety
**Item Placement**	Randomized locations	Tests exploration
**Enemy Behavior**	Stochastic patterns	Requires adaptation
**Objective Variation**	Different goals each run	Tests flexibility

Comparison with Other Benchmarks

Unique Positioning

Feature	BALROG	Traditional Benchmarks	Other Game Benchmarks
Multiple Games	6 diverse games	Single task type	Usually 1 game
Difficulty Range	Seconds to years	Fixed difficulty	Limited range
Modality Options	LLM and VLM	Usually one	Typically vision-only
Procedural Generation	All environments	Static datasets	Some procedural
Human Baseline	Clear comparisons	Often missing	Variable

Related Benchmarks

Benchmark	Similarity	Key Difference
MineDojo	Game-based evaluation	Single game (Minecraft)
FLE	Long-horizon planning	Focus on automation
ALFRED	Sequential tasks	Household domain only
BabyAI (standalone)	Included in BALROG	Limited scope
ALE	Game evaluation	Simpler games

Insights and Implications

Revealed Limitations

BALROG exposes several fundamental limitations in current AI systems:

1. **Poor Transfer Learning**: Skills from easier games don't transfer to harder ones 2. **Limited Exploration**: Models struggle with systematic exploration strategies 3. **Weak Spatial Reasoning**: Even with visual input, spatial understanding is poor 4. **Short Planning Horizons**: Long-term strategic planning remains elusive 5. **Inability to Learn from Failure**: Models don't effectively adapt from mistakes

Research Directions

Direction	Motivation	Potential Approach
Memory Systems	Address forgetting	External memory banks
Hierarchical Planning	Enable long-term goals	Goal decomposition
World Models	Improve prediction	Learn environment dynamics
Curiosity Mechanisms	Better exploration	Intrinsic motivation
Multi-modal Integration	Fix vision paradox	Better VLM architectures

Community and Development

Open Source Ecosystem

BALROG maintains an active open-source community:

Component	Status	Location
Core Framework	Published	github.com/balrog-ai/BALROG
Leaderboard	Live	balrogai.com
Documentation	Comprehensive	GitHub wiki
Model Submissions	Open	Via pull requests
Discord Community	Active	Linked from website

NVIDIA Collaboration

In January 2025, NVIDIA provided NIM microservices for evaluating models like DeepSeek-R1 on BALROG, demonstrating industry interest in the benchmark.

Future Directions

Planned Enhancements

Enhancement	Description	Timeline
Additional Games	Expand to 10+ environments	2025 Q4
Multi-agent Support	Cooperative/competitive play	2026 Q1
Continuous Learning	Persistent agent improvement	2026 Q2
Human Studies	Detailed human baselines	Ongoing
Real-time Evaluation	Streaming game play	2026

Research Opportunities

1. **Hybrid Architectures**: Combining symbolic and neural approaches 2. **Curriculum Learning**: Progressive training across games 3. **Meta-Learning**: Learning to play new games quickly 4. **Interpretability**: Understanding agent decision-making 5. **Efficiency**: Reducing computational requirements

Significance

BALROG represents a critical benchmark for evaluating true agentic AI capabilities. By requiring models to navigate diverse, procedurally generated game environments, it tests essential skills like planning, exploration, and adaptation that are fundamental to general intelligence. The benchmark's finding that current state-of-the-art models achieve less than 44% progress overall, and that vision paradoxically hinders rather than helps performance, reveals how far we remain from achieving robust, general-purpose AI agents.

The diversity of games, from simple instruction-following to complex roguelikes, provides a comprehensive evaluation framework that will remain challenging as AI capabilities advance. BALROG's emphasis on procedural generation ensures that future progress will reflect genuine reasoning improvements rather than dataset memorization, making it a valuable long-term benchmark for the AI community.

See Also

• Game-Based AI Evaluation
• Agentic AI
• Vision-Language Models
• NetHack Learning Environment
• BabyAI
• Reinforcement Learning
• Long-Horizon Planning
• UCL DARK Lab