Crusader Bench

Crusader Bench is an AI benchmark that tests AI agents by having them develop a settlement's economy in Stronghold Crusader: Definitive Edition. Models receive screenshots and structured readings of the game, then act through mouse-and-keyboard tools. The project publishes its results, recordings and software, called Crusader Arena. It is maintained by @hesamsheikh and describes itself as independent of Firefly Studios and AI laboratories.^[1][2]

Task and evaluation setup

The scored task uses Free Build mode without enemies. The agent must increase the settlement's wealth while keeping people housed and fed. Soldiers and defences are forbidden by the instructions, although raw mouse input is not a complete technical barrier to building them. The benchmark therefore measures economic management and computer use, not combat performance.^[4][6]

Each map has two versions of the task. One counts stored weapons and armour in net worth; the other gives those goods no stored value, although selling them still produces gold that counts. The leaderboard dated October 19, 2026 compared ten models with six games each: one game for each task on each of the three maps. These are different task conditions, not six identical statistical repetitions.^[4][5]

The harness pauses the game while a model reasons or only reads information. Actions consume game time. Models cannot change the game speed or pause it themselves. Runs record model settings and fingerprints of the relevant prompts, tools and code; the October 19 leaderboard includes version 1.7 games, using patches 1.7.1 and 1.7.3, rather than pooling every earlier experiment.^[5][8]

Agent interface

The agent is an image-capable language model with tool calling. Its 27 tools include building placement, trade, camera movement, inspections, raw clicks and a notebook. Higher-level tools perform sequences of interface actions and check their results; for example, a building tool can retry a blocked placement. Each reply permits at most eight actions. This is an assisted interface, rather than a test of raw screenshots and clicks alone.^[6]

A reader supplies quantities such as gold, stored goods, population and popularity from the running game. It reads each sample twice and rejects inconsistent readings, but does not guarantee an atomic snapshot. The model also receives screenshots when needed. Its plan and notebook survive conversation compaction. Separate learning-series experiments can carry a playbook between episodes; the ranked version 1.7 games do not share memory between their two tasks on a map.^[6][7]

Scoring

The website combines three final measurements into a score:^[3]

score = 100 * (0.8 * W + 0.1 * P + 0.1 * L)

Net worth is gold plus stored goods valued at the game's fixed marketplace selling prices. Buildings contribute through their output rather than a resale valuation. Popularity uses the game's own scale, not the human-normalization formula used for wealth and population. Scores can fall below zero when the settlement ends poorer than doing nothing. The website averages the two task scores for each map, then averages the three maps equally.^[3]

Records and limitations

Published runs include messages, tool calls, screenshots, final readings, per-minute economic data and videos. Post-game interviews are separate from scoring. An interview is the model's account of its play, not an independently verified explanation of its decisions. Written run analyses may also be model-generated and do not determine the numerical score.^[8][10]

The project cautions that neither game nor model randomness is fixed, and that small score differences may be noise. Each map has only one human reference game. Tests have used one game-machine setup and one game build, so the results do not establish performance across other environments.^[5][9]

The status documentation also records a trade-tool fault: sales from sufficiently large inventories can sell more goods than requested because the tool miscounts the game's sale increment. Final net worth is read from the game, but the fault can affect an agent's actions and reported trade quantities. Recorded runs used a private image guide; a fresh checkout instead supplies a text guide unless equivalent images are prepared.^[9]

The repository's code is MIT-licensed. Logs, tables and model outputs in the dataset use CC BY 4.0, but game screenshots and footage are excluded from that licence. Reproduction requires a separate copy of the game.^[2][10]

References

1. ^Crusader Bench. "About Crusader Bench." Accessed October 23, 2026. crusbench.com/about
2. ^1 ^2hesamsheikh. "CrusaderArena." GitHub repository and README, accessed October 23, 2026. github.com/...CrusaderArena
3. ^1 ^2Crusader Bench. "How the Benchmark Works." Methodology, version 1.7.3, updated October 19, 2026. crusbench.com/methodology
4. ^1 ^2 ^3 ^4 ^5Crusader Bench. "The Benchmark." Documentation at commit e7ba4db, October 19, 2026. crusbench.com/...benchmark
5. ^1 ^2 ^3Crusader Bench. "Independent Analysis of AI Agents Playing Stronghold Crusader." Leaderboard and game counts updated October 19, 2026; accessed October 23, 2026. crusbench.com
6. ^1 ^2 ^3Crusader Bench. "The Agent." Documentation at commit e7ba4db, October 19, 2026. crusbench.com/...agent
7. ^Crusader Bench. "The Game-State Reader." Documentation, accessed October 23, 2026. crusbench.com/...reader
8. ^1 ^2Crusader Bench. "The Harness and Dashboard." Documentation at commit e7ba4db, October 19, 2026. crusbench.com/...harness
9. ^1 ^2Crusader Bench. "Status." Documentation at commit e7ba4db, October 19, 2026. crusbench.com/...status
10. ^1 ^2hesamation. "crusader-arena-runs." Hugging Face dataset card and published run tables, accessed October 23, 2026. huggingface.co/...crusader-arena-runs