# PROWL-1

> Source: https://aiwiki.ai/wiki/prowl_1
> Updated: 2026-09-16
> Fact-checked: 2026-09-16
> Categories: Machine Learning, Reinforcement Learning, Training & Optimization, World Models
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "PROWL-1." aiwiki.ai, 16 Sept 2026. https://aiwiki.ai/wiki/prowl_1
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**PROWL-1** is a training framework from the AI lab [Odyssey](https://aiwiki.ai/wiki/odyssey_ai) in which a [reinforcement learning](https://aiwiki.ai/wiki/reinforcement_learning) agent is paid to break a [world model](https://aiwiki.ai/wiki/world_model). The agent acts inside a real game environment, the world model tries to predict what those actions will produce, and the gap between the prediction and what actually happened becomes the agent's reward. Trajectories where the model was most wrong are collected into a buffer and fed back as fine-tuning data, so the failures the agent finds become the model's next curriculum. Odyssey announced PROWL-1 in a post by CTO Jeff Hawke on 12 May 2026 [1]. The underlying research paper, titled simply "PROWL: Prioritized Regret-Driven Optimization for World Model Learning", was posted to arXiv the day before as arXiv:2605.18803, with a revised version following on 29 May 2026 [2].

"Regret" in the name is a concrete quantity rather than a decision-theoretic abstraction: it is the root-mean-square error between the world model's predicted latents and the latents of the real rollout, and, in the paper's training loop (Algorithm 1), it serves as the adversary's terminal reward; the paper's method section instead describes the full trajectory score as the terminal reward, and the two passages disagree [2]. Odyssey files PROWL-1 on its research page under "Reinforcement Learning" rather than among its world-model releases, since it is a method for improving world models rather than a world model itself [1][3].

## Publication and authorship

The paper's authors are Ahmet H. Güzel (corresponding author, University College London AI Centre and Odyssey), Jenny Seidenschwarz, Benjamin Graham, Jonathan Sadeghi and Jeffrey Hawke (all Odyssey), and Ilija Bogunovic (University of Basel and University College London AI Centre) [2]. Odyssey's blog credits the first five as the team behind the work and describes Bogunovic as a University College London advisor who gave feedback; the paper itself lists him as a co-author [1][2]. Odyssey's post links directly to the arXiv preprint and offers a BibTeX download [1].

| Item | Value |
| --- | --- |
| Announcement | "Introducing PROWL-1: Learning Through Discovery", Jeff Hawke, 12 May 2026 [1] |
| Paper | arXiv:2605.18803, cs.LG and cs.AI, v1 11 May 2026, v2 29 May 2026 [2] |
| Environment | MineRL, with BASALT human demonstrations [2] |
| World model | Wan2.1-T2V-1.3B diffusion transformer, action-conditioned [2] |
| Adversary | PPO policy initialized from and anchored to VPT-2x [2] |

## The problem PROWL targets

Action-conditioned video world models can generate long, visually convincing rollouts and still get the immediate response to a control input wrong. Long-horizon visual evaluations, the paper argues, "often obscure a critical failure mode: models can produce visually plausible trajectories while failing to correctly respond to actions at short horizons" [2]. For planning and control, small errors in the first few transitions compound through autoregressive rollout, so short-horizon action fidelity is the thing worth fixing first [2].

The second half of the argument is about data. Human demonstration datasets are dominated by common, successful behaviour and systematically under-sample the rare, interaction-critical transitions where a model is most likely to break, and the gap grows rather than shrinks as action spaces get more combinatorial [2]. PROWL's answer is to generate the missing data on purpose.

## Method

### Phase 1: passive pretraining

Phase 1 trains an action-conditioned [diffusion transformer](https://aiwiki.ai/wiki/diffusion_transformer) world model on passive BASALT FindCave demonstrations using the [Diffusion Forcing](https://aiwiki.ai/wiki/diffusion_forcing) objective, which allows different noise levels across time steps so that teacher-forced prediction and full-sequence generation share one formulation [2][4]. Frames are encoded by a frozen [VAE](https://aiwiki.ai/wiki/variational_autoencoder) into latents grouped into chunks of three latent frames, which corresponds to 12 pixel frames at 20 fps (0.6 seconds) given the VAE's temporal compression ratio of four. History frames receive low noise, target frames higher noise, and the loss is applied only to the target chunk [2].

The backbone is the publicly released [Wan2.1](https://aiwiki.ai/wiki/wan_2_1)-T2V-1.3B diffusion transformer [5]. Its cross-attention pathway, originally built for text conditioning, is repurposed for action conditioning: each chunk's per-frame MineRL action vector is serialized into a short text token such as `forward jump look(up,left)`, run-length-encoded, and pushed through the frozen UMT5-XXL encoder, whose embeddings the cross-attention layers consume exactly as they would text embeddings [2]. Phase 1 updates all of the roughly 1.3 billion Wan2.1 parameters with the VAE and the text encoder frozen, and the 10k-step checkpoint was selected to initialize Phase 2 [2].

### Phase 2: the adversarial curriculum

Phase 2 is written as an asymmetric two-player game between the world model and an adversarial policy: the policy maximizes a trajectory score measuring how wrong the world model was, minus a Kullback-Leibler penalty pulling it back toward a frozen behavioural reference; the world model minimizes its loss on a mixture of passive data and the adversary's discoveries [2]. The paper is explicit that this is an alternating, memory-mediated procedure rather than a simultaneous saddle-point optimization: the policy searches against a fixed model, which is then fine-tuned in batches on a prioritized mixture of past failures and passive data [2].

The adversary is initialized from the pretrained VPT-2x foundation model (the `rl-from-early-game-2x` checkpoint) and optimized with [PPO](https://aiwiki.ai/wiki/ppo), with the same frozen checkpoint serving throughout as the behavioural reference [2][6][7]. The KL anchor is the mechanism Odyssey describes as keeping the agent "constrained to stay near to realistic exploration behavior so that it does not exploit the model with unnatural actions" [1]. During Phase 2 the world model's spatial-temporal backbone is frozen and only the action-conditioning subset is updated: the cross-attention layers and the UMT5 action-text adapter, roughly 280 million of the 1.3 billion parameters trained in Phase 1 [2].

### The trajectory score

Each candidate trajectory is scored with three ingredients [2]:

- **Latent regret**, the RMSE between predicted and real VAE latents over the predicted horizon. Algorithm 1 uses this alone as the terminal reward handed to PPO (see the note above on the paper's inconsistency).
- **Action-Follow Score (AFS)**, an endpoint error between the optical flow of the predicted decoded frames and the optical flow of the real frames, computed at native 480x832 resolution with SEA-RAFT [8].
- **Learning progress**, the signed change in the same trajectory's latent regret between consecutive buffer rescoring cycles.

The first two terms are z-normalized over the current buffer and combined with a weight on the AFS term; the progress term is added in raw units. The paper argues the flow term is not redundant with appearance metrics: on a held-out action-heavy sequence combining forward-left locomotion with sustained camera rotation, latent regret, LPIPS and SSIM stayed nearly tied between the Phase 1 and PROWL checkpoints while the flow error showed a two-to-three-fold gap, because appearance metrics reward a plausible-looking frame even when it depicts the wrong motion [2].

### The PAT buffer

The Prioritized Adversarial Trajectory (PAT) buffer converts a stream of failures into a curriculum. It adapts Prioritized Level Replay, which samples training instances by a mixture of difficulty score and staleness, but ranks trajectories rather than environment levels and uses world-model prediction error as the notion of difficulty [2][9]. The buffer holds 256 trajectories, evicting the lowest-priority entry when full, and every entry is rescored under the updated model after each world-model cycle. Odyssey's post describes the effect plainly: as the world model learns to solve easier failure cases, PAT deprioritizes them and brings harder unresolved trajectories into training [1].

Each adversarial iteration runs a six-step cycle: roll out one episode, score it against the ground truth, insert it into the buffer, assign latent regret as the terminal PPO reward, update the policy every 16 episodes, and fine-tune the world model every 24 iterations on a 50/50 mixture of buffer and passive samples followed by a full rescore [2]. Decoupling frequent policy updates from infrequent model updates is what keeps the loop tractable.

### Training configuration

| Component | Setting | Value |
| --- | --- | --- |
| World model | Wan2.1 backbone, dim / layers / heads | 1536 / 30 / 12 [2] |
| | FFN dim / chunk size / frames | 8960 / 3 / 21 [2] |
| | VAE compression (T, H, W) | (4, 8, 8) [2] |
| Fine-tuning | Learning rate, batch size, precision | 1e-5, 1, bfloat16 [2] |
| | PAT to passive replay ratio, cycle cadence | 0.5 : 0.5, every 24 adversarial iterations [2] |
| | Hardware | 7x NVIDIA H200, DDP [2] |
| PAT buffer | Capacity, staleness weight, default AFS weight | 256 trajectories, 0.1, 0.25 [2] |
| PPO policy | Learning rate, clip, entropy coefficient | 3e-5, 0.2, 0.05 [2] |
| | Update cadence, epochs / minibatch | every 16 episodes, 4 / 256 [2] |
| Rollout | Seed / horizon chunks | 2 / 2, i.e. 12 + 12 pixel frames [2] |
| | Context window, CFG scale, sampler steps | 21 latent frames, 1.5, 20 (DPM) [2] |
| | Resolution | 480x832 px [2] |

Six configurations were trained along two axes: the KL anchor strength at values 0.5, 1.0 and 1.5 with the AFS weight held at 0.25, and the AFS weight at 0.05 and 0.10 with the anchor held at 1.0 [2]. Two operating points carry most of the reported results: a broad-exploration arm ("lam010", anchor 1.0 and AFS weight 0.10) and a focused-specialist arm ("kl150", anchor 1.5 and AFS weight 0.25) [2].

## Evaluation in MineRL

PROWL was implemented and evaluated in the MineRL framework, the Minecraft research platform introduced with a large-scale dataset of human demonstrations by Guss and colleagues in 2019 and used for a series of NeurIPS competitions [2][10]. The demonstration data comes from BASALT, the MineRL competition track built around tasks with no programmatic reward function, including MakeWaterfall, BuildVillageHouse and CreateVillageAnimalPen [2][11].

Three checkpoints are compared throughout: the Phase 1 pretrained model, PROWL, and a matched-compute Phase 2 baseline that runs the identical fine-tuning loop, optimizer, mixture ratio and update budget but sources its rollouts from the frozen VPT reference policy instead of an adversarial one [2]. That baseline, "Phase 2 (VPT-frozen)", isolates what adversarial discovery adds beyond simply spending more fine-tuning compute. Short-horizon evaluation uses two seed chunks and two predicted chunks (2.4 seconds); the long-horizon setting extends to 18 predicted chunks (10.8 seconds). Metrics are latent regret, AFS endpoint error and LPIPS (lower is better), plus SSIM at long horizon (higher is better) [2].

### Held-out BASALT tasks

Phase 1 trained on FindCave only, so the other three BASALT tasks are out of distribution for every training source, including the baseline's rollouts. Across 300 human clips (100 per task), PROWL's broad-exploration arm improved on every metric, with the margin widening on the hardest subsets ranked by Phase 1 latent regret [2].

| Method | Mean latent | Mean AFS | Mean LPIPS | Top 10% latent | Top 10% AFS | Top 10% LPIPS |
| --- | --- | --- | --- | --- | --- | --- |
| Phase 1 world model | 0.6514 | 28.16 | 0.5089 | 0.9888 | 37.14 | 0.6717 |
| Phase 2 (VPT-frozen) | 0.6446 | 25.61 | 0.4990 | 0.9173 | 31.56 | 0.6414 |
| PROWL (lam010) | 0.6288 | 24.60 | 0.4952 | 0.9026 | 29.38 | 0.6333 |

Relative to Phase 1, the lam010 arm cut mean latent regret by 3.5 percent, AFS by 12.6 percent and LPIPS by 2.7 percent; on the hardest tenth of clips those figures were 8.7, 20.9 and 5.7 percent. Against the matched-compute baseline the gains were smaller but consistent, at 2.5, 3.9 and 0.8 percent on the mean and 1.6, 6.9 and 1.3 percent on the hardest tenth [2].

### Hard cases, stability and long horizon

The focused-specialist arm was tested on three evaluations it was predicted to suit: cross-buffer adversarial trajectories (384 trajectories, 64 from each of six PAT buffers, scored strictly off-diagonal so no checkpoint is graded on its own buffer), 64 held-out FindCave clips checking for catastrophic forgetting, and an 18-chunk rollout measured at frame 200 [2].

| Method | Cross-buffer latent | Cross-buffer AFS | FindCave latent | FindCave AFS | Long-horizon LPIPS | Long-horizon SSIM |
| --- | --- | --- | --- | --- | --- | --- |
| Phase 1 world model | 0.8261 | 39.87 | 0.5566 | 28.96 | 0.7432 | 0.4226 |
| Phase 2 (VPT-frozen) | 0.7265 | 35.26 | 0.5444 | 28.17 | 0.7447 | 0.4491 |
| PROWL (kl150) | 0.6846 | 32.13 | 0.5255 | 26.09 | 0.6947 | 0.4685 |

These are the largest margins over the matched-compute baseline reported in the paper, including an 8.9 percent reduction in AFS endpoint error on cross-buffer adversarial trajectories and a 6.7 percent reduction in long-horizon LPIPS [2]. The FindCave panel is the evidence that the 50/50 passive mixture prevents forgetting of the pretraining distribution; the long-horizon panel is presented as evidence that short-horizon gains compound through autoregressive rollout, not as a directly optimized target [2].

### Novel action compositions

To test whether the adversary reaches behaviour that passive data and non-adversarial fine-tuning never produce, each rollout window is summarized by a deterministic action fingerprint built from hard thresholds on camera rotation and button-press frequency, with no learned classifier [2]. A composite mode counts as strictly novel when it has zero occurrences in the BASALT and VPT data and zero in the VPT-frozen buffer. Across the 27 strictly novel modes found this way, PROWL improved the cross-mode mean by 5.5 percent on latent regret, 11.5 percent on AFS and 3.4 percent on LPIPS over Phase 1, and by 4.5, 9.1 and 3.5 percent over the matched-compute baseline [2].

## Reward hacking and the KL anchor

The most quotable negative result in the paper concerns the weakly anchored arm. With the KL coefficient set to 0.5, the adversary did generate high prediction error, but it did so by thrashing the camera: its measured camera-action velocity rebounded late in training to roughly twice that of the stable arms, and it contributed zero entries to the strictly novel composite modes [2]. Its buffer regret stayed high without the world model learning anything transferable from it. The paper labels this [reward hacking](https://aiwiki.ai/wiki/reward_hacking) and draws the general conclusion that prediction error alone is not a usable reward signal, because the adversary must surface failures the world model can productively absorb [2][12].

The authors connect that to the novelty-and-learnability criterion from the open-endedness literature, noting that the KL-anchored regimes deliver both while the unanchored regime delivers only novelty [2][13]. The KL coefficient and the AFS weight therefore define a spectrum with three regimes: an unanchored one that reward-hacks, a broad-exploration one covering the widest variety of action compositions, and a focused-specialist one concentrating on behaviourally valid hard cases.

## Qualitative capabilities reported by Odyssey

Odyssey's announcement shares a set of qualitative behaviours the team observed, framed as a sample rather than as measurements, and points readers to the preprint for quantitative results [1]. The rows below paraphrase those observations. They are the company's own reported comparisons against its baseline, not independently verified benchmark results.

| Reported capability | What Odyssey describes |
| --- | --- |
| Action-following | PROWL-1 follows the intended action more faithfully than the baseline, which often predicts the wrong direction or ignores controls entirely [1] |
| Visual quality | Persistent artifacts such as rotation seams, colour banding, popping geometry and unstable textures are removed [1] |
| Crosshair persistence | 3D-anchored interface elements such as the block-placement crosshair and held items stay stable in world space as the camera moves, where the baseline lets them drift, warp or vanish [1] |
| Emergent behaviour | An RL-discovered 180-degree pivot and dash lies outside the human-demonstration distribution; PROWL-1 renders it faithfully while the baseline collapses mid-turn [1] |
| Temporal coherence | Hard scene transitions such as closing the inventory and returning to the game show less ghosting, flicker and incorrect re-rendering [1] |
| Out-of-distribution robustness | Higher visual and dynamical quality is maintained in held-out Minecraft scenes, with less drift and fewer incorrect scene evolutions [1] |
| Environmental dynamics | Non-rigid motion such as flowing waterfalls is captured, where the baseline freezes, distorts or hallucinates the dynamics [1] |
| Object placement and persistence | Object position, scale and identity are preserved under camera motion, and a block the agent places stays solid and anchored [1] |

Odyssey's summary line, that scalable world models benefit not only from larger datasets but also from selectively generating informative training data, appears in both the post and the paper's abstract, in both cases as a conclusion the authors draw rather than as a measured quantity [1][2].

## Limitations

The paper states four limitations directly [2]. Not all adversarial data is useful: the unanchored arm's high-error trajectories transferred poorly. Every configuration was run with a single seed, a deliberate choice to spend the compute on mapping the two-parameter regime axis rather than on variance estimates, with multi-seed evaluation left as future work. The long-horizon evaluation sits outside the horizon where adversarial pressure is actually applied, so it should be read as evidence of compounding rather than as a direct optimization target. And generalization beyond MineRL and BASALT is untested.

Phase 2 also updates only the action-conditioning parameters, leaving the visual prior from pretraining intact, which the authors flag as a design choice to revisit [2].

## Place in Odyssey's research

Odyssey builds foundation world models, and PROWL is its mechanism for improving them from interaction rather than from more collected video. The [Agora-1](https://aiwiki.ai/wiki/agora_1) announcement of 18 May 2026 refers back to PROWL when arguing that single-participant world models limit the reinforcement learning environments a simulator can support [14]. The [Odyssey-3](https://aiwiki.ai/wiki/odyssey_3) announcement of 15 September 2026 cites PROWL under its "Odyssey-3 Can Train AIs" section, describing agents that uncover failures which guide improvements to the world model while the simulated experience trains the agents in turn [15]. Odyssey's July 2026 essay "Our Path to Superintelligence" calls PROWL-1 the experiment that answered whether RL agents can improve world models by discovering where they fail [16], and a June 2026 essay proposes a next generation of PROWL hunting for strategic rather than visual failures [17].

The general shape, a system generating the training data for its own next iteration, places PROWL alongside work on [recursive self-improvement](https://aiwiki.ai/wiki/recursive_self-improvement) and automated [curriculum learning](https://aiwiki.ai/wiki/curriculum_learning), though the paper claims nothing about capability escalation beyond the settings it measures.

## Relation to prior work

The paper positions PROWL against three neighbouring literatures [2]:

| Line of work | How PROWL differs |
| --- | --- |
| World models for RL (Dreamer family, transformer and diffusion variants) | Those train a policy inside the model to solve a task; PROWL trains the policy to expose the model's failures so the model itself improves |
| Active data acquisition (Plan2Explore, Ready Policy One) | Those use ensemble disagreement or model variance; PROWL uses realized prediction regret against ground-truth rollouts, needs no ensemble, and KL-anchors the adversary so failures stay learnable |
| Adversarial curricula, prioritized replay and unsupervised environment design (PAIRED, ACCEL, PLR) | Those generate challenging environments to make an agent robust; PROWL adapts the prioritized buffer to train the simulator instead |

The paper cites the unsupervised environment design line explicitly in its formulation and related-work sections, and summarizes the distinction as training "the simulator, not the agent inhabiting it" [2]. Its [experience replay](https://aiwiki.ai/wiki/experience_replay) priorities follow Prioritized Level Replay's difficulty-plus-staleness sampling, rewritten over trajectories [2][9].

## References

1. Jeff Hawke, "Introducing PROWL-1: Learning Through Discovery", Odyssey, 12 May 2026. https://odyssey.systems/introducing-prowl-1
2. Ahmet H. Güzel, Jenny Seidenschwarz, Benjamin Graham, Jonathan Sadeghi, Jeffrey Hawke, Ilija Bogunovic, "PROWL: Prioritized Regret-Driven Optimization for World Model Learning", arXiv:2605.18803, 11 May 2026 (v2, 29 May 2026). https://arxiv.org/abs/2605.18803
3. Odyssey, "Leading World Model Research". https://odyssey.systems/research
4. Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, Vincent Sitzmann, "Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion", arXiv:2407.01392. https://arxiv.org/abs/2407.01392
5. Wan Team, "Wan: Open and Advanced Large-Scale Video Generative Models", arXiv:2503.20314. https://arxiv.org/abs/2503.20314
6. Bowen Baker et al., "Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos", arXiv:2206.11795. https://arxiv.org/abs/2206.11795
7. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, "Proximal Policy Optimization Algorithms", arXiv:1707.06347. https://arxiv.org/abs/1707.06347
8. Yihan Wang, Lahav Lipson, Jia Deng, "SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow", arXiv:2405.14793. https://arxiv.org/abs/2405.14793
9. Minqi Jiang, Edward Grefenstette, Tim Rocktäschel, "Prioritized Level Replay", arXiv:2010.03934. https://arxiv.org/abs/2010.03934
10. William H. Guss et al., "MineRL: A Large-Scale Dataset of Minecraft Demonstrations", arXiv:1907.13440. https://arxiv.org/abs/1907.13440
11. Rohin Shah, Cody Wild, Steven H. Wang, Neel Alex, Brandon Houghton, William Guss, Sharada Mohanty, Anssi Kanervisto, Stephanie Milani, Nicholay Topin, Pieter Abbeel, Stuart Russell, Anca Dragan, "The MineRL BASALT Competition on Learning from Human Feedback", arXiv:2107.01969. https://arxiv.org/abs/2107.01969
12. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, David Krueger, "Defining and Characterizing Reward Hacking", arXiv:2209.13085. https://arxiv.org/abs/2209.13085
13. Edward Hughes et al., "Open-Endedness is Essential for Artificial Superhuman Intelligence", arXiv:2406.04268. https://arxiv.org/abs/2406.04268
14. Odyssey, "Agora-1: The Multi-Agent World Model", 18 May 2026. https://odyssey.systems/introducing-agora-1
15. Odyssey, "Introducing Odyssey-3: A General-Purpose Physical Intelligence", 15 September 2026. https://odyssey.systems/introducing-odyssey-3
16. Odyssey, "Our Path to Superintelligence", 8 July 2026. https://odyssey.systems/our-path-to-superintelligence
17. Odyssey, "The Era of Multi-Agent Imagined Experience", 12 June 2026. https://odyssey.systems/the-era-of-multi-agent-imagined-experience