SPADE (Self-Play in Adaptive Synthetic Executable Environments)

SPADE, short for Self-Play in Adaptive Synthetic Executable Environments, is a reinforcement-learning framework in which one language model alternates between designing executable training environments and learning to solve them. The Environment Designer writes a Python program with state, actions, rewards, and verification logic. The Reasoning Agent then interacts with that program. Both roles share one set of model weights and are updated in the same training loop.^[1][2]

The framework was introduced in an arXiv v1 preprint submitted on Aug. 19, 2026. The paper labels itself a work in progress. Its authors report results with Qwen3 models in cognitive-game and simulated tool-use settings, and released code, data, and some checkpoints. No independent replication or peer-reviewed evaluation of SPADE was located as of Aug. 21, 2026.^[1][3]

SPADE is sometimes described as a step toward open-ended or continuous self-improvement. Its implemented scope is narrower. It adapts which environments a model trains on and updates the model through a fixed reinforcement learning recipe. Humans still choose the base model, corpora, skill categories, prompts, validation rules, reward mixture, optimizer, schedule, and evaluation suite. SPADE does not rewrite its own training code or learning rule.^[1]

Key facts

Scroll sideways for more →

Architecture

One model, two prompted roles

SPADE uses one shared large language model policy with different system prompts for its two roles. In the Environment Designer role, the policy emits a self-contained program implementing a Gym-style reset and step interface. The step method encodes the state transition and reward function, so the generated file is an environment and verifier rather than only a natural-language question. A one-step environment can grade one answer; a multi-turn environment can maintain hidden state, branch on actions, return partial rewards, and terminate after a sequence of interactions.^[1]

The same policy then takes the Reasoning Agent role. It receives observations from the environment and returns sequential actions. The generated program evaluates those actions. The training cycle first collects designer trajectories, then Reasoning Agent trajectories from the accepted environments, and finally applies one joint update to the shared weights. Role-specific prompts separate the behaviors, but an update learned in either role can affect both.^[1]

This design gives the model more expressive task construction than choosing parameters inside a fixed maze or question template. It also gives generated code control over task state and reward. That flexibility creates two separate questions: whether an environment teaches useful behavior and whether its executable implementation is correct and safe.

Generation, validation, and play

A cycle begins with a domain or skill prompt, a newly sampled corpus document, and selected records from an environment memory. The designer writes an environment, after which a programmatic smoke test instantiates its class, calls reset, and steps several probe actions. A rejected candidate can be regenerated. Accepted environments enter a temporary training pool.^[1][17]

The tool-use setting adds two semantic checks. A deterministic reset gate tests whether success criteria fail on newly reset states under several seeds. An LLM screen attempts to identify unreachable, already satisfied, or underivable criteria. These checks reduce obvious failures, but neither proves that the verifier matches the intended task. The paper reports that 84.9 percent of raw generated games ran as emitted and 90.3 percent ran after removing a stray Markdown fence. Its sanitizer and filter reached 100 percent post-filter executability in the analyzed sample. That figure measures whether code passed the pipeline, not whether every reward was correct or the program was secure.^[1]

After validation, another designer-side call sees the environment source and writes a privileged hint. The Reasoning Agent never sees the source code. It plays the environment both without the hint and with it. The unhinted trajectories train the Reasoning Agent, while the difference between the two arms helps train the Environment Designer.^[1]

Objectives and reward semantics

Reasoning Agent reward

The Reasoning Agent receives the scalar reward produced by each generated environment. The algorithm presents this return on a scale from -1 to 1. A game can use partial rewards over several turns. In the reported tool-use configuration, the Reasoning Agent receives success reward only after completing every instruction. Because the generated program owns the verifier, a high return means success under that program's rules. It is not external proof that the task definition is valid.^[1]

Hint-based regret

For an environment e and hint h, the raw Environment Designer signal is:

r_D(e) = mean Reasoning Agent return with h - mean Reasoning Agent return without h

The main experiments use 16 fresh plays in each arm at the regeneration step. A large positive gap is interpreted as a task near the Reasoning Agent's learning frontier: the agent can solve it with guidance but not yet reliably without guidance. A task solved in both arms is already mastered. A task failed in both arms is probably too difficult or defective.^[1]

The actual training reward is not the raw gap alone. Negative regret is floored at zero and normalized by a fixed scale. This component has weight 0.4. A flat-top difficulty anchor has weight 0.6 and rewards environments whose observed unhinted win rate falls in a target band, normally 0.4 to 0.6, with a linear decline outside that band. The difficulty anchor therefore regulates broad solvability while hint-regret distinguishes the more teachable environments inside that region.^[1]

Environment sets remain active for several rollouts so the difficulty anchor can observe repeated play. The designer update is delayed by the same interval, which makes its data off-policy; the authors apply truncated importance sampling. They also normalize advantages separately by role. Reasoning Agent returns are standardized within an environment's rollout group, while designer rewards are mean-centered within a skill. Less frequent designer trajectories are upweighted before the shared GRPO update.^[1]

The paper relates hint-regret to minimax regret under assumptions that include a hint-equipped policy reaching the optimum for each environment and the existence of a uniformly optimal policy across the considered environment class. Those assumptions do not hold automatically in a stochastic model and model-generated task. The authors explicitly say the implemented signal is motivated by regret theory but is not proved to yield an optimal curriculum.^[1][13]

Adaptive curriculum

Corpus grounding

A generator conditioned only on its prior outputs can repeat familiar patterns. SPADE samples external documents to broaden the subject matter of new environments. The games setting uses 10,000 mathematics documents and 5,000 science documents drawn from DCLM and MegaScience. The tool-use setting uses 15,000 documents from a Nemotron pretraining code corpus, including algorithm implementations and API documentation.^[1]

The paper attributes most measured environment diversity to this grounding. In a 30B games analysis, the normalized Vendi statistic was 0.68 for full SPADE and 0.04 after removing the corpus. During one no-corpus interval, the designer emitted the same rotating-maze environment 41 consecutive times. The statistic depends on the authors' embedding and sampling procedure, but the ablation supports a specific conclusion: an external corpus reduced mode collapse in that experiment.^[1]

Environment memory

Corpus context determines what a task can be about. Environment memory supplies information about what has already been tried and how difficult it was. The memory stores past environments with regret scores and skill tags. High-regret records become seeds for variation, while mastered and intractable examples can be shown as patterns to avoid. The released implementation caps this buffer at 200 records and removes the oldest entries first. The active environment pool is replaced at each regeneration.^[1][3]

This mechanism resembles curriculum learning, but the curriculum is not wholly discovered by the model. The six game skills, the corpus sources, the memory schema, the target win-rate band, and the regeneration schedule are fixed by the researchers. The adaptive component selects and generates concrete environments within that scaffold.

Training settings

The main experiments train three backbones: Qwen3-4B-Instruct-2507, Qwen3-8B, and Qwen3-30B-A3B-Instruct-2507. The last model is a sparse mixture of experts and is the primary scale studied. The 4B and 30B runs use instruct models. The 8B run enables thinking behavior for both roles.^[1]

The games recipe runs 400 rollouts with 24 environments per rollout and 16 Reasoning Agent samples per group. Six skills rotate three at a time: mathematical reasoning, logical deduction, spatial reasoning, pattern recognition, optimization, and causal inference. The environment set regenerates every four rollouts. The authors use a constant learning rate of 0.000001 and permit at most 25 interaction turns per episode.^[1]

Tool-use environments instead define simulated tools in OpenAI function calling format, a backend state changed by those tools, and three to five instructions that arrive one at a time. Each instruction has a state-based success check. These environments are more expensive to generate, so their set regenerates every eight rollouts. The paper says no benchmark examples or evaluation data are deliberately shown to the Environment Designer.^[1]

The public repository contains a backend-independent core, distributed training through Slime with SGLang inference, Megatron-LM policy updates, and Ray orchestration, plus a Tinker integration. A representative README command describes one 30B games run on a single eight-GPU node. The public materials do not give the GPU model, total GPU-hours, complete wall time, energy use, or a cost ledger.^[3]

Evaluation

Cognitive games and held-out benchmarks

The games experiments evaluate eight metrics across four groups: AIME 2025 and 2026 with Avg@32, GPQA-Diamond accuracy, LiveCodeBench-v6 Pass@1, and four hard-difficulty Reasoning-Gym win rates. The authors retrain two fixed-environment baselines for 400 iterations from each same base model. One uses the official RLVE environment set, and the other uses a static pool generated by GPT-5.5.^[1][14]

For Qwen3-30B-A3B, the unweighted eight-metric average is 50.2 for the base model, 51.4 for static-pool GRPO, 53.0 for fixed-environment RLVE, and 58.3 for SPADE. The often cited +5.3 result is therefore an absolute-point gain over the strongest fixed-environment baseline. The gain over the untrained base is +8.1. These are different comparison bases.^[1]

At 4B, the suite average changes from 38.9 for the base to 44.1 for SPADE, a +5.2 point difference. At 8B, it changes from 49.8 to 55.5, a +5.7 point difference. At 30B-A3B, the largest gains over the base occur on the four Reasoning-Gym categories. AIME changes are much smaller, so the aggregate should not be paraphrased as uniform improvement across all reasoning tasks.^[1]

An appendix reports a separate run on Nemotron-30B-A3B-BF16. The four Reasoning-Gym endpoint gains over that model's untrained base are +2.6 for math, +9.3 for algorithmic reasoning, +9.6 for cognition, and +3.2 for logic. AIME 2025/2026, GPQA-Diamond, and LiveCodeBench-v6 were not evaluated for this arm, and the paper does not provide a matched cross-family or fixed-environment comparison. This is a restricted second-backbone-family check, not broad evidence of transfer across model families.^[1]

Tool use

The tool-use suite covers BFCL v4 multi-turn, tau2-bench in retail, airline, and telecom domains, and ACEBench-Agent multi-step and multi-turn tasks. At 30B-A3B, the authors report BFCL increasing from 49.0 to 54.7, tau2-bench from 49.0 to 52.6, and ACEBench-Agent from 62.0 to 75.9. These are absolute changes of +5.7, +3.6, and +13.9 points. The three-benchmark aggregate changes from 53.3 to 61.1, reported as +7.7 because the aggregate is calculated before displayed values are rounded.^[1]

The paper also prints results from AgentScaler, Agent-World, AWM, and EnvScaler. Those rows were transcribed from their respective papers. The training data, compute budgets, base models, omitted benchmarks, and evaluation protocols differ, so the table is contextual rather than a controlled common-protocol leaderboard.^[1]

Ablations and checkpoint selection

In the 30B games ablation, the eight-metric average is 58.3 for full SPADE, 53.2 without memory, 53.5 without corpus grounding, 40.5 without designer training and memory, and 53.0 with a frozen GPT-5.5 designer. Replacing hint-regret with an exponential-moving-average learning-potential signal reaches 55.9.^[1]

Each ablation reports the best checkpoint on the same eight-benchmark suite used in the table. The paper's expanded table identifies checkpoint 303 for the full 30B games run. The immutable public 30B games model card instead labels the released artifact iteration 79, says it was the best of 21 evaluated checkpoints, and says later scores declined. The public materials do not explain whether these numbers describe different runs or how iteration 79 maps to checkpoint 303, so the released artifact cannot be equated directly with the paper-table checkpoint. Selecting on the displayed evaluation suite can also make results optimistic relative to a predeclared final checkpoint. The 8B games card instead identifies its final checkpoint and says no offline sweep was run for that arm.^[1][16]

Released artifacts and licenses

The public GitHub repository snapshot checked for this article is commit 65421ccb15a6d501ad6217bd969816146da15e11, dated Aug. 20, 2026. It had one public commit and no tags or GitHub releases. Package metadata declares version 0.1.0 and supports Python 3.10 through 3.12. That mutable branch version is not a tagged release.^[3][4]

The core source is MIT-licensed. Vendored RLVE code retains MIT terms, while adapted BFCL and PRIME evaluation components use Apache-2.0 terms. The third-party notice says the exact upstream revisions for its BFCL and PRIME snapshots were not recorded. It also warns that the source license does not determine the licenses or redistribution rights of external datasets.^[4][5]

The README and Hugging Face collection description say six checkpoints were released, covering games and tool use at 4B, 8B, and 30B-A3B. At the research cutoff, however, the collection entries and public model API exposed only three games checkpoints. Their immutable revisions and card licenses were:^[6][7]

Scroll sideways for more →

No public tool-use model appeared in that inventory. Six dataset repositories were public. Five cards used Apache-2.0 metadata: two 15,000-document grounding corpora, generated 30B games environments, generated tool-use environments, and the static GPT-5.5 games pool. The static GPT-5.5 tool-use pool used the label other. Because the repository notice also records unresolved provenance or redistribution details for some corpora, the complete release should not be assigned one blanket license.^[5][8]

Security and trust boundary

SPADE environments are arbitrary Python programs. In the pinned source, SyntheticGameEnv creates an in-process module and calls Python exec on the generated source. The namespace includes standard-library modules, and generated code can import additional modules or execute statements when it loads.^[9]

The orchestrator puts validation and environment calls in a thread-pool future and stops waiting after a timeout. This limits how long the training loop waits for a hung game, but it is not process isolation. A timed-out Python thread cannot be killed safely by that mechanism and can retain access to the parent process, filesystem, environment variables, network, and credentials. The source comments acknowledge that hung workers cannot be killed.^[10]

The paper says it re-executed raw generations in a sandbox for one executability analysis. That does not establish that the released training path sandboxes every environment. Smoke tests, timeouts, deterministic reset checks, and an LLM feasibility screen can reject malformed tasks, but they are not malicious-code controls.^{[1][9][10][17]}

Generated code can therefore read or alter available files, expose secrets, make network calls, spawn processes, exhaust memory, or create persistent side effects. A subtler risk is reward hacking: the environment's verifier can be incomplete or exploitable even when the program runs. The same model family may also generate a flawed task, hint, and validation judgment with correlated blind spots.

Operators should treat every generated environment as untrusted code. A safer deployment uses a separate disposable process or container, no credentials, network denied by default, strict CPU, memory, process, output, and wall-time quotas, minimal read-only inputs, and no writable host mounts. This recommendation follows from the released execution boundary; the paper does not report a deliberate exploit.

Relation to prior work

SPADE builds on several distinct self-play traditions. SPIRAL trains language models through multi-agent, multi-turn zero-sum language games. SPADE follows its practice of normalizing advantages separately by role, but the new designer emits complete executable environments rather than only playing a fixed game.^[11]

SPICE uses information asymmetry between a corpus-grounded Challenger and a Reasoner to generate question-answer tasks. SPADE carries that grounding principle into stateful programs with transitions, intermediate actions, rewards, and verification code. The Environment Designer is updated online together with the Reasoning Agent.^[12]

PAIRED introduced minimax-regret environment design with a protagonist, antagonist, and adversarial designer in parameterized spaces. SPADE uses the hinted Reasoning Agent as an approximate upper-bound policy instead of training a separate antagonist. The analogy motivates hint-regret but does not establish the same guarantee for model-generated code.^[13]

POET explored open-ended co-evolution of environments and agents through evolutionary search, while Absolute Zero Reasoner later applied shared-model self-play to verifiable reasoning tasks. SPADE's distinguishing combination is a learned shared-policy designer, a code-defined multi-turn environment, corpus grounding, cross-episode memory, and paired hinted and unhinted returns.^[1][15]

Reproducibility and limitations

Reproducing a reported run requires the repository commit, exact base checkpoint, submodule revisions, corpus and environment-dataset revisions, prompts, memory state, random seeds, reward weights, model-serving stack, evaluation data, and checkpoint-selection rule. It also requires substantial compute that is not fully specified. Hosted dependencies and mutable benchmark data can change independently of the repository.^[1][3]

The public release improves inspectability but is incomplete. Only three of the claimed six checkpoints were visible, exact upstream revisions for some adapted evaluators were not recorded, and the public run log does not retain every per-environment scalar reward used in the analysis. Artifact cards and the repository notice also leave conflicting or artifact-specific licensing boundaries.^[5][6][7][8]

The empirical scope is three main Qwen3 backbones, one restricted Nemotron appendix run, six cognitive-game skills, simulated tool APIs, and a fixed set of benchmarks. The Nemotron arm covers only four Reasoning-Gym categories, so the paper does not show broad transfer across unrelated model families, real production tools, physical systems, or adversarial environments. Large external corpora can broaden task generation but can also contain errors or concepts overlapping evaluation tasks. The statement that benchmark examples were not deliberately supplied to the designer is not a complete contamination audit.^[1]

SPADE's designer is bounded by the base model, context window, prompts, and compute budget. Its verifier can be wrong, its hint can reveal too much or too little, and its regret estimate has sampling noise. Best-checkpoint selection on the evaluation suite further limits the interpretation of reported gains. Most importantly, the fixed GRPO optimizer means the system adapts its task distribution and weights, not its own learning algorithm.^[1]

As of the cutoff, SPADE remained an author-evaluated arXiv work in progress. Its results are evidence that adaptive executable environments helped the tested models under the reported setup. They are not evidence of general recursive self-improvement, secure autonomous code execution, or independently established open-ended capability growth.

See also

• Recursive self-improvement

References

1. ^Bo Liu et al., "SPADE: Self-Play in Adaptive Synthetic Executable Environments," arXiv:2608.19197v1, Aug. 19, 2026, arxiv.org/...2608.19197.
2. ^SPADE team, "SPADE project page," accessed Aug. 21, 2026, spade-rl.github.io.
3. ^SPADE team, "SPADE repository README," pinned commit 65421ccb15a6d501ad6217bd969816146da15e11, Aug. 20, 2026, github.com/...README.md.
4. ^SPADE team, "SPADE package metadata and license declaration," pinned commit 65421ccb15a6d501ad6217bd969816146da15e11, Aug. 20, 2026, github.com/...pyproject.toml.
5. ^SPADE team, "Third-party notices," pinned commit 65421ccb15a6d501ad6217bd969816146da15e11, Aug. 20, 2026, github.com/...THIRD_PARTY_NOTICES.md.
6. ^SPADE team, "SPADE models and data collection," Hugging Face, accessed Aug. 21, 2026, huggingface.co/...spade.
7. ^Hugging Face, "Public model inventory for spade-rl," full API response accessed Aug. 21, 2026, huggingface.co/...models.
8. ^Hugging Face, "Public dataset inventory for spade-rl," API response accessed Aug. 21, 2026, huggingface.co/...datasets.
9. ^SPADE team, "SyntheticGameEnv dynamic environment loader," pinned commit 65421ccb15a6d501ad6217bd969816146da15e11, Aug. 20, 2026, github.com/...synthetic_game_env.py.
10. ^SPADE team, "SPADE training orchestrator and environment timeouts," pinned commit 65421ccb15a6d501ad6217bd969816146da15e11, Aug. 20, 2026, github.com/...orchestrator.py.
11. ^Bo Liu et al., "SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning," arXiv:2506.24119, 2025, arxiv.org/...2506.24119.
12. ^Bo Liu et al., "SPICE: Self-Play in Corpus Environments Improves Reasoning," arXiv:2510.24684, 2025, arxiv.org/...2510.24684.
13. ^Michael Dennis et al., "Emergent Complexity and Zero-Shot Transfer via Unsupervised Environment Design," arXiv:2010.03934, 2020, arxiv.org/...2010.03934.
14. ^Zhiyuan Zeng et al., "RLVE: Scaling Up Reinforcement Learning for Language Models with Verifiable Rewards," arXiv:2511.07317, 2025, arxiv.org/...2511.07317.
15. ^Rui Wang et al., "Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions," arXiv:1901.01753, 2019, arxiv.org/...1901.01753.
16. ^SPADE team, "SPADE-Qwen3-30B-A3B-Games model card," immutable revision e529c23985fadef779af79ed1b2de22044b1b9bc, Hugging Face, accessed Aug. 21, 2026, huggingface.co/...README.md.
17. ^SPADE team, "Generated-game smoke-test helper," pinned commit 65421ccb15a6d501ad6217bd969816146da15e11, Aug. 20, 2026, github.com/...game_files.py.