# RSI-Exam

> Source: https://aiwiki.ai/wiki/rsi_exam
> Updated: 2026-09-05
> Fact-checked: 2026-09-05
> Categories: AI Agents, AI Benchmarks
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "RSI-Exam." aiwiki.ai, 5 Sept 2026. https://aiwiki.ai/wiki/rsi_exam
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

RSI-Exam is a [benchmark](https://aiwiki.ai/wiki/benchmark) that tests whether an AI agent can take a working but weak executable method, improve it through hours of autonomous experimentation, and produce a version that still performs better on data it never saw. Its first release, RSI-Exam 0.1, contains 88 tasks drawn from six fields, of which 35 are public with their complete grading containers and 53 are held back.[1][2] The benchmark is run by the GitHub organization aiming-lab, whose profile describes it as "AIMING Lab @ UNC-Chapel Hill" and lists the X account of Huaxiu Yao, who made the repository's initial commit on August 26, 2026.[3][4] The website went live the same day, the public tasks and evaluation code followed on August 28, and a Hugging Face dataset was created on August 29, 2026.[2][5]

The benchmark's name refers to [recursive self-improvement](https://aiwiki.ai/wiki/recursive_self-improvement), but its authors draw the boundary narrowly. The accompanying blog post says RSI-Exam "studies improvement of that working artifact through sustained experimentation, not modification of the model's weights," and that it "does not evaluate models modifying their own weights, nor does it constitute unrestricted recursive self-improvement."[6] What it measures is whether an agent, given a fixed budget of up to 12 hours, can make an inherited method better in a way that transfers to a sealed hidden split.[2]

## Origin and organization

The GitHub repository `aiming-lab/RSI-Exam` was created on August 26, 2026 under the [MIT License](https://aiwiki.ai/wiki/mit_license), with a description that opens "RSI-Exam: Measuring Recursive Self-Improvement on Long-Horizon, Executable Research Tasks" (followed by a Discord invite link) and the topics `benchmark` and `recursive-self-improvement`.[3] The website, README, and Hugging Face card use a slightly different title, "Benchmarking Recursive Self-Improvement through Executable Research," and the site's citation block names the authors as "RSI-Exam Team" and points to the GitHub repository rather than a paper.[1][2][6] A search of the arXiv API for "RSI-Exam" on September 5, 2026 returned no results, so as of that date there is no preprint; the contribute page instead describes papers to accompany future 1.0 and 2.0 releases.[7]

The commit history shows two phases. Huaxiu Yao made the initial commit and two README updates on August 26, 2026, adding the project website link. A second contributor published the evaluation infrastructure on August 29, 2026, followed the same day by commits adding Discord and WeChat community links, a call for contributors, and a pointer to the new Hugging Face dataset location.[4] The aiming-lab organization profile on [GitHub](https://aiwiki.ai/wiki/github) expands the acronym as "Adaptive Intelligence through Alignment, Interaction and Learning" and gives its location as the United States.[8]

| Date | Event | Source |
| --- | --- | --- |
| August 26, 2026 | Repository created; initial commit by Huaxiu Yao; website rsi-exam.ai live with the 88-task leaderboard, per-domain task pages, the evaluation pipeline, three trajectory case studies, and the authoring guide | [2][3][4] |
| August 28, 2026 | Public task set and evaluation infrastructure announced as live ("Data and code live") | [2] |
| August 29, 2026 | Hugging Face dataset `RSI-Exam/RSI-Exam` created; evaluation infrastructure commit "Publish the evaluation infrastructure" and community links added to the repository | [4][5] |
| September 4, 2026 | Huaxiu Yao and Qwen Developers post about the Qwen3.8-Max-0902 result | [9][10] |
| September 15, 2026 | Contribution deadline for authorship on the paper accompanying RSI-Exam 1.0 (planned) | [7] |
| December 15, 2026 | Contribution deadline for authorship on the paper accompanying RSI-Exam 2.0 (planned) | [7] |

## What the benchmark measures

Each task targets one of two artifact types: "the method that solves the problem, or the harness that runs a frozen model."[1] The blog post gives examples of inherited starting points: "a query optimizer restricted to nested-loop joins, a gate-sizing method that assigns maximum drive strength to every instance, and a wind-retrieval method that solves only part of the governing system."[6] Starting from a functioning artifact rather than an empty repository is deliberate; the authors say it "requires the agent to diagnose and improve an existing approach."[6]

The blog post lists four design principles: long-horizon improvement (agents "spend hours proposing, measuring, retaining, and discarding changes"), hidden-set re-execution (the deliverable is executable code, and the verifier "runs it unchanged on the hidden set"), domain-native evaluation (tasks keep the metrics of their source fields, mapped onto a common scale), and inspectable research trajectories (saved versions, experiment descriptions, resource use, and hidden-set outcomes stay available).[6] The website summarizes the same idea as two axes, "Recursive improvement & Generalization," and warns that "topping the visible set is not the same as improving."[1]

## Evaluation protocol

A rollout begins with a single agent alone in a fresh container that holds the task environment and the inherited method.[6] Within the task's time and resource limits the agent modifies the artifact, evaluates candidates on the visible split, and saves numbered versions. At the end it submits one executable artifact; the verifier then replays that artifact from scratch on a sealed hidden split, and that single number is the score. The README states the consequence plainly: "An improvement that does not transfer earns nothing."[2]

The infrastructure runs on harbor, which its website describes as "A framework for evaluating and optimizing sandboxed agents and models."[11] The README says harbor "provides the sandbox, the agent/grader image split, and grading," while the RSI-Exam repository adds "the autoresearch protocol given to the agent, plus the network policy and per-harness overlays that seal the run."[2] Each task builds two separate images from its `environment/` and `tests/` directories; the agent works in the first, and "the only thing that ever crosses is the artifact it submits."[5] Inside the sealed verifier, "a trusted parent process loads the protected targets and keeps them outside the child environment before importing the submission," and network access and retraining are disabled.[6]

Two files are mounted into the agent's container. The first is the autoresearch instruction, a Jinja template that opens "You are a fully autonomous researcher. You have inherited a weak baseline in /app/methods/main/, and your mission is to make it strong." It tells the agent that "the score you can see is only a proxy," to decide early how it will judge generalization, to prefer the simpler of two equally good approaches, and to loop: run an experiment, log it to `experiment_log.md`, snapshot every version under `/app/methods/versions/`, and return to step one.[12] The second is `budget.py`, a reminder that rides along with each self-check measurement and, in the last 10% of the window, becomes a wrap-up notice.[2] The agent timeout is 43,200 seconds (12 hours) and the output-token limit is 500,000.[2]

The network policy is declared per task and per phase. The default is `public` during environment build (when harbor installs the agent CLI), `no-network` for the agent phase, and `no-network` for a separate verifier environment.[13] Model endpoints are never written into a task; they are supplied at run time with `--allow-agent-host`, so "no endpoint is ever baked into a task."[2] The blog post adds that during the leaderboard runs "Literature, external leaderboards, and package indexes were inaccessible," the Claude harness ran with WebSearch and WebFetch disabled, no run used a skill library or MCP server, and every model received an instruction rendered from the same template.[6] Four of the 35 public tasks call a model from inside the task itself, as a rubric judge or as the frozen base that the agent optimizes around: `locomo_longterm_memory`, `legal_matter_caseload_regulatory`, `lean_formal_proof_workflow_design`, and `discoveryworld_agent_harness_low2`.[2]

If an agent stops early because of a timeout or error, the verifier still grades whatever method it left in the container. The blog notes that "A score of 0 therefore means that the method did not improve on the inherited baseline; it does not necessarily indicate a runtime failure."[6] The README also cautions that tasks are stochastic, so "a single trial is a sample, not a measurement," and recommends repeated trials with `-k` when a number has to be defensible.[2]

## Scoring and normalization

Every task keeps its native metric and maps it onto a shared scale using measured anchors. The inherited starter is fixed at 0.00. When a mathematical, theoretical, or oracle upper bound exists it is fixed at 1.00. When the task author provides a strong frontier solution, the team verifies it with the same sealed verifier and uses its measured performance as an optional "Frontier-calibrated SOTA" anchor at 0.60.[6] Between finite anchors the mapping is linear or log-linear; when no finite upper bound exists, an exponential tail lets further gains earn credit while the score approaches but never reaches 1.00. Each task's anchors and mapping are frozen before comparison.[6] The contribute page stresses that "A number from a paper does not set" the 0.60 mark; "the method has to run on your data."[7]

The task pages show these anchors concretely. On Scientific Discovery Agent Harness Design, the starter's native metric of 0.134722 maps to 0.0, the frontier-calibrated reference of 0.571667 maps to 0.6, and 1 maps to 1.0.[14] On TPU v6e Masked GQA Kernel Optimization the starter speedup of 1.0x maps to 0.0 and an upper bound of 43.7608x maps to 1.0, with no frontier anchor listed.[15] The Hugging Face card notes that on the aggregate leaderboard "No agent reaches the reference."[5]

## Task construction and review

Each task passes four stages before entering the bank: problem selection, environment construction, score calibration, and end-to-end review.[6] Problem selection starts from published research problems with an executable baseline and a quantitative objective; a candidate must "leave meaningful headroom beyond the inherited method, support multiple rounds of substantive experimentation, and admit evaluation on private data," and candidates that the pilot agent already solves are excluded.[6] Environment construction packages the problem into the two isolated runtime environments. Score calibration measures the anchors. End-to-end review covers the problem definition, constraints, data split, leakage surface, reference methods, and verifier implementation, followed by a long-horizon pilot that checks four properties: headroom, difficulty ("progress requires substantive experiments rather than a local patch"), distribution consistency ("visible-set gains transfer to the hidden set"), and horizon.[6]

The website describes the human roles behind this pipeline. A domain-expert author proposes the problem and metric and runs the weak baseline and at least one stronger reference end to end. A developer turns it into the required format. A first reviewer applies "A standing rubric of 90+ checks: value, measurability, data provenance and licensing, and every leakage path that could be looked up or memorised." A second reviewer analyzes a full agent trajectory against the anchors and sends back a fix, an enhancement, or a filter. The site says authors "are PhD students, postdocs, faculty and industry practitioners, each writing in their own research area."[1]

## Task bank

RSI-Exam 0.1 holds 88 active tasks across six domains. The Hugging Face card gives the public and private split per domain.[5]

| Domain | What it covers (per the dataset card) | Public | Private | Total |
| --- | --- | --- | --- | --- |
| AI Models & Agents | Agent harnesses, LLM memory, post-training and sampler recipes, offline RL | 8 | 10 | 18 |
| Physical Sciences & Engineering | Inference and extrapolation from measurement data, control policies, physical design | 4 | 15 | 19 |
| Optimization, Planning & Control | Routing, scheduling, cutting and black-box search under a hard budget | 6 | 12 | 18 |
| Systems & Hardware | GPU and TPU kernels, EDA placement, vector search, QEC decoders | 9 | 4 | 13 |
| Life Sciences & Medicine | Single-cell genomics, molecular structure, clinical policy under shift | 5 | 7 | 12 |
| Finance, Law & Business | Valuation, alpha factors, long-horizon professional casework | 3 | 5 | 8 |
| Total | | 35 | 53 | 88 |

The website lists every task title by domain. Examples in AI Models & Agents include LLM Agent Memory Architecture Design, Small-Model Math Reasoning Post-Training, [Lean](https://aiwiki.ai/wiki/lean) 4 Proof-Search LLM agent Workflow Design, TriFinger Cube-Pushing Offline RL, Budget-Constrained 3D Gaussian Splat Compression, and Scientific Discovery Agent Harness Design.[1] Systems & Hardware holds kernel tasks such as TPU v6e Masked GQA Kernel Optimization, Ragged GQA Training Kernel Optimization, Multi-Stream Gated FlashFFTConv Kernel Optimization, and Mamba-3 Training Kernel Optimization, alongside three EDA tasks (standard-cell gate sizing, flip-flop banking and placement, and cell placement, sizing, and buffering), a Quantum Color-Code Decoder, a PostgreSQL-Compatible Server over SQLite, and 10M-Scale Metadata-Filtered ANN Search.[1] Physical Sciences & Engineering ranges from Climate Subgrid Convection Emulation and Exoplanet Transmission-Spectrum Atmospheric Retrieval to Time-Optimal Drone Racing and Wake-Aware Wind Farm Layout Optimization. Optimization, Planning & Control includes 2048 Search-Policy Optimization, Real-Time Battle Tetris Planning Agent, City-Scale Traffic Signal Control Optimization, and Online Railway Disruption Recovery and Rescheduling. Life Sciences & Medicine includes 12-Lead ECG Chagas Disease Screening, Cross-Site Chest X-Ray Triage Policy Optimization, and Cross-Dataset Single-Cell Label Transfer. Finance, Law & Business includes Cross-Sectional Stock Return Ranking, Equity DCF Value-Driver Forecasting, and Multi-Matter Legal Research and Analysis Agent.[1]

Each public task is a directory with `task.toml` (metric, timeouts, network policy, artifact declarations), `instruction.md` (the brief the agent receives), `environment/` (the agent's container with the starting method, visible data, and tools), `tests/` (the grading container with `grade.py`, anchors, and the sealed split), and, where one exists, `solution/` (the frontier-calibrated reference).[5] The Small-Model Math Reasoning Post-Training task illustrates the level of detail: the agent inherits a 1.7-billion-parameter student checkpoint (Qwen/Qwen3-1.7B-Base), a frozen 8-billion-parameter teacher, and an answer-only [LoRA](https://aiwiki.ai/wiki/lora) recipe, with a hard budget of 8 CPUs, two 48 GB GPUs, and 8 hours. It must submit a merged standalone student model, which the sealed verifier samples on disjoint integer-answer problems under a fixed protocol (temperature 0.6, top-p 0.95, top-k 20) and grades by exact integer accuracy without revealing problem text, answers, or per-example feedback.[16]

## Leaderboard as of September 5, 2026

The site publishes three boards: the full 88-task bank, the 35 public tasks, and the 53 private tasks. Each row names the model, the agent harness, and the reasoning-effort setting. The scores below are copied from the site as of September 5, 2026; they are mean hidden-set normalized scores from one rollout per model per task.[1][6] The blog post and README describe a nine-model panel; the site's boards as of September 5 carry a tenth row, Qwen3.8-Max-0902 run through Claude Code, which the blog's tables do not include.[1][2][6]

Full 88 tasks:[1]

| Rank | Model | Harness | Effort | Mean score |
| --- | --- | --- | --- | --- |
| 1 | [Opus 5](https://aiwiki.ai/wiki/claude_opus_5) | claude code | max | 0.464 |
| 2 | GPT-5.6-sol | codex | max | 0.433 |
| 3 | [GLM 5.3](https://aiwiki.ai/wiki/glm_5_3) | claude code | max | 0.403 |
| 4 | [Qwen3.8 Max-0902](https://aiwiki.ai/wiki/qwen3_8_max) | claude code | xhigh | 0.392 |
| 5 | [Kimi K3](https://aiwiki.ai/wiki/kimi_k3) | kimi cli | max | 0.382 |
| 6 | [Grok 4.6](https://aiwiki.ai/wiki/grok_4_6) | grok | xhigh | 0.367 |
| 7 | [GPT-5.5](https://aiwiki.ai/wiki/gpt-5.5) | codex | xhigh | 0.331 |
| 8 | [DeepSeek V4 Pro](https://aiwiki.ai/wiki/deepseek_v4_pro) | claude code | max | 0.322 |
| 9 | [Qwen3.8 Max](https://aiwiki.ai/wiki/qwen3_8_max) | qwen coder | xhigh | 0.322 |
| 10 | [Gemini 3.7 Flash](https://aiwiki.ai/wiki/gemini_3_7_flash) | antigravity | high | 0.309 |

Public 35 tasks:[1]

| Rank | Model | Harness | Effort | Mean score |
| --- | --- | --- | --- | --- |
| 1 | [Opus 5](https://aiwiki.ai/wiki/claude_opus_5) | claude code | max | 0.461 |
| 2 | GPT-5.6-sol | codex | max | 0.430 |
| 3 | [Qwen3.8 Max-0902](https://aiwiki.ai/wiki/qwen3_8_max) | claude code | xhigh | 0.418 |
| 4 | [GLM 5.3](https://aiwiki.ai/wiki/glm_5_3) | claude code | max | 0.378 |
| 5 | [Grok 4.6](https://aiwiki.ai/wiki/grok_4_6) | grok | xhigh | 0.367 |
| 6 | [Kimi K3](https://aiwiki.ai/wiki/kimi_k3) | kimi cli | max | 0.350 |
| 7 | [GPT-5.5](https://aiwiki.ai/wiki/gpt-5.5) | codex | xhigh | 0.322 |
| 8 | [Qwen3.8 Max](https://aiwiki.ai/wiki/qwen3_8_max) | qwen coder | xhigh | 0.304 |
| 9 | [DeepSeek V4 Pro](https://aiwiki.ai/wiki/deepseek_v4_pro) | claude code | max | 0.287 |
| 10 | [Gemini 3.7 Flash](https://aiwiki.ai/wiki/gemini_3_7_flash) | antigravity | high | 0.276 |

Private 53 tasks:[1]

| Rank | Model | Harness | Effort | Mean score |
| --- | --- | --- | --- | --- |
| 1 | [Opus 5](https://aiwiki.ai/wiki/claude_opus_5) | claude code | max | 0.466 |
| 2 | GPT-5.6-sol | codex | max | 0.435 |
| 3 | [GLM 5.3](https://aiwiki.ai/wiki/glm_5_3) | claude code | max | 0.420 |
| 4 | [Kimi K3](https://aiwiki.ai/wiki/kimi_k3) | kimi cli | max | 0.403 |
| 5 | [Qwen3.8 Max-0902](https://aiwiki.ai/wiki/qwen3_8_max) | claude code | xhigh | 0.375 |
| 6 | [Grok 4.6](https://aiwiki.ai/wiki/grok_4_6) | grok | xhigh | 0.367 |
| 7 | [DeepSeek V4 Pro](https://aiwiki.ai/wiki/deepseek_v4_pro) | claude code | max | 0.346 |
| 8 | [GPT-5.5](https://aiwiki.ai/wiki/gpt-5.5) | codex | xhigh | 0.337 |
| 9 | [Qwen3.8 Max](https://aiwiki.ai/wiki/qwen3_8_max) | qwen coder | xhigh | 0.334 |
| 10 | [Gemini 3.7 Flash](https://aiwiki.ai/wiki/gemini_3_7_flash) | antigravity | high | 0.331 |

The ordering shifts between splits. Opus 5 and GPT-5.6-sol hold the top two places on all three boards, but Qwen3.8 Max-0902 is third on the public split and fifth on the private one, while Kimi K3 is fourth on the private split and sixth on the public one.[1] Every score sits below the 0.60 frontier-calibrated reference; the blog post describes Opus 5 as having "the highest mean in this single-rollout panel."[6]

The README lists the harness commands used for the original panel. [Claude Code](https://aiwiki.ai/wiki/claude_code) was used for Opus 5, GLM 5.3 (through the z.ai Anthropic-compatible endpoint), and DeepSeek V4 Pro; [Codex](https://aiwiki.ai/wiki/openai_codex) for GPT-5.6-sol and GPT-5.5; qwen-coder for Qwen3.8 Max through an Alibaba endpoint; the [Antigravity](https://aiwiki.ai/wiki/antigravity) CLI for Gemini 3.7 Flash via browser OAuth; and grok-build for Grok 4.6 via subscription OAuth. The README says the commands "differ in only three places," namely allowed hosts, the web-tool kill switch, and the effort flag.[2] The blog's experimental-setup table lists Kimi K3's reasoning effort as "Not specified," while the leaderboard rows label it "max."[1][6]

The authors attach two caveats to every ranking. First, each result reflects "a model operating through a particular agent harness," so "leaderboard differences should not be attributed to the underlying model alone."[6] Second, each model has at most one graded rollout per task; the aggregate is expected to have low run-to-run variance because it averages 88 tasks, but any individual task score "remains a single observation." The blog says the next minor release will include repeated runs.[6]

## Qwen3.8-Max-0902 result

On September 4, 2026, Huaxiu Yao posted that Qwen3.8 Max-0902 "just made a major leap on the RSI-Exam 0.1 leaderboard," with its score jumping "from 0.322 to 0.392, a nearly 22% improvement."[9] Qwen Developers quoted the post the same day: "We did comprehensive training on coding and cowork for Qwen3.8-Max-0902, with complex, long-horizon tasks in mind. Glad to see that work generalizes to a 22% improvement on RSI-Exam. A small step toward RSI."[10]

The two numbers come from different rows of the same board, and the rows differ in more than the model snapshot. The 0.322 row is the original Qwen3.8 Max run through the qwen coder harness at xhigh effort; the 0.392 row is Qwen3.8 Max-0902 run through Claude Code at xhigh effort.[1] Because RSI-Exam's own authors say tool use, context management, and stopping behavior are part of the evaluated system, the comparison is between two model-harness pairs, not a like-for-like measurement of the model update.[6] The percentage is consistent with the two scores as published (0.392 is 21.7% above 0.322), but it bundles the harness change with the snapshot change.[1][9]

Per-task pages show how uneven the gain is. On TPU v6e Masked GQA Kernel Optimization, the 0902 run through Claude Code reached a hidden-set speedup of 8.3524x, normalized 0.688, the top score on that task, against 8.2178x (0.590) for the earlier Qwen3.8 Max run through qwen coder.[15] On Scientific Discovery Agent Harness Design the 0902 run scored 0.134 normalized, eighth of ten, against 0.010 for the earlier run.[14] On Small-Model Math Reasoning Post-Training the 0902 run scored 0.033; the best rows on that page (GPT-5.6-sol and GLM 5.3) scored 0.050, and five of the ten rows, including Opus 5, scored 0.[16]

## Domain, transfer, and resource analyses

Because native rewards cannot be compared across tasks, the blog post computes a within-task z-score for each run relative to the nine runs on the same task and averages it by domain. On that view, Opus 5 "has its largest relative advantages in Systems & Hardware, Optimization, and Finance, while GPT-5.6-sol has the highest domain mean in Physical Sciences." Life Sciences & Medicine "shows no consistent model-level advantage."[6]

For the 37 tasks whose visible-set and hidden-set outcomes share the same 0-1 scale, the post compares each model's mean visible score with its mean hidden score. Gemini 3.7 Flash had the largest visible-to-hidden decrease at 24.7%; Kimi K3 had the smallest at 11.7% and moved from sixth to fourth; the other decreases lay between 13% and 19%.[6]

The post also tabulates each model-harness pair's mean runtime, recorded spend, and output tokens over all 88 tasks, with the explicit note that "it is not an efficiency ranking."[6]

| Model | Harness | Mean score | Mean run time | Mean spend | Mean output tokens |
| --- | --- | --- | --- | --- | --- |
| Opus 5 | claude code | 0.464 | 5.6 h | $43 | 288k |
| GPT-5.6-sol | codex | 0.433 | 4.6 h | $53 | 234k |
| GLM 5.3 | claude code | 0.403 | 5.7 h | $18 | 273k |
| Kimi K3 | kimi cli | 0.382 | 4.8 h | $10 | 134k |
| Grok 4.6 | grok | 0.367 | 1.9 h | $14 | 252k |
| GPT-5.5 | codex | 0.331 | 1.7 h | $13 | 70k |
| DeepSeek V4 Pro | claude code | 0.322 | 4.7 h | $7 | 508k |
| Qwen3.8 Max | qwen coder | 0.322 | 3.4 h | $20 | 229k |
| Gemini 3.7 Flash | antigravity | 0.309 | 2.7 h | $9 | 395k |

The three highest-scoring systems all ran for at least 4.6 hours on average, and Opus 5 and GPT-5.6-sol had the two largest mean spends. Output volume followed a different pattern: DeepSeek V4 Pro and Gemini 3.7 Flash produced the most output tokens yet ranked in the bottom three, while GPT-5.5 produced the fewest and scored above both. The post concludes that runtime, spend, and tokens "capture different properties of a model-harness pair, including iteration style, provider pricing, stopping behaviour, and task difficulty," and that with one rollout per pair the summaries "should not be read as estimates of the causal return to extra budget."[6] The Qwen3.8 Max-0902 row is not in this table, which predates it.[6]

## Trajectory case studies

The blog post walks through three saved trajectories. In a GPT-5.6-sol rollout on Scientific Discovery Agent Harness Design, the post reports that the visible suite mean rose from 0.10 to 0.63 over a 10.5-hour run costing $160, and that the hidden-set rerun scored 0.256 (normalized 0.45). It describes four phases: replacing the inherited stateless ReAct loop with a stateful harness, adding a second deterministic controller, keeping the simpler of two tied changes, and adding evidence-collection improvements worth "a few thousandths each." By the second phase, the post says, "neither scenario consults the language model at all."[6]

The second case contrasts two GPT-5.6-sol runs. On Small-Model Math Reasoning Post-Training, the post says the model produced 41 versions and a hidden-set score of 0.225, spending almost the whole run on supervised fine-tuning while the reference method used on-policy [distillation](https://aiwiki.ai/wiki/knowledge_distillation); a post-training expert who read the log "found no fault with its rigour" but noted that "The discipline simply served a single idea."[6] On TPU v6e Masked GQA Kernel Optimization, the post says the same model produced 145 versions and a hidden-set score of 0.708: the first 23 versions tuned the official kernel to about 7.2x, and the remaining 120 built a new Pallas kernel reaching 9.76x on the visible set, beating the reference approach through head packing, a scratchless schedule, and a causal prefix decomposition.[6]

The live task pages as of September 5, 2026 show different GPT-5.6-sol rollout statistics for these three tasks than the case studies describe: 14 versions, 235 minutes, $68.75, and normalized 0.2697 on the discovery-harness task; 10 versions, 434 minutes, and normalized 0.05 on the math task; and 10 versions, 331 minutes, 8.9389x, and normalized 0.6355 on the TPU task.[14][15][16] The site does not state whether the case-study trajectories and the leaderboard rollouts are the same runs. The blog does say the case studies "illustrate contrasting search processes; they are not estimates of how frequently either pattern occurs."[6]

## Contribution program

The contribute page describes four roles: task author (an estimated 18 hours to write a task: "Your problem, a weak starter that runs, and a stronger method you can run yourself. We do the packaging."), reviewer (about 4 hours per task), auditor (about 2 hours to read one trajectory end to end), and advisor, which the page calls "the core-contributor track."[7] Contributions are recorded as credits: 15 for an accepted task written by the contributor, 10 for a task the team implements from a contributor's problem and references, up to 10 for a task that fails a gate after real work, 4 for a review with a written verdict, 2 for a rollout audit, 2 to 8 for an adopted pipeline or evaluation change, and 0 for a referral (acknowledged only). Fifteen credits qualify a contributor for paper authorship; authors outside the core group are listed alphabetically.[7]

The page states that the bank wants both CPU-only and GPU tasks, that agent rollouts can run on the project's compute, that a task a frontier agent simply solves "does not ship as it stands" (often fixed with a harder hidden split), and that the project will not ask for unpublished or embargoed data or exclusivity.[7] Task authors receive frontier-model results on their problem with full trajectories. The README directs groups that want to be evaluated on the full 88-task suite to contact@rsi-exam.ai, and community discussion runs on Discord and a WeChat group.[2]

## Relation to other benchmarks

RSI-Exam belongs to a group of benchmarks that evaluate agents on research-style work rather than single answers. [MLE-bench](https://aiwiki.ai/wiki/mle_bench) (October 2024) curates 75 machine learning engineering competitions from Kaggle and uses Kaggle's public leaderboards as human baselines.[17] [RE-Bench](https://aiwiki.ai/wiki/re_bench) (November 2024) consists of seven open-ended ML research engineering environments with data from 71 eight-hour attempts by 61 human experts.[18] [PaperBench](https://aiwiki.ai/wiki/paperbench) (April 2025) asks agents to replicate 20 ICML 2024 papers from scratch, graded against 8,316 rubric items co-developed with the papers' authors.[19] Closer in framing is [Frontis-MA1](https://aiwiki.ai/wiki/frontis_ma1) (July 2026), whose paper describes machine learning engineering as "a concrete, executable testbed" for recursive self-improvement and evaluates its model on MLE-Bench Lite under a 12-hour per-task budget.[20]

RSI-Exam differs from these in three stated ways. It spans fields well outside machine learning engineering, with only 18 of 88 tasks in the AI Models & Agents domain.[5] It requires an executable artifact that the verifier re-runs on a sealed split, rather than a predictions file or a graded write-up.[6] And it normalizes every task against measured anchors so that a frontier-calibrated reference sits at a common 0.60.[6] The blog frames the result as turning "recursive self-improvement from an abstract capability claim into an executable, reproducible research loop," while limiting the claim to "improvement within a bounded, executable research setting."[6]

## Limitations stated by the authors

The blog post's limitations section lists four items. Evaluation scope: the benchmark measures method improvement in bounded environments and does not evaluate weight modification. Model-harness pairs: results belong to the pair, not the model alone. Starting conditions: every rollout begins from a task-specific inherited artifact that, under a fixed budget, "can still shape the search path and which alternatives are reached." Single rollouts: task-level scores are single observations, and repeated runs are planned for the next minor release.[6] The README adds a practical warning for one harness: the Antigravity CLI needs a patch after every harbor upgrade, "without it any task longer than five minutes is silently cut short and still graded."[2]

## See also

- [Recursive self-improvement](https://aiwiki.ai/wiki/recursive_self-improvement)
- [AI agents](https://aiwiki.ai/wiki/ai_agents)
- [Harness (AI)](https://aiwiki.ai/wiki/harness)
- [MLE-bench](https://aiwiki.ai/wiki/mle_bench)
- [PaperBench](https://aiwiki.ai/wiki/paperbench)
- [RE-Bench](https://aiwiki.ai/wiki/re_bench)
- [Qwen3.8-Max](https://aiwiki.ai/wiki/qwen3_8_max)
- [Benchmark (AI)](https://aiwiki.ai/wiki/benchmark)

## References

1. [RSI-Exam: Benchmarking Recursive Self-Improvement through Executable Research (home page, leaderboard, and task bank)](https://rsi-exam.ai/) - rsi-exam.ai (RSI-Exam Team), accessed September 5, 2026.
2. [aiming-lab/RSI-Exam README](https://github.com/aiming-lab/RSI-Exam) - GitHub (aiming-lab), last updated August 29, 2026; accessed September 5, 2026.
3. [Repository metadata for aiming-lab/RSI-Exam](https://api.github.com/repos/aiming-lab/RSI-Exam) - GitHub REST API, accessed September 5, 2026.
4. [Commit history of aiming-lab/RSI-Exam](https://github.com/aiming-lab/RSI-Exam/commits/main) - GitHub, August 26 to 29, 2026; accessed September 5, 2026.
5. [RSI-Exam Public Task Set (dataset card)](https://huggingface.co/datasets/RSI-Exam/RSI-Exam) - Hugging Face (RSI-Exam), created August 29, 2026; accessed September 5, 2026.
6. [RSI-Exam: Benchmarking Recursive Self-Improvement through Executable Research (blog post)](https://rsi-exam.ai/blog.html) - rsi-exam.ai (RSI-Exam Team), 2026; accessed September 5, 2026.
7. [Call for contributors](https://rsi-exam.ai/contribute.html) - rsi-exam.ai (RSI-Exam Team), accessed September 5, 2026.
8. [AIMING Lab organization profile](https://github.com/aiming-lab) - GitHub, accessed September 5, 2026.
9. [Post on Qwen3.8 Max-0902's RSI-Exam 0.1 score](https://x.com/HuaxiuYaoML/status/2095861167939805252) - X (Huaxiu Yao, @HuaxiuYaoML), September 4, 2026.
10. [Post on Qwen3.8-Max-0902 training and its RSI-Exam improvement](https://x.com/QwenDevs/status/2095888382064988439) - X (Qwen Developers, @QwenDevs), September 4, 2026.
11. [Harbor](https://www.harborframework.com/) - harborframework.com, accessed September 5, 2026.
12. [infra/prompts/autoresearch.j2](https://github.com/aiming-lab/RSI-Exam/blob/main/infra/prompts/autoresearch.j2) - GitHub (aiming-lab), accessed September 5, 2026.
13. [docs/network-policy.md](https://github.com/aiming-lab/RSI-Exam/blob/main/docs/network-policy.md) - GitHub (aiming-lab), accessed September 5, 2026.
14. [Scientific Discovery Agent Harness Design (task page)](https://rsi-exam.ai/tasks/discoveryworld_agent_harness_low2.html) - rsi-exam.ai, accessed September 5, 2026.
15. [TPU v6e Masked GQA Kernel Optimization (task page)](https://rsi-exam.ai/tasks/splash_attention_strata_speedup.html) - rsi-exam.ai, accessed September 5, 2026.
16. [Small-Model Math Reasoning Post-Training (task page)](https://rsi-exam.ai/tasks/teacher_student_math_posttraining.html) - rsi-exam.ai, accessed September 5, 2026.
17. [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://arxiv.org/abs/2410.07095) - arXiv (Jun Shern Chan et al.), October 9, 2024.
18. [RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts](https://arxiv.org/abs/2411.15114) - arXiv (Hjalmar Wijk et al.), November 22, 2024.
19. [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://arxiv.org/abs/2504.01848) - arXiv (Giulio Starace et al.), April 2, 2025.
20. [Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering](https://arxiv.org/abs/2607.28568) - arXiv (Junlin Yang et al.), July 30, 2026.