# WeirdML

> Source: https://aiwiki.ai/wiki/weirdml
> Summary: WeirdML is a benchmark for evaluating how well large language models can do hands on machine learning engineering. It was created by Norwegian researcher Håvard Tveit Ihle of the Norwegian Defence Research Establishment (FFI) and introduced on LessWrong on January 16, 2025.
> Updated: 2026-08-07
> Fact-checked: 2026-08-07
> Categories: AI Benchmarks, Developer Tools
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "WeirdML." aiwiki.ai, 7 Aug 2026. https://aiwiki.ai/wiki/weirdml
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

| WeirdML |  |
| --- | --- |
| Overview |  |
| Full name | WeirdML (Weird Machine Learning) |
| Description | Benchmark testing whether [large language models](https://aiwiki.ai/wiki/large_language_model) can do real ML engineering on small, unusual datasets by writing PyTorch code, getting feedback, and iterating |
| First released | January 16, 2025 (v1) |
| Latest version | WeirdML v2 (June 2025) |
| Author | Håvard Tveit Ihle |
| Affiliation | Norwegian Defence Research Establishment (FFI) |
| Hosting and support | [Epoch AI](https://aiwiki.ai/wiki/epoch-ai) Benchmarking Hub; [METR](https://aiwiki.ai/wiki/metr) sponsored API costs |
| Technical Details |  |
| Type | ML engineering, [code generation](https://aiwiki.ai/wiki/code_generation), agentic iteration |
| Tasks (v1) | 6 |
| Tasks (v2) | 19 (6 public, 13 hidden) |
| Iterations per run | 5 submissions, 4 rounds of feedback |
| Hardware | NVIDIA TITAN V GPU, 12 GB memory |
| Per submission timeout | 600 seconds (v1); 120 seconds (v2) |
| Runs per model and task | At least 15 in v1 (5 for o1 preview); at least 5 in v2 (2 for the most expensive models) |
| Scoring | Mean across runs of the best test accuracy across the 5 iterations |
| Language | Python (PyTorch) |
| Performance |  |
| Top score (v1, Jan 2025) | About 51% (Claude 3.5 Sonnet) |
| Top score (v2 public leaderboard) | 91.9% (Claude Fable 5, max setting, as of August 8, 2026) |
| Saturated | No |
| Resources |  |
| Website | [htihle.github.io/weirdml.html](https://htihle.github.io/weirdml.html) |
| v1 archive | [htihle.github.io/weirdml_v1.html](https://htihle.github.io/weirdml_v1.html) |
| Introductory post | [LessWrong, Jan 16, 2025](https://www.lesswrong.com/posts/LfQCzph7rc2vxpweS/introducing-the-weirdml-benchmark) |
| Time horizons code | [github.com/htihle/weirdml-time-horizons](https://github.com/htihle/weirdml-time-horizons) |
| Epoch AI listing | [epoch.ai/benchmarks/weirdml](https://epoch.ai/benchmarks/weirdml) |
| License | MIT (time horizons repository) |

**WeirdML** is a [benchmark](https://aiwiki.ai/wiki/benchmark) for evaluating how well [large language models](https://aiwiki.ai/wiki/large_language_model) can do hands on [machine learning](https://aiwiki.ai/wiki/machine_learning) engineering. It was created by Norwegian researcher Håvard Tveit Ihle of the Norwegian Defence Research Establishment (FFI) and introduced on [LessWrong](https://aiwiki.ai/wiki/lesswrong) on January 16, 2025. Each task is a small, deliberately quirky ML problem with limited training data or an unusual input representation. The model has to read the prompt, design an approach, write a complete PyTorch script that loads the data, trains, and evaluates, then iterate on its solution after seeing terminal output and a held out test accuracy.[1][2]

WeirdML v1 covered six tasks. WeirdML v2, announced in June 2025, kept the six public tasks and added 13 hidden tasks for 19 total, plus tracking of API cost, output tokens, and lines of code. WeirdML v2 has been integrated into [Epoch AI](https://aiwiki.ai/wiki/epoch-ai)'s Benchmarking Hub, with [METR](https://aiwiki.ai/wiki/metr) helping fund the API spend.[3][4][5]

## Origins and motivation

Ihle is a former astrophysicist who worked on cosmological data pipelines for the [COMAP](https://aiwiki.ai/wiki/comap) and [Cosmoglobe](https://aiwiki.ai/wiki/cosmoglobe) experiments before shifting toward AI evaluation, generalization, and robustness. In the LessWrong post he framed WeirdML as a response to a gap: standard ML benchmarks fix a dataset and reward leaderboard climbing, while code benchmarks cover short functions or competitive puzzles. Neither tells you whether a model can sit with a small unfamiliar dataset and figure out what to do.[1][6]

WeirdML targets four capabilities at once: understanding the data and structure of the problem, picking a sensible architecture and training setup, producing PyTorch code that runs, and using feedback to fix bugs. Ihle built the automated pipeline as a part time project over about two months. The original v1 evaluation cost roughly $200 in API calls, dominated by o1 preview at about two dollars per run.[1] The word "weird" is meant literally: some tasks format their data in ways that defeat the most obvious approach (images as unordered patches, shapes as point clouds) to push models past pattern matching against tutorials seen in pretraining.

## How a WeirdML task works

Each task is presented as a self contained prompt with the problem description, training and test data paths, and a small example of how to load the data with [NumPy](https://aiwiki.ai/wiki/numpy) or [PyTorch](https://aiwiki.ai/wiki/pytorch). The model returns a Python script that handles everything from data loading through final evaluation. The code runs inside an isolated [Docker](https://aiwiki.ai/wiki/docker) container on a single NVIDIA TITAN V GPU with 12 GB of memory and a strict timeout (600 seconds in v1; the current v2 page lists 120 seconds). Network access is disabled, so the model cannot pull pretrained weights at run time.[1][2][3]

A single run gives the model five submissions. After each attempt, the [harness](https://aiwiki.ai/wiki/harness) returns terminal output (errors and test accuracy if the run completed) and asks for a revision. The accuracy reported for the run is the best across the five submissions. In v1, each model got at least 15 runs per task, with only o1 preview cut to five; v2 lowers this to at least five runs per task, with the most expensive models (o3 pro, Claude 4 Opus, GPT-4.5) getting two.[2][3] The headline score is the mean of best per run accuracies. A full evaluation can cost thousands of dollars, which is why METR's funding mattered for v2.[1][4]

## Tasks in WeirdML

WeirdML v2 has 19 tasks; only six are public. The other 13 are held out so the benchmark stays informative as public solutions accumulate. The six public tasks shipped with v1 in January 2025.[2][3]

| Task | Setup | What makes it tricky |
| --- | --- | --- |
| Shapes (Easy) | Classify five shapes (circle, square, triangle, pentagon, star) from 512 noisy 2D coordinates. Centered, fixed orientation and size. 1000 training examples. | Inputs are unordered point clouds, not images; the model must handle permutation invariance and noise. |
| Shapes (Hard) | Same as Easy but with random translation, rotation, and scaling per sample. | Adds invariant features or aggressive data augmentation. |
| Image Patch Shuffling (Easy) | Reconstruct 27x27 grayscale Fashion MNIST images from nine shuffled 9x9 patches. | A jigsaw problem rather than classification. |
| Image Patch Shuffling (Hard) | Reconstruct from RGB patches randomly sampled from larger Imagenette images with varying backgrounds. | Performance hovers near chance for almost every model. |
| Chess Game Outcome | Predict win, loss, or draw from algebraic notation move sequences. 1000 amateur games. | Sequence input with no pretrained chess knowledge available at run time. |
| Unsupervised Digit Recognition | Classify digits with only 26 labeled and around 16,000 unlabeled samples. | A semi supervised pipeline must be built end to end. |

The 13 hidden v2 tasks broaden the suite to cover more imaging problems, more sequential and tabular data, additional [unsupervised](https://aiwiki.ai/wiki/unsupervised_machine_learning) setups, and tasks designed to span a wider difficulty range so that the leaderboard does not collapse around a small number of saturated entries.[3][4]

## Methodology details

| Choice | Why it matters |
| --- | --- |
| Best of five within a run | Rewards getting a working solution at any iteration |
| Mean over many runs (15+ in v1, 5+ in v2) | Reduces the high run to run variance typical in code generation benchmarks |
| Strict resource limits | Forces models to engineer a solution that fits, not brute force a giant net |
| Test set isolation | Prevents peeking even if the model's file handling is sloppy |
| No internet during execution | Blocks downloads of pretrained weights mid run |

Ihle reported that on v1, the gap between five independent tries and five iterations with feedback was smaller than expected for non reasoning models. Most of the value of iteration came from more shots on goal, with extra benefit from feedback concentrated in [reasoning models](https://aiwiki.ai/wiki/reasoning_models) such as o1 mini, o1 preview, and gemini-2.0-flash-thinking. Newer reasoning models have widened that gap.[1][2]

## WeirdML v1 results

When Ihle published the v1 leaderboard in January 2025, [Claude 3.5 Sonnet](https://aiwiki.ai/wiki/claude-3.5-sonnet) led at about 51% mean accuracy across the six tasks, with OpenAI's o1 preview close behind.[1][2]

| Model | Average across 6 tasks |
| --- | --- |
| Claude 3.5 Sonnet | 50.94% |
| o1 preview | 48.82% |
| o1 mini | 45.58% |
| Claude 3.5 Haiku | 43.75% |
| Gemini 2.0 Flash Thinking | 42.82% |

Shapes (Easy) was effectively solved (o1 preview reached about 98%). Shapes (Hard) topped out near 60% on [Claude 3.5 Sonnet](https://aiwiki.ai/wiki/claude_3_5_sonnet). Chess outcome prediction stalled around 74% for the same model. Image Patch Shuffling (Hard) was unsolved, with most models near chance. Unsupervised Digit Recognition had a high first attempt failure rate but [Claude 3](https://aiwiki.ai/wiki/claude_3).5 Sonnet averaged around 80% when its pipeline worked.[1][2]

## WeirdML v2 results

Ihle announced v2 in June 2025 alongside the Epoch AI integration. The v2 leaderboard uses 17 of the 19 tasks for the public score and reports much higher numbers than v1, partly because the model lineup improved and partly because v2 averages across a wider task set where some are easier on average.[3][4][7]

The board scores each model configuration separately; the parenthetical labels are reasoning effort or thinking settings, running from no thinking up to xhigh and max. The top of the leaderboard as of August 8, 2026, taking the best configuration per model:[3][13]

| Rank | Model (setting) | WeirdML v2 score | Release |
| --- | --- | --- | --- |
| 1 | [Claude Fable 5](https://aiwiki.ai/wiki/claude_fable_5) (max) | 91.9% | Jun 2026 |
| 2 | [Claude Opus 5](https://aiwiki.ai/wiki/claude_opus_5) (max) | 91.8% | Jul 2026 |
| 3 | GPT-5.6 Sol Pro (max) | 89.4% | Jul 2026 |
| 4 | [GPT-5.6 Sol](https://aiwiki.ai/wiki/gpt_5_6) (high) | 88.8% | Jul 2026 |
| 5 | GPT-5.5 (xhigh) | 84.9% | Apr 2026 |
| 6 | [Claude Opus 4.8](https://aiwiki.ai/wiki/claude_opus_4_8) (xhigh) | 82.9% | May 2026 |
| 7 | [Kimi K3](https://aiwiki.ai/wiki/kimi_k3) (max) | 82.6% | Jul 2026 |
| 8 | GPT-5.6 Terra (high) | 78.3% | Jul 2026 |
| 9 | [Claude Opus 4.6](https://aiwiki.ai/wiki/claude_opus_4_6) (high) | 77.9% | Feb 2026 |
| 10 | GPT-5.3 Codex (xhigh) | 77.9% | Feb 2026 |

Ihle has posted snapshots tied to specific releases. When [GPT-5](https://aiwiki.ai/wiki/gpt-5) launched, he reported it leading at 56.3% (beating o3 pro at 53.9%) with gpt-5 mini matching o3 at a fraction of the cost. GPT-5 wrote much more code per attempt (median 324 lines vs about 133 for o3).[7] GPT-5.2 (xhigh) reached 72.2% after its December 2025 release, and 2026 models pushed far past that, as the table above shows. The reasoning setting matters a great deal: GPT-5.5 scores 67.1% with thinking disabled and 84.9% at xhigh.[8][13] Small open weight baselines sit far lower, with Mixtral 8x22B Instruct around 3.2%, and the board covers 152 model configurations as of August 8, 2026.[13]

Per task patterns shifted. Shapes (Hard) is no longer near chance for top models, with the strongest reasoning models reaching about 90%. Image Patch Shuffling (Hard) is still the toughest public task, though leaders have crept above chance. Most of the headroom now lives in the hidden v2 tasks.[3][4]

Ihle also posts results for notable new releases. When Meta's [Muse Spark](https://aiwiki.ai/wiki/muse_spark) 1.2 came out on August 5, 2026, he reported a score of 60.3% at the xhigh reasoning effort setting, tied with GPT-5.4 Mini (high) and just behind GPT-5 (high) at 60.7%, and called the model "pretty good, but far from the frontier". That places it around 40th of the 152 configurations on the board, about 30 points behind the leaders.[13][14]

## Agentic WeirdML

Ihle has also published Agentic WeirdML, a companion evaluation in which the model, instead of submitting code blind for single scored executions, drives a coding agent that can explore the data and iterate inside a Docker sandbox (no internet access, a GPU, and a 20 million token budget per task) before spending its five scored submissions. GPT-5.5 run through the [Codex CLI](https://aiwiki.ai/wiki/codex_cli) harness averaged 88.3% and [Claude Opus 4.7](https://aiwiki.ai/wiki/claude_opus_4_7) through the [Claude Code](https://aiwiki.ai/wiki/claude_code) harness averaged 87.9% over two full 17 task runs, graded exactly like the regular benchmark. The scaffold lifts each model well above its best non agentic configuration (84.9% and 76.4% respectively), mostly not by setting new per task records but by landing near the best known score on almost every task; the average of the best score any model has achieved on each task is 92.3%. Agentic runs cost more per task, about $5.30 versus $2.60 for GPT-5.5 and $9.00 versus $1.80 for Opus 4.7.[15]

## WeirdML time horizons

In February 2026 Ihle published a follow up analysis called "WeirdML Time Horizons" on LessWrong, with code in the [weirdml time horizons](https://github.com/htihle/weirdml-time-horizons) GitHub repository (MIT license). The idea borrows from [METR](https://aiwiki.ai/wiki/metr)'s task duration framing: estimate how long a median professional ML researcher would need to solve each task without AI help, then ask at what task length each model crosses 50% success.[9][10]

Ihle uses a panel of four LLMs to estimate per task human completion times at five accuracy thresholds (25%, 50%, 70%, 90%, 95%). Estimates become hours (1 day = 8h, 1 week = 40h) and feed a logistic fit, with block bootstrap resampling for uncertainty. The headline result: WeirdML time horizons roughly double every five months, from about 24 minutes for [GPT-4](https://aiwiki.ai/wiki/gpt-4) in June 2023 to roughly 38 hours for Claude Opus 4.6 in February 2026. That doubling rate is close to METR's reported seven month doubling, despite different tasks and methodology.[9][10]

| Model | Release | Time horizon (50% success) |
| --- | --- | --- |
| Claude Opus 4.6 (adaptive) | Feb 2026 | About 37.7 hours |
| GPT-5.2 (xhigh) | Dec 2025 | About 30.6 hours |
| Gemini 3 Pro (high) | Nov 2025 | About 22.3 hours |
| GPT-5 (high) | Aug 2025 | About 14.5 hours |
| o3 pro (high) | Jun 2025 | About 11.8 hours |
| o1 preview | Sep 2024 | About 6.2 hours |
| Claude 3.5 Sonnet | Jun 2024 | About 1.9 hours |
| GPT-4 | Jun 2023 | About 24 minutes |

The LLM panel likely overestimates absolute completion times, especially at high accuracy thresholds, so the hours should be read with skepticism. The doubling rate is more robust. A calibrated variant in the repository gives smaller absolute values but a similar doubling time of about six months.[9]

## Comparison with other benchmarks

| Benchmark | How it differs from WeirdML |
| --- | --- |
| [HumanEval](https://aiwiki.ai/wiki/humaneval) and [MBPP](https://aiwiki.ai/wiki/mbpp) | No data, training, or iteration; pure short function generation |
| [SWE-bench](https://aiwiki.ai/wiki/swe-bench) | Software engineering on existing repos, not ML modeling from scratch |
| [MLE-bench](https://aiwiki.ai/wiki/mle-bench) | Kaggle style ML competitions with larger datasets and longer budgets |
| MLAgentBench | ML research style tasks; broader scope, less focus on small weird datasets |
| [GPQA](https://aiwiki.ai/wiki/gpqa) | Graduate science multiple choice; no code execution |
| [SciCode](https://aiwiki.ai/wiki/scicode) | Scientific computing problems decomposed into subproblems |
| RE-Bench | Open ended research engineering tasks judged by experts |

WeirdML is distinguished by three features: tasks are deliberately small and quirky, the harness automates a five iteration loop with execution feedback, and strict GPU and time limits force practical ML thinking.[1][3][5]

## Reception and use

[Epoch AI](https://aiwiki.ai/wiki/epoch-ai)'s benchmarking hub added WeirdML v2 alongside Aider Polyglot, Balrog, and the Factorio Learning Environment when it expanded to feature trusted external leaderboards. WeirdML scores feed into the Epoch Capabilities Index, an aggregate measure across many benchmarks.[5][11] Independent aggregators such as NeoSignal cite WeirdML alongside SWE-bench Verified and GPQA. Frontier releases since mid 2025 have included WeirdML scores in third party comparisons, especially for [GPT-5](https://aiwiki.ai/wiki/gpt-5), [Claude Opus 4](https://aiwiki.ai/wiki/claude-opus-4) variants, [Gemini](https://aiwiki.ai/wiki/gemini) 3 Pro, and Grok 4.[7][8] Ihle has framed WeirdML's role as keeping a meaningful signal alive while standard benchmarks saturate.[1][12]

## Limitations

| Limitation | Description |
| --- | --- |
| Fixed framework | All solutions are PyTorch; JAX, TensorFlow, and Julia are not measured |
| Small task count | 19 tasks in v2 (17 in the public score); per task noise is non trivial |
| Hardware specific | TITAN V GPU and 12 GB memory are unusual versus modern production hardware |
| Short attempt window | The execution timeout (600 seconds in v1, 120 in v2) rules out longer training runs |
| Hidden task drift | Hidden v2 tasks cut overfitting risk but make per task interpretation harder |
| Cost of evaluation | Full evaluation of reasoning models can cost thousands of dollars |

## See also

- [Benchmark](https://aiwiki.ai/wiki/benchmark)
- [Code generation](https://aiwiki.ai/wiki/code_generation)
- [LLM evaluation](https://aiwiki.ai/wiki/llm_evaluation)
- [Reasoning models](https://aiwiki.ai/wiki/reasoning_models)
- [SWE-bench](https://aiwiki.ai/wiki/swe-bench)
- [MLE-bench](https://aiwiki.ai/wiki/mle-bench)
- [SciCode](https://aiwiki.ai/wiki/scicode)
- [METR](https://aiwiki.ai/wiki/metr)
- [Epoch AI](https://aiwiki.ai/wiki/epoch-ai)

## References

1. Ihle, Håvard Tveit. "Introducing the WeirdML Benchmark." LessWrong, January 16, 2025. https://www.lesswrong.com/posts/LfQCzph7rc2vxpweS/introducing-the-weirdml-benchmark
2. Ihle, Håvard Tveit. "WeirdML v1." Personal website. https://htihle.github.io/weirdml_v1.html
3. Ihle, Håvard Tveit. "WeirdML." Personal website (current v2 page). https://htihle.github.io/weirdml.html
4. Ihle, Håvard Tveit. "WeirdML v2 is now out." Twitter post, June 27, 2025. https://x.com/htihle/status/1938603525702930849
5. Epoch AI. "WeirdML (v2) benchmark page." https://epoch.ai/benchmarks/weirdml
6. Ihle, Håvard Tveit. Personal homepage. https://htihle.github.io/
7. NeoSignal. "WeirdML benchmark scores." https://neosignal.io/benchmarks/weirdml
8. Mowshowitz, Zvi. "GPT-5.5: Capabilities and Reactions." Don't Worry About the Vase, 2026. https://www.lesswrong.com/posts/5ytcFayxqZsXN8rNw/gpt-5-5-capabilities-and-reactions
9. Ihle, Håvard Tveit. "WeirdML Time Horizons." LessWrong, February 16, 2026. https://www.lesswrong.com/posts/hoQd3rE7WEaduBmMT/weirdml-time-horizons
10. Ihle, Håvard Tveit. "weirdml-time-horizons." GitHub repository. https://github.com/htihle/weirdml-time-horizons
11. Epoch AI. "We've added four new benchmarks to the Epoch AI Benchmarking Hub." Twitter post, May 2025. https://x.com/EpochAIResearch/status/1919831883875062184
12. Ihle, Håvard Tveit. LessWrong shortform comments. https://www.greaterwrong.com/posts/Mrcsc7bEKjSbjq2op/havard-tveit-ihle-s-shortform
13. Ihle, Håvard Tveit. "WeirdML leaderboard data (CSV)." Personal website, accessed August 8, 2026. https://htihle.github.io/data/weirdml_data.csv
14. Ihle, Håvard Tveit. "Muse Spark 1.2 (xhigh) scores 60.3% on WeirdML." Twitter post, August 7, 2026. https://x.com/htihle/status/2085680447330058625
15. Ihle, Håvard Tveit. "Agentic WeirdML." Personal website. https://htihle.github.io/agentic_weirdml.html