WeirdML

Scroll sideways for more →

WeirdML is a benchmark for evaluating how well large language models can do hands on machine learning engineering. It was created by Norwegian researcher Håvard Tveit Ihle of the Norwegian Defence Research Establishment (FFI) and introduced on LessWrong on January 16, 2025. Each task is a small, deliberately quirky ML problem with limited training data or an unusual input representation. The model has to read the prompt, design an approach, write a complete PyTorch script that loads the data, trains, and evaluates, then iterate on its solution after seeing terminal output and a held out test accuracy.^[1][2]

WeirdML v1 covered six tasks. WeirdML v2, announced in June 2025, kept the six public tasks and added 13 hidden tasks for 19 total, plus tracking of API cost, output tokens, and lines of code. WeirdML v2 has been integrated into Epoch AI's Benchmarking Hub, with METR helping fund the API spend.^[3][4][5]

Origins and motivation

Ihle is a former astrophysicist who worked on cosmological data pipelines for the COMAP and Cosmoglobe experiments before shifting toward AI evaluation, generalization, and robustness. In the LessWrong post he framed WeirdML as a response to a gap: standard ML benchmarks fix a dataset and reward leaderboard climbing, while code benchmarks cover short functions or competitive puzzles. Neither tells you whether a model can sit with a small unfamiliar dataset and figure out what to do.^[1][6]

WeirdML targets four capabilities at once: understanding the data and structure of the problem, picking a sensible architecture and training setup, producing PyTorch code that runs, and using feedback to fix bugs. Ihle built the automated pipeline as a part time project over about two months. The original v1 evaluation cost roughly $200 in API calls, dominated by o1 preview at about two dollars per run.^[1] The word "weird" is meant literally: some tasks format their data in ways that defeat the most obvious approach (images as unordered patches, shapes as point clouds) to push models past pattern matching against tutorials seen in pretraining.

How a WeirdML task works

Each task is presented as a self contained prompt with the problem description, training and test data paths, and a small example of how to load the data with NumPy or PyTorch. The model returns a Python script that handles everything from data loading through final evaluation. The code runs inside an isolated Docker container on a single NVIDIA TITAN V GPU with 12 GB of memory and a 600 second timeout. Network access is disabled, so the model cannot pull pretrained weights at run time.^[1][3]

A single run gives the model five submissions. After each attempt, the harness returns terminal output (errors and test accuracy if the run completed) and asks for a revision. The accuracy reported for the run is the best across the five submissions. Each model gets at least 15 runs per task; the headline score is the mean of best per run accuracies. The most expensive reasoning models (o1 preview, o3 pro, Claude thinking variants) get only five runs. A full evaluation can cost thousands of dollars, which is why METR's funding mattered for v2.^[1][4]

Tasks in WeirdML

WeirdML v2 has 19 tasks; only six are public. The other 13 are held out so the benchmark stays informative as public solutions accumulate. The six public tasks shipped with v1 in January 2025.^[2][3]

Scroll sideways for more →

The 13 hidden v2 tasks broaden the suite to cover more imaging problems, more sequential and tabular data, additional unsupervised setups, and tasks designed to span a wider difficulty range so that the leaderboard does not collapse around a small number of saturated entries.^[3][4]

Methodology details

Scroll sideways for more →

Ihle reported that on v1, the gap between five independent tries and five iterations with feedback was smaller than expected for non reasoning models. Most of the value of iteration came from more shots on goal, with extra benefit from feedback concentrated in reasoning models such as o1 mini, o1 preview, and gemini-2.0-flash-thinking. Newer reasoning models have widened that gap.^[1][2]

WeirdML v1 results

When Ihle published the v1 leaderboard in January 2025, Claude 3.5 Sonnet led at about 51% mean accuracy across the six tasks, with OpenAI's o1 preview close behind.^[1][2]

Scroll sideways for more →

Shapes (Easy) was effectively solved (o1 preview reached about 98%). Shapes (Hard) topped out near 60% on Claude 3.5 Sonnet. Chess outcome prediction stalled around 74% for the same model. Image Patch Shuffling (Hard) was unsolved, with most models near chance. Unsupervised Digit Recognition had a high first attempt failure rate but Claude 3.5 Sonnet averaged around 80% when its pipeline worked.^[1][2]

WeirdML v2 results

Ihle announced v2 in June 2025 alongside the Epoch AI integration. The v2 leaderboard uses 17 of the 19 tasks for the public score and reports much higher numbers than v1, partly because the model lineup improved and partly because v2 averages across a wider task set where some are easier on average.^[3][4][7]

Scroll sideways for more →

Ihle has posted snapshots tied to specific releases. When GPT-5 launched, he reported it leading at 56.3% (beating o3 pro at 53.9%) with gpt-5 mini matching o3 at a fraction of the cost. GPT-5 wrote much more code per attempt (median 324 lines vs about 133 for o3).^[7] Later releases such as GPT-5.4 (around 57.4%), GPT-5.5 (around 67.1%), and Claude Opus 4.7 (around 76.4%) continued the climb.^[8] Smaller open source baselines stay below 10%, with Mixtral 8x7B around 3.17%, and the pool exceeds 30 evaluated models.^[7]

Per task patterns shifted. Shapes (Hard) is no longer near chance for top models, with the strongest reasoning models reaching about 90%. Image Patch Shuffling (Hard) is still the toughest public task, though leaders have crept above chance. Most of the headroom now lives in the hidden v2 tasks.^[3][4]

WeirdML time horizons

In February 2026 Ihle published a follow up analysis called "WeirdML Time Horizons" on LessWrong, with code in the weirdml time horizons GitHub repository (MIT license). The idea borrows from METR's task duration framing: estimate how long a median professional ML researcher would need to solve each task without AI help, then ask at what task length each model crosses 50% success.^[9][10]

Ihle uses a panel of four LLMs to estimate per task human completion times at five accuracy thresholds (25%, 50%, 70%, 90%, 95%). Estimates become hours (1 day = 8h, 1 week = 40h) and feed a logistic fit, with block bootstrap resampling for uncertainty. The headline result: WeirdML time horizons roughly double every five months, from about 24 minutes for GPT-4 in June 2023 to roughly 38 hours for Claude Opus 4.6 in February 2026. That doubling rate is close to METR's reported seven month doubling, despite different tasks and methodology.^[9][10]

Scroll sideways for more →

The LLM panel likely overestimates absolute completion times, especially at high accuracy thresholds, so the hours should be read with skepticism. The doubling rate is more robust. A calibrated variant in the repository gives smaller absolute values but a similar doubling time of about six months.^[9]

Comparison with other benchmarks

Scroll sideways for more →

WeirdML is distinguished by three features: tasks are deliberately small and quirky, the harness automates a five iteration loop with execution feedback, and strict GPU and time limits force practical ML thinking.^[1][3][5]

Reception and use

Epoch AI's benchmarking hub added WeirdML v2 alongside Aider Polyglot, Balrog, and the Factorio Learning Environment when it expanded to feature trusted external leaderboards. WeirdML scores feed into the Epoch Capabilities Index, an aggregate measure across many benchmarks.^[5][11] Independent aggregators such as NeoSignal cite WeirdML alongside SWE-bench Verified and GPQA. Frontier releases since mid 2025 have included WeirdML scores in third party comparisons, especially for GPT-5, Claude Opus 4 variants, Gemini 3 Pro, and Grok 4.^[7][8] Ihle has framed WeirdML's role as keeping a meaningful signal alive while standard benchmarks saturate.^[1][12]

Limitations

Scroll sideways for more →

See also

• Benchmark
• Code generation
• LLM evaluation
• Reasoning models
• SWE-bench
• MLE-bench
• SciCode
• METR
• Epoch AI

References

1. Ihle, Håvard Tveit. "Introducing the WeirdML Benchmark." LessWrong, January 16, 2025. https://www.lesswrong.com/posts/LfQCzph7rc2vxpweS/introducing-the-weirdml-benchmark ↩
2. Ihle, Håvard Tveit. "WeirdML v1." Personal website. https://htihle.github.io/weirdml_v1.html ↩
3. Ihle, Håvard Tveit. "WeirdML." Personal website (current v2 page). https://htihle.github.io/weirdml.html ↩
4. Ihle, Håvard Tveit. "WeirdML v2 is now out." Twitter post, June 27, 2025. https://x.com/htihle/status/1938603525702930849 ↩
5. Epoch AI. "WeirdML (v2) benchmark page." https://epoch.ai/benchmarks/weirdml ↩
6. Ihle, Håvard Tveit. Personal homepage. https://htihle.github.io/ ↩
7. NeoSignal. "WeirdML benchmark scores." https://neosignal.io/benchmarks/weirdml ↩
8. Mowshowitz, Zvi. "GPT-5.5: Capabilities and Reactions." Don't Worry About the Vase, 2026. https://www.lesswrong.com/posts/5ytcFayxqZsXN8rNw/gpt-5-5-capabilities-and-reactions ↩
9. Ihle, Håvard Tveit. "WeirdML Time Horizons." LessWrong, February 16, 2026. https://www.lesswrong.com/posts/hoQd3rE7WEaduBmMT/weirdml-time-horizons ↩
10. Ihle, Håvard Tveit. "weirdml-time-horizons." GitHub repository. https://github.com/htihle/weirdml-time-horizons ↩
11. Epoch AI. "We've added four new benchmarks to the Epoch AI Benchmarking Hub." Twitter post, May 2025. https://x.com/EpochAIResearch/status/1919831883875062184 ↩
12. Ihle, Håvard Tveit. LessWrong shortform comments. https://www.greaterwrong.com/posts/Mrcsc7bEKjSbjq2op/havard-tveit-ihle-s-shortform ↩