LiveNerf
LiveNerf (styled livenerf in its repository) is an open-source, pre-registered benchmark that tracks whether a frontier large language model gets worse after it is released. It was started by the GitHub user ninjahawk on September 22, 2026, the day Anthropic released Claude Opus 5.5, and it measures that one model as served through headless Claude Code on a Claude Max subscription.[1][6][11] The repository describes itself as "a long-running, deterministic-as-possible benchmark for detecting whether a frontier model gets quietly worse after launch."[1] It runs a fixed panel of 78 benchmark questions once a day for 30 days and compares later 10-day windows against a launch-week baseline using paired, item-clustered statistics fixed in advance.[1][2] As of its last published update (October 16, 2026), the first post-baseline window showed no change, and the earliest point at which the pre-registered rule could declare a change is around October 24, 2026.[1] The code is MIT-licensed. A September 2026 Hacker News thread about the project, titled "Livenerf: Has Opus 5.5 been nerfed yet?", reached 926 points.[7]
Background
In discussions of AI services, "nerfing" refers to user reports that a model seems to get worse some days or weeks after launch. The LiveNerf README says that for months there have been reports that Anthropic "nerfs" models after release, and summarizes the suspected mechanisms as "quantization, a smaller model behind the same name, lower effort, or routing changes," and adds that it "could also mean nothing happened and people are pattern-matching on noise."[1] Its stated motivation is that "Nobody has had a clean day-0 baseline to check against, so every argument ends up as vibes versus vibes."[1]
Anthropic has twice published engineering reports tracing user complaints about degraded Claude output to specific causes:
| Date | Anthropic report | What it described |
|---|---|---|
| September 17, 2025 | "A postmortem of three recent issues" | Three infrastructure bugs between August and early September 2025: a context-window routing error affecting Claude Sonnet 4 requests (16% of Sonnet 4 requests at the worst hour on August 31), an output-corruption misconfiguration on TPU servers, and an approximate top-k miscompilation in the XLA:TPU compiler.[9] |
| April 23, 2026 | "An update on recent Claude Code quality reports" | Three changes affecting Claude Code, the Claude Agent SDK and Claude Cowork: a lowered default reasoning effort (high to medium, March 4 to April 7), a caching change that dropped prior reasoning every turn (fixed April 10), and a verbosity-reducing system prompt instruction (reverted April 20). Anthropic said the API was not impacted.[10] |
In the 2025 postmortem Anthropic wrote: "To state it plainly: We never reduce model quality due to demand, time of day, or server load. The problems our users reported were due to infrastructure bugs alone."[9] The 2026 report says: "We never intentionally degrade our models, and we were able to immediately confirm that our API and inference layer were unaffected."[10] LiveNerf's plan cites the 2025 incidents directly, noting that they "were infrastructure bugs (context-window routing, token corruption, a compiler top-k bug), not deliberate downgrades," and uses that as the reason it tests for change in either direction rather than assuming a mechanism.[3]
History
| Date (2026) | Event |
|---|---|
| September 22 | Anthropic releases Claude Opus 5.5.[11] The ninjahawk/livenerf repository is created the same day with a measurement plan and a v0 harness; the author posts it as a Show HN.[6][8] |
| September 23 | Calibration screen of candidate questions begins; MMLU-Pro and competition math are added to the pool after AIME 2025-26 proves to be answered largely from memory.[2][4] |
| September 24 | Protocol v2 is pre-registered, the 78-question panel is locked, instrument validation passes, and the schedule changes from hourly to once a day. Series day 1 runs at 22:10 UTC, about 2.5 days after launch.[1][2][6] |
| September 29 | The Hacker News thread "Livenerf: Has Opus 5.5 been nerfed yet?" is posted; it reached 926 points and 394 comments.[7] |
| October 3 | Day 10 completes the baseline; an MIT license is added; a bug that would have mis-aligned the baseline cutoff (issue #9) is fixed before any post-baseline data.[2][6] |
| October 5 | The realized minimum detectable effect from baseline data is recorded as 6.6 points per 10-day window. A crashed day-12 run is excluded and rerun.[2] |
| October 12 | Day 19 runs on Anthropic API credits because the Max plan's weekly meter was exhausted; its logs are labeled funding_source=anthropic_api.[1][2] |
| October 14 | The first decision window (days 11-20) is published: no change.[1][6] |
| October 16 | Latest update at the time of writing: 23 of 30 days collected, none missed.[1][6] |
Design
What is measured
The measured model is claude-opus-5-5, called through claude -p (headless Claude Code) at effort high on every sample.[2] The README stresses that this is "Opus 5.5 as served through Claude Code on a subscription," which it calls "the thing most nerf reports are actually about," and not the raw API model.[1] The evaluation card lists what it does not measure: absolute capability, the raw API model, newer Claude Code releases, claude.ai chat, other effort levels, long context, tool use, agentic behavior, or the cause of any change.[5]
Because the model cannot be made deterministic (the plan notes that sampling parameters are unavailable and thinking cannot be turned off on Opus 5.5), the project fixes everything else and measures distributions over many samples.[1][3] Every call is hermetic: a short frozen system prompt, no tools, no MCP servers, no settings files or hooks, no CLAUDE.md or memory, one turn, and a fixed empty working directory.[1][2] The Claude Code CLI version is pinned in a CLAUDE_CLI_VERSION file (2.1.280), and the runner refuses to run if claude --version stops matching it, because "a changed harness looks exactly like a changed model."[1] Samples that Claude Code retried, that another model served in part, or that ended in a refusal are rejected and counted as classifier events rather than scored.[2] The harness is built on Inspect, the open-source evaluation framework from the UK AI Security Institute.[1][13]
The question panel
Questions were chosen by calibration. 2,336 candidate questions from GPQA Diamond, MMLU-Pro, competition math and AIME 2025-26 were each sampled 4 times; a question was eligible if Opus 5.5 got it right 1 to 3 times out of 4 and it triggered no classifier event.[2][4] A question the model always gets right cannot show a drop, and the README reports that 97% of screened questions were either always right or always wrong across their four screening samples.[1]
| Benchmark | Questions screened | First-try accuracy | Eligible |
|---|---|---|---|
| GPQA Diamond | 198 | 92.8% | 12 |
| MMLU-Pro (seeded 2,000-question subset) | 2,000 | 92.8% | 59 |
| Competition math (BRUMO, CMIMC, HMMT Feb 2025, APEX) | 78 | 96.2% | 6 |
| AIME 2025-26 | 60 | 98.3% | 3 |
| Total | 2,336 | 80 |
Source: docs/CALIBRATION.md.[4] Two of the 80 were later excluded after classifier events during a confirmation stage, leaving the 78-question panel (59 MMLU-Pro, 12 GPQA Diamond, 4 competition math, 3 AIME).[2][5] Eight fresh samples per question showed the selection effect: the panel's mean pass rate was 54.7% on the screen and 62.0% on fresh samples, so the power calculation uses the fresh rates.[1][4] Scoring is exact match with no LLM judge, "since the judge would drift too."[1] GPQA questions are never committed to the repository at the dataset authors' request; only panel question ids and hashes are published.[1]
A control arm runs the panel's 12 GPQA questions on Claude Opus 5 through the same harness each day. If both models move together, the project attributes the change to the harness or platform rather than to Opus 5.5.[1][2]
Statistics and decision rule
The analysis follows Evan Miller's paper "Adding Error Bars to Evals" (arXiv:2411.00640).[2][12] The primary statistic is, for each panel item, its mean score in a 10-day window minus its mean in the baseline (days 1-10), averaged over items, with standard errors clustered by item. A change is declared only when all of the following hold:[2]
- The 99% interval excludes zero in two consecutive 10-day windows, in the same direction.
- The effect is at least 3 points in both windows.
- The harness (CLI version and sample-shaping code hash) is identical to the baseline's.
- The sample error rate in those windows is below 5%.
If the control arm shows a matching move, the result is reported as a harness or platform change. Anything short of the rule is reported as "no change detected" together with the minimum detectable effect (MDE).[2] The test is two-sided, and the pre-registration commits to publishing every 10-day result, including nulls and improvements.[2] Median output tokens per sample are tracked as a secondary signal; the README says "If a model quietly starts thinking less, this is where it shows up first, often before accuracy moves at all."[1]
The design predicted an MDE of about 7.5 points per 10-day window at roughly 3.6% of the Max plan's weekly usage meter; the realized MDE computed from baseline data alone was 6.6 points.[1][2]
Instrument validation
Before the baseline, the rig was tested on known manipulations of the same panel.[2][4]
| Arm vs Opus 5.5 at effort high | Accuracy change (points) ± SE | Output tokens |
|---|---|---|
| Opus 5.5, effort medium | -4.2 ± 3.9 | -26% |
| Opus 5.5, effort low | -8.3 ± 4.5 | -62% |
| Claude Opus 5 (model swap) | -3.8 ± 6.3 | -23% (99% CI -46% to +8%) |
The validation passed its pre-registered criterion, but the model swap was not distinguishable from Opus 5.5 at 99%. As the pre-registration required, the README states that "This instrument can't detect a same-family model swap of that size in a validation's worth of samples."[1][2] An A/A check split the high-effort samples in half and found +6.4 ± 3.6 points (z = 1.79), which the pre-registration calls "Consistent with 0, but close to the threshold."[2]
Known caveats
A report-only audit of the 80 selected questions classed 42 as sound, 30 as ambiguous and 8 as having a suspect answer key; nothing was dropped, and a pre-registered sensitivity analysis reruns the result without them. The auditor was Claude, which the project flags as a conflict of interest.[2][4] The README also discloses that "a lot of this repo is written with the help of Claude, which is the model being measured," and cites that as the reason for pure-function graders, pre-registered thresholds and public raw data.[1] Other stated threats include a single daily run time (results "don't generalize to peak hours"), few samples per question, and safety-classifier behavior on the serving path.[2] The pre-registration's deviations log records each change made after it was written, including the move from hourly to daily sampling, two budget-guard overrides, the excluded crash, and the API-funded day.[2]
Results
As of the latest update (October 16, 2026; checked October 17, 2026):[1]
| Window | Samples | Score | Change vs baseline ± SE | Median output tokens | Control arm change | Decision |
|---|---|---|---|---|---|---|
| Days 1-10 (from September 24) | 780 | 0.584 ± 0.036 | baseline | 588 | baseline | baseline |
| Days 11-20 (from October 4) | 780 | 0.592 ± 0.037 | +0.007 ± 0.018 (78 items) | 606 | +0.035 ± 0.049 (12 items) | no change |
Window 2 (days 21-30) had 3 of 10 days collected, and all 23 days so far ran the full 90 samples on the same harness hash and pinned CLI.[1] Because the decision rule requires two consecutive qualifying windows, LiveNerf cannot declare a change before the second window closes.[1][2] The scores are not comparable to published benchmark results, since the panel deliberately contains only questions the model sometimes misses.[5]
Reception
The Hacker News thread posted on September 29, 2026 reached 926 points and 394 comments.[7] Discussion split along familiar lines:
- Skeptics argued that perceived nerfs are mostly psychological. One commenter who had worked on a chat app with full control of its stack said users still regularly complained that models got nerfed, and others pointed to honeymoon effects or users hitting a model's complexity ceiling after the first days.[7]
- Believers shared anecdotal reports of drops after launch and speculated about load-dependent serving, peak-hour behavior, or changes tied to subscription usage meters. Several asked whether subscription and API traffic are treated differently.[7]
- Methodological critics noted that the instrument could not distinguish Opus 5 from Opus 5.5 in validation. The author replied that the swap was a validation test, that the 10-day windows hold more samples but "I haven't demonstrated that they're sufficient," and that "A null result is a result too."[7] Others suggested Anthropic could recognize the benchmark's prompts; the author agreed this is a limitation of any public benchmark and said they would like to add held-out or periodically refreshed panels later, without quietly changing the current one.[7] Another commenter suggested comparing against Claude served by other providers such as Amazon Bedrock.[7]
- Commenters also linked Anthropic's two postmortems and other trackers, including BridgeBench's Nerf Bench and Marginlab's Claude Code tracker.[7]
Related trackers
Several other public projects track post-launch performance of AI models. Each uses its own method and is not directly comparable to LiveNerf.
| Tracker | Operator | Method (as described on its site) |
|---|---|---|
| Nerf Bench | BridgeBench (BridgeMind) | Each model starts at 100% on its first test; 90% to 110% is treated as normal variance. As of its October 16, 2026 test it listed Claude Opus 5.5 at 105.8% of launch, within normal variance.[14] |
| Claude Code Opus 5.5 Performance Tracker | Marginlab | Daily runs on a curated subset of SWE-Bench Pro in the Claude Code CLI; as of its September 29, 2026 update it was collecting a new Opus 5.5 baseline with degradation detection paused.[15] |
| IsItNerfed? | IsItNerfed | Continuously runs coding tasks against models and charts success rates over time; a separate "Vibe Check" collects user impressions.[16] |
References
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27ninjahawk. "livenerf" (README). GitHub. github.com/...livenerf
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24ninjahawk. "livenerf pre-registration (suite v1)". GitHub. github.com/...PREREGISTRATION.md
- ^1 ^2ninjahawk. "LiveNerf: Measurement Plan" (PLAN.md). GitHub. github.com/...PLAN.md
- ^1 ^2 ^3 ^4 ^5 ^6ninjahawk. "livenerf pre-baseline record" (docs/CALIBRATION.md). GitHub. github.com/...CALIBRATION.md
- ^1 ^2 ^3ninjahawk. "livenerf eval card (suite v1)" (docs/EVAL_CARD.md). GitHub. github.com/...EVAL_CARD.md
- ^1 ^2 ^3 ^4 ^5 ^6ninjahawk/livenerf commit history. GitHub. github.com/...main
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9"Livenerf: Has Opus 5.5 been nerfed yet?" Hacker News, September 29, 2026. news.ycombinator.com/item
- ^"Show HN: Livenerf - a benchmark for whether Opus 5.5 gets nerfed". Hacker News, September 22, 2026. news.ycombinator.com/item
- ^1 ^2Anthropic. "A postmortem of three recent issues". September 17, 2025. anthropic.com/...a-postmortem-of-three-recent-issues
- ^1 ^2Anthropic. "An update on recent Claude Code quality reports". April 23, 2026. anthropic.com/...april-23-postmortem
- ^1 ^2Anthropic. "Introducing Claude Opus 5.5". September 22, 2026. anthropic.com/...claude-opus-5-5
- ^Miller, Evan. "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations". arXiv:2411.00640, 2024. arxiv.org/...2411.00640
- ^UK AI Security Institute. "Inspect". inspect.aisi.org.uk
- ^BridgeBench. "Nerf Bench". bridgebench.ai/nerf-bench
- ^Marginlab. "Claude Code Opus 5.5 Performance Tracker". marginlab.ai/...claude-code
- ^IsItNerfed. "Is It Nerfed? Continuous LLMs Evaluation". isitnerfed.org
Improve this article
Add missing citations, update stale details, or suggest a clearer explanation. Every suggestion is reviewed for sourcing before it goes live.
2 revisions · v2 · 2,530 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Reviewer note: Independent verification 2026-10-17 (xg19 V6): repo files, commit history, HN thread and Anthropic postmortem quotes checked; 3 minor fixes applied. Results are a dated snapshot.
Cite this page: AI Wiki. "LiveNerf." aiwiki.ai, updated 16 Oct 2026, fact-checked 16 Oct 2026. CC BY 4.0. https://aiwiki.ai/wiki/livenerf