Stanford retested 50 well-known LLM findings on 20 models and found 44% had expired

How long does a published finding about LLMs stay true? Owen Queen and James Zou of Stanford University retested 50 widely cited claims on 20 models and conclude that most do not last long. Their project page, dated 2 November, says that “within 15 months of publication, most of the LLM claims we tested either no longer held or showed substantially diminished effects.” The work is published as a web page with a citation entry, not as a peer-reviewed paper, and we did not find a paper version.

What they did

The 50 claims range from sycophancy and jailbreaks to long-context recall and cultural bias. For each, the authors reproduced the original result on the paper’s model or a labelled substitute, then ran the same setup on 20 models released between July 2024 and September 2026 from eight developers. They fitted a trend against release date and measured how much of the original effect remains on 30 September 2026. A claim counts as expired when the effect falls to a fifth of the paper’s or less.

The headline numbers

• 44% expired (22 claims), 34% weakened (17), 16% persist (8) and 6% strengthened (3).
• Among the 22 expired claims, time from publication to expiry averaged 15.7 months, with a median of 14.7 months.
• Across all 50, the estimated share expired was 4% after six months, 16% after one year and 39% after two years.
• Older findings had more time to age: 11 of 21 claims from 2022 to 2023 had expired, against 2 of 10 from 2025.

Expired claims are mostly capability failures newer models no longer make: counting letters, finding facts in the middle of long contexts, fabricating citations, and abandoning correct answers when challenged. Weakened claims include the reversal curse, invented package names and susceptibility to jailbreaks and prompt injection. What persists is mostly preference and judgment: flattering and agreeing with users, judges that favour their own answers, and a left political lean.

The fake-citation case shows the shape. In 2023, GPT-3.5 invented 55% of its citations in the original study. The authors’ reproduction on GPT-3.5 Turbo got 59%, Claude Sonnet 5.5 fabricated 1.37% and GPT-6.1 Sol none. But gpt-oss-120b and Qwen3.7-Flash still fabricated 64 to 69%, results the authors exclude from the trend as outliers. Models released in the same quarter differ widely: 11 expired claims on DeepSeek V4 Pro against 30 on GPT-6.1 Sol.

Caveats the authors state

Four 2024 models anchor the early cohorts, so early expiry dates rest on few observations. Retired original models required substitutes, which give weaker reproduction evidence; 25 of 48 reproductions used an exact model match. Each claim uses one minimal design. The authors used Claude Code and Codex to build the analysis pipelines.

Why this matters

If a design choice, guardrail or eval in your stack rests on an LLM finding from 2023 or 2024, this work suggests rechecking it on the models you actually run, because some findings expired on one current model and not another. That is the same discipline as reading an eval table without being fooled: note which models and dates a number comes from. The cheap fix is the one in benchmarking a change without fooling yourself: rerun the claim on your own setup before you build on it.