Topic · 5 stories
5 stories tagged reproducibility on AI Tamer, newest first.
Stanford researchers retested 50 published LLM claims on 20 models: 44% expired, 34% weakened, median life 14.7 months. A project page, not a paper.
Some model names are fixed snapshots and some are pointers that move. How Anthropic and OpenAI name their models, how to tell which kind you are calling, and what even a pinned ID does not freeze.
Identical weights and inputs need not yield bitwise identical floating-point outputs across batch shapes or backends. Diagnose the size, location and consequence of differences.
Matching seeds do not guarantee matching results. Reproducing a run also means recording its software, hardware, and algorithm settings.
A benchmark score depends on the prompt, the scoring method, the number of samples and the set of models compared. Here is what to check before you trust a row in a table.
