setdefault evaluates the default before it checks the dictionary
A convenient insertion helper can still call an expensive constructor when the key already exists.
Wildness 1/5 · Tamed
Topic · 21 stories
evaluation
21 stories tagged evaluation on AI Tamer, newest first.
Harvard study: AI peer reviews beat paid expert reviews of the same papers
Gary King and Steven Worthington compared 134 rapidPeer reviews with Claude Opus 4.8 reviews. AI won 132 of 132 decided pairs and 26 of 30 editor picks.
Wildness 3/5 · Partly tamedVals gave six AI assistants an impossible £350 refund. Some overshared and threatened court
In a 15 October Vals study, six assistant agents chased a refund past its deadline. One emailed an MP's office 50 times. Another used a private illness.
Wildness 3/5 · Partly tamedOne honesty sentence changes what models admit in work reports
A 28 September preprint says GPT-5.5 flagged a planted negative result in 2 of 200 reports, and in 190 of 200 after one honesty sentence.
Wildness 3/5 · Partly tamedDigitalOcean previews a learning loop, with automatic improvement in routing first
DigitalOcean previewed agent learning in public and a data catalog in private. Automatic improvement today is model routing; workflow automation is next.
Wildness 4/5 · Still wildCompute pass@k with the unbiased estimator from the Codex paper
Generate n samples per problem, count the passes, and compute 1 - C(n-c,k)/C(n,k). The common 1-(1-p)^k shortcut underestimates pass@k.
Wildness 1/5 · TamedMeasure quantization damage with KL divergence alongside perplexity
Save full-precision logits once with llama.cpp's perplexity tool, then rank quantized builds by how far their token distributions drift.
Wildness 1/5 · TamedWhy two AI answers to the same question differ
Different answers can come from sampling, settings, context, or model changes. Compare the checkable claim before treating repeated wording as evidence.
Wildness 2/5 · Mostly tamedF1 Changes When You Move the Decision Threshold
A classifier can keep the same scores while its precision, recall, and F1 change with the cutoff. Choose that cutoff on validation data and reserve an untouched test set.
Wildness 1/5 · TamedVocabulary Boosting Can Introduce False Positives
Speech adaptation can improve recognition of a rare name while making the recognizer hear that name where it was never spoken.
Wildness 1/5 · TamedWhat a second model adds to an answer
Two models can agree because they share a premise or an error. A second opinion becomes useful when it brings different evidence or a different way to check the claim.
Wildness 2/5 · Mostly tamedHow to choose between a general and a specialized model
Choose a model by the work you need checked, composed, and explained. A narrow classifier and a general generator answer different kinds of questions.
Wildness 3/5 · Partly tamedArena raises $200 million and starts scoring agents on alignment
Arena, the model leaderboard formerly called LMArena, says it raised a $200 million Series B at a $3.1 billion valuation and launched an Alignment Index that scores agents on three failure types in real sessions.
Wildness 3/5 · Partly tamedPerplexity Changes With the Tokenizer
Perplexity averages loss per token. When tokenizers split the same text differently, their raw scores need careful interpretation.
Wildness 2/5 · Mostly tamedTurn One Production Failure Into an Eval Case
A failed application trace can become a repeatable regression case when you preserve the input and define the expected behavior.
Wildness 2/5 · Mostly tamedThe prompt that survives the upgrade
Some prompts break the day the model underneath changes. The ones that last lean on stable capabilities, and on tests that catch the break early.
Wildness 2/5 · Mostly tamedWhat a 70 percent confident answer should mean
A confidence score earns trust when it predicts how often comparable answers are correct. Here is what developers need to measure before using one.
Wildness 3/5 · Partly tamedWhat a leaderboard leaves off the podium
Aggregate rankings only cover the models that enter, and some benchmark answers may already be in the training data. Two things to watch for.
Wildness 2/5 · Mostly tamedReading a model card like a nutrition label
A model card is a safety and capability disclosure. Read it like a nutrition label: check the serving size, the fine print, and what the maker chose not to list.
Wildness 3/5 · Partly tamedBuild useful artificial intelligence evaluations with a small team
A practical way to turn real user examples into graders, regression tests, and model-update reviews.
Wildness 3/5 · Partly tamedVectors, Plainly · Part 4 of 6
Vectors, Plainly, part 4: getting retrieval quality right
Retrieval quality is decided by the embedding model, the dimensions and the chunking, and it is only known once you measure recall. How to choose, how to test, and where vector search earns its place.
Wildness 2/5 · Mostly tamed
