Topic · 13 stories
13 stories tagged evals on AI Tamer, newest first.
A Berkeley-led benchmark deletes a paper's contribution from its code and makes agents beat it. Claude Opus 5.5 passes 58%; no agent passes 16 of 48 tasks.
Scope, an MIT-licensed research preview, gives Copilot and Claude Code one task, scores runs on a criteria graph with an LLM Judge, and keeps the evidence.
OpenAI's MultiLangBench has models build a SQLite-compatible engine in 20 languages, and a COBOL migration eval checks output byte for byte. Vendor-run.
A ByteDance Seed preprint says token offset can swing DeepSeek-V4 retrieval. The 40-point gap is the base checkpoint. Post-trained gaps are smaller.
Build a three-sample Inspect task with a solver chain and a built-in scorer, run it with inspect eval, and read the log to tell a wrong answer from a format mismatch.
A benchmark score depends on the prompt, the scoring method, the number of samples and the set of models compared. Here is what to check before you trust a row in a table.
NaiveAI’s Naive-N0.5-Flash (Sep 27, 2026): open-weight 309B MoE / 15.5B active, native 1M hybrid SWA–DSA, MIT on HF. API ($0.10/$0.40/$0.01) and NaiveRT remain future-tense—NaiveRT GitHub still 404 at the time of writing (promised by Oct 12). Benches/tok/s are NaiveAI harness claims only.
Holo4 (Sep 28, 2026): 27B dense (Qwen3.8) + 35B-A3B MoE (Qwen3.6) for GUI/code/MCP/API computer-use. License split: 27B weights CC BY-NC 4.0; 35B-A3B Apache-2.0. Both on H Models API + HF. OSWorld/cost figures are H Company harness claims only. Prefer BF16; context 256K (config max 262,144).
HF (2026-09-30) launched the Open TTS Leaderboard—objective WER/CER (Qwen3 ASR), RTFx, TTFA, and WavLM SIM for open-source and multilingual TTS/voice cloning. Eval infrastructure, not a model launch: ASR/WER is a proxy for intelligibility and does not measure listener preference or naturalness. Named ranks = blog snapshot; eval scripts still “soon.”
NVIDIA announced Kumo Tabular (HF blog 2026-09-29): open tabular foundation models for cls/reg via structured-data-models. Weights OpenMDW-1.1 on HF; library code Apache-2.0. No Inference Provider—local weights path. The TabArena and #1 claims are NVIDIA’s own.
Google DeepMind announced Gemini 4 Argon on 2026-09-30: phased to Fairwind cyber defenders first, 1M output tokens (not full context), intro $2/$10 then $4/$20 per 1M tokens on the blog—no public model ID or ai.google.dev pricing row yet.
SpaceXAI’s Grok 4.7 matches Grok 4.6 API token prices. Independent evals show Coding Agent Index gains with Grok Build at xhigh — and higher token use that can raise cost-per-task even when $/token is flat.
OpenAI cut GPT-6 Sol and Luna API prices roughly in half versus GPT-5.6 promo rates. Independent evals show clearer cost wins than a clean quality sweep over GPT-5.6 or Astra.
