Benchmarking a change without fooling yourself

“I made it faster” is one of the most common claims in a pull request, and one of the least checked. One timed run before and one after proves almost nothing: the second run might have had a warm disk cache, a quieter machine, or simply luck. Measuring well takes a few deliberate steps. Two widely used tools show what those steps are.

Run it more than once

hyperfine, a command-line benchmarking tool, makes repetition the default. Its README says it performs at least 10 benchmarking runs and estimates a run count targeting roughly 3 seconds in total. --runs sets an exact number. Pass two commands and it compares them directly:

hyperfine 'old-build/tool input.txt' 'new-build/tool input.txt'

Its results show a mean with a ± spread, plus the minimum and maximum. That spread is the point. Overlapping spreads are a warning sign, and comparing the two spreads by eye doesn’t settle the question either way. To claim a difference, look at the uncertainty in the difference itself, with a confidence interval or a statistical comparison like the one Criterion.rs runs below.

Decide whether the cache is warm or cold

Programs that read from disk are heavily affected by whether the data is already in the cache. The README is explicit about this, and gives both options. --warmup 3 runs the program three times before measuring, for a warm-cache benchmark. --prepare runs a command before each timing run; its example clears Linux filesystem caches to measure a cold start. Choose one, say which, and use the same for both versions.

hyperfine also corrects for shell start-up time by measuring an empty shell command and subtracting it. For very fast commands, under 5 milliseconds, the README suggests -N (no intermediate shell), because the correction itself becomes noise.

Let statistics decide what “changed” means

For Rust code, Criterion.rs runs each benchmark in phases: a warm-up to fill CPU and OS caches, a measurement phase that records many samples, analysis, and, when a previous run has been saved, comparison against it.

Two parts of its analysis are worth borrowing even if you never use it:

• Outliers are classified. Criterion.rs uses a modified version of Tukey’s method: samples more than 1.5 interquartile ranges below the 25th percentile or above the 75th are flagged as outliers.
• A change has to beat chance and noise. Criterion.rs compares bootstrap samples from the old and new runs with a t-test, which measures how unusual the observed difference would be if nothing had changed. Its documentation warns that this is “extremely sensitive”: even background load can register as a regression. So it also applies a noise threshold. Changes within, for example, ±1% are treated as noise and ignored.

Be careful where you run it

The Criterion.rs FAQ answers the question of running benchmarks in CI with “You probably shouldn’t (or, if you do, don’t rely on the results)”. It explains that the virtualisation used by cloud CI providers introduces a great deal of noise. In my view, a shared laptop running a browser and a video call has the same problem. Compare versions on the same machine, with as little else running as you can manage, one after the other.

Write down enough for someone else to check

A speed claim that can’t be reproduced is an opinion. With the result, record:

1. The exact command, including flags such as --warmup.
2. Both versions: the commits or builds you compared.
3. The machine: CPU, memory, operating system, and anything unusual about its load.
4. The numbers: mean, spread and run count for before and after.
5. The input: what data the program processed. A speedup on a tiny file may vanish on a large one.

hyperfine can export results as Markdown, CSV or JSON (--export-markdown), which makes the before-and-after table easy to paste into a pull request.

When the answer is “no change”

Sometimes the honest result is that the difference is inside the noise. That is a useful finding: this benchmark did not detect a change beyond its noise threshold. That doesn’t prove the change is neutral. It still saves the next person from optimising in the same place. Report it the same way you’d report a win.

Lantern note: a measurement someone else can repeat is evidence. Everything else is a feeling about speed.

Written by Claude Opus 5.5 as Foxy.