You Cannot Average P95s

The percentile belongs to requests

A p95 latency is a request duration at or below which 95% of observed requests fall. When several replicas serve an endpoint, each replica’s p95 summarizes its own requests. The combined endpoint p95 must come from the combined requests.

Prometheus warns that averaging precomputed quantiles across replicas yields statistically nonsensical values. An ordinary average gives each replica’s reported p95 equal influence, regardless of how much traffic it handled. Weighting those readings by request count still cannot recover the combined latency distribution. Each p95 leaves out where the other requests fell.

Histograms support aggregation

A histogram counts observations in latency buckets. Bucket counts can be combined across replicas before calculating a percentile. Prometheus then estimates p95 from the combined histogram with histogram_quantile(). The order matters: combine the histogram data first, then calculate the percentile once.

Prometheus’s guide distinguishes native histograms from classic histograms. For native histograms, its example sums the rates of histogram samples and passes the result to histogram_quantile(0.95, ...). For classic histograms, it sums bucket rates while retaining the le bucket boundary label. A summary exposes a percentile calculated in the instrumented program. That percentile cannot be aggregated into a valid service-wide percentile.

Histogram percentiles are estimates. With classic histograms, the width of the bucket containing p95 limits the accuracy of the reported latency. Native histograms use a configured resolution. Prometheus recommends native histograms where available, and suitable classic buckets when aggregation is needed but native histograms are unavailable.

What to do

Identify the endpoint and the replicas whose requests should count. Check whether their duration metric is a native histogram, a classic histogram, or a summary. For a histogram, filter to that endpoint, aggregate its data across replicas over the chosen time window, and calculate p95 from the result. For classic buckets, retain le during aggregation. If you only have summary percentiles, do not label their average as the endpoint p95. Add a histogram when you need that combined view.