A percentile describes a position in an ordered set of measurements. The 95th percentile, often written P95, is the value at or below which roughly 95 percent of observations fall, depending on the calculation convention. For website response times, it is a useful way to describe slower requests without focusing only on a single worst case.
Suppose most inference requests finish in one second but a few take six seconds. The average may conceal how often users encounter those slow responses. P50 describes the middle of the distribution, while P95 and P99 show behavior closer to its slow tail.
Percentiles are not percentages of speed, and they are sensitive to sample size, traffic mix and the time window being measured. A P95 computed separately for each server also cannot simply be averaged to obtain the fleet's true P95. Compare percentiles only across measurements collected using consistent methods.
For AI or edge deployments, track P50, P95 and failure rates alongside throughput. Segment by region, model and request size so that one slow workload does not distort conclusions about an otherwise well-performing system.