Many evaluations count passes and fails, such as how many tasks a model solved out of the total. For a proportion p measured on n independent items, the standard error is the square root of p times (1 minus p) divided by n.
An example shows how much sample size matters. A model that solves 80 percent of 100 tasks has a standard error of about 4 percentage points, so a rough 95 percent range is about 72 to 88 percent. The same 80 percent on 1,000 tasks has a standard error of about 1.3 points.
The formula assumes that the items are independent and the sample is not tiny. It works poorly for results very near 0 or 100 percent on small tests. When two models are tested on the same tasks, a paired comparison is more precise than comparing two separate ranges.
Before you rank models by a few points, work out the standard error, and collect more test cases if the ranges overlap.