A confidence interval is a range of values likely to contain the true result, showing how much a measured score could vary. If an agent scores a given percentage on a test, the interval gives the range in which its real success rate probably lies, given that the test used only a limited number of tasks and trials.
It matters because small differences between scores often mean little. If the intervals of two agents overlap widely, the data may not support saying that one is better, even though the numbers are different.
An interval is itself based on assumptions, such as that tasks are independent and representative, and the usual confidence level is a convention. A wide interval signals that more tasks or trials are needed, while a narrow one does not guarantee that the test measures what you care about. It also does not mean the true value lies inside the range for certain.
When comparing results, look for intervals or the number of tasks and trials needed to calculate them, and be cautious about drawing conclusions from small gaps.