A grader is the code, model or person that scores whether an AI agent's attempt succeeded. It might be an automated test that checks the result, another AI model that judges the answer or a human reviewer.
It matters because the scores are only as good as the grading. A weak grader can reward wrong answers or fail correct ones, and results from different graders are not directly comparable. The choice of grader affects cost and speed too: code is cheap and consistent, while human review is slow and expensive.
Each type has weaknesses. Automated checks can miss valid solutions that differ from what was expected, models can be inconsistent or biased, and people can disagree with each other. A grader that can be gamed lets an agent score well without really doing the task.
To use graders well, check them against a sample of cases reviewed by a person, report which type was used and look for ways the agent could pass without solving the task.