Elo ratings were developed to compare competitors from the outcomes of matches. In AI model comparisons, a similar system can turn pairwise preferences, such as a human choosing the better of two chatbot answers, into relative ratings. The expected result depends on the difference between competitors' ratings.
A higher rating indicates stronger performance within the evaluated population and test procedure. It is not an absolute measure of intelligence, a guarantee of accuracy or evidence that a model will be better at every kind of task. Ratings depend on the prompts, voters, matchups and rating method.
Some leaderboards derive Elo-like scores from pairwise results rather than ordinary standalone benchmark percentages. Direct test scores do not automatically become valid Elo ratings without an explicit comparison or conversion method, so the leaderboard's methodology matters.
When comparing open- and closed-weight models, check the number of matchups, uncertainty intervals, whether tasks match your use case and whether the leaderboard tested the exact model version. Small rating gaps may not be meaningful.