Cohen's kappa measures agreement between two raters who assign items to categories. It compares their observed agreement with the amount of agreement expected by chance given how often each rater uses the available categories.
A kappa of 1 indicates perfect agreement, 0 indicates agreement no better than the model's chance expectation and negative values indicate agreement below that expectation. It is not an ordinary accuracy score because neither rater needs to be treated as the correct answer.
Kappa can be sensitive to the prevalence of categories and to whether raters favor the same labels. Two reviewers can agree on most examples yet receive a surprisingly low kappa when one category dominates. Weighted variants exist for ordered categories.
For AI evaluation or failure coding, report the number of reviewed examples, category definitions and raw agreement alongside kappa. Resolve disagreements using clear rules, and do not treat a high agreement statistic as proof that the labels themselves are valid.