Learn · Intermediate
Inter-rater reliability: when two graders disagree
Inter-rater reliability measures how consistently different people or systems apply the same judgment process. It matters because an evaluation score is harder to interpret when equally qualified graders disagree about the same cases.
Consider two reviewers checking whether an AI assistant’s answer is safe. If one calls a response acceptable and the other calls it unsafe, their disagreement is information about the evaluation, not just an inconvenience to be averaged away. It may reveal an ambiguous rule, a genuinely difficult case, or different assumptions about context.
Agreement is not the same as accuracy
Two reviewers can agree perfectly and still both be wrong. Conversely, two experts can disagree because a case is genuinely uncertain. Reliability asks whether the measurement process is consistent; validity asks whether it measures the intended thing. A reliable but poorly designed rubric can faithfully measure the wrong target. The distinction is central to AI benchmarking and to models used as judges.
Start with a simple measure: count the cases on which the raters make the same choice, then divide by the total. If they agree on 90 of 100 cases, observed agreement is 90%. That number is intuitive, but it can be misleading when one label is very common. Suppose both reviewers independently call almost everything safe; they may agree frequently even without a shared ability to identify the difficult cases.
Adjusting for chance agreement
Jacob Cohen’s 1960 paper on the kappa coefficient proposed comparing observed agreement with the agreement expected from the raters’ overall label frequencies. Cohen’s kappa is (observed agreement − expected agreement) / (1 − expected agreement). If observed agreement is 0.90 and the expected agreement computed from the label frequencies is 0.82, kappa is about 0.44. The adjustment reveals that the raw 90% agreement provides less information than it first appears to.
The expected term is calculated from the raters’ marginal frequencies, not from a universal constant. In a two-label case, multiply the fraction of safe labels assigned by the first rater by the fraction assigned by the second, do the same for unsafe labels, and add the two products. This is a statistical reference model, not proof that any individual rater actually guessed.
Kappa also has limits. When a label is rare, it can be low even with high raw agreement; the pattern of prevalence and the reviewers’ label tendencies matters. Report the underlying counts, raw agreement, label distribution, and a measure of uncertainty alongside an adjusted coefficient. A single cut-off cannot tell readers whether the reliability is adequate for every use: triaging a high-stakes incident needs a different tolerance from sorting a casual recommendation list.
Design the evaluation around disagreement
First, write a rubric with clear categories, examples and explicit rules for borderline cases. Then have at least two graders independently score the same representative sample. Independence matters: if the second reviewer sees the first person’s decision, apparent agreement can reflect influence rather than reproducibility.
Build a confusion matrix: rows record one rater’s labels and columns record the other’s. Inspect disagreements by type, not just their total. Do reviewers disagree about factual correctness, authorization, severity, or whether a response completed the requested work? These patterns tell a team whether to clarify instructions, change categories, collect missing context or accept that some judgments are intrinsically contestable.
An adjudicator can resolve a disagreement for a final dataset, but keep the original ratings. Replacing them with a single consensus label hides the uncertainty that the evaluation should reveal. If an AI grader is involved, compare it against multiple independent human judgments and measure whether errors cluster in particular categories. Human consensus itself is not an infallible gold standard.
Reliability depends on the sample and task. Reviewers who agree on obvious cases may disagree on borderline failures; scores from a balanced test set may not transfer to a deployment where severe failures are rare. When categories have a natural order, disagreements one step apart may be less serious than disagreements at opposite ends, but a weighted agreement measure requires choosing and reporting those weights in advance.
The same caution applies to research results. OpenAI’s safety-framing report uses hidden rubric items, but its public account does not provide enough information to calculate rater agreement for those assessments. The general lesson is not that its scores are wrong; it is that transparent grading procedures and uncertainty help readers understand what a score can support. Pair reliability analysis with cross-validation and a clear validity argument before treating an evaluation number as a stable fact.
Key questions
Does high agreement mean the graders are correct?
Why use Cohen’s kappa instead of agreement alone?
What should a team do with disputed cases?
Cite this
APA
Ground Truth. (2026, November 6). Inter-rater reliability: when two graders disagree. Ground Truth. https://groundtruth.day/learn/inter-rater-reliability-and-cohens-kappa.html
BibTeX
@misc{groundtruth:inter-rater-reliability-and-cohens-kappa,
title = {Inter-rater reliability: when two graders disagree},
author = {{Ground Truth}},
year = {2026},
month = {nov},
url = {https://groundtruth.day/learn/inter-rater-reliability-and-cohens-kappa.html}
}