Why percent agreement is not enough
If two annotators label 95% of items as OK because 95% of items are OK, they will agree most of the time even when they disagree on every item that matters. Percent agreement rewards that luck. Kappa subtracts it: it compares the observed agreement with the agreement two raters with the same label frequencies would reach if they labelled at random, and scales the result so that 1 is perfect agreement and 0 is chance level. Negative values mean less agreement than chance.
Step 1: the agreement table
Put one rater in the rows and the other in the columns and count the items in each combination. The diagonal holds the agreements. With COUNTIFS on a list of item labels, each cell is =COUNTIFS(RaterA,"Positive",RaterB,"Negative") and so on. Add row and column totals: in the example annotator A used Positive 50 times, Neutral 45 and Negative 55, annotator B 52, 43 and 55.
Step 2: observed and chance agreement
Observed agreement is the diagonal divided by the total: =(B2+C3+D4)/E5 = (44 + 31 + 46) ÷ 150 = 0.807.
Chance agreement multiplies each category's row total by its column total, adds the products and divides by the total squared: =(E2*B5+E3*C5+E4*D5)/E5^2 = (50 × 52 + 45 × 43 + 55 × 55) ÷ 150² = 7,560 ÷ 22,500 = 0.336. It is the share of agreements you would expect if each annotator kept their own label frequencies but assigned labels at random.
Step 3: Cohen's kappa and how to read it
=(E7-E8)/(1-E8) gives (0.807 − 0.336) ÷ (1 − 0.336) = 0.709. The most quoted scale, by Landis and Koch (1977), reads 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial and above 0.80 as almost perfect. These bands are a convention, not a test. Many labelling teams set their own threshold, often around 0.7 or 0.8, before a label set is used for training, and the right bar depends on how costly a wrong label is.
Worked example: two agreement tables, two kappas
The counts are inputs (rows: annotator A, columns: annotator B). Totals, agreement shares and kappa are formulas. The upper table has three balanced categories, the lower one a rare label.
| A | B | C | D | E | |
|---|---|---|---|---|---|
| 1 | A ↓ / B → | Positive | Neutral | Negative | Total |
| 2 | Positive | 44 | 5 | 1 | 50 |
| 3 | Neutral | 6 | 31 | 8 | 45 |
| 4 | Negative | 2 | 7 | 46 | 55 |
| 5 | Total | 52 | 43 | 55 | 150 |
| 6 | |||||
| 7 | Observed agreement p_o | 80.7% | |||
| 8 | Chance agreement p_e | 33.6% | |||
| 9 | Cohen's kappa κ | 0.709 | |||
| 10 | Landis & Koch band | Substantial | |||
| 11 | |||||
| 12 | A ↓ / B → | Toxic | OK | Total | |
| 13 | Toxic | 4 | 5 | 9 | |
| 14 | OK | 4 | 187 | 191 | |
| 15 | Total | 8 | 192 | 200 | |
| 16 | |||||
| 17 | Observed agreement p_o | 95.5% | |||
| 18 | Chance agreement p_e | 91.9% | |||
| 19 | Cohen's kappa κ | 0.447 | |||
| 20 | Positive agreement on Toxic, 2a ÷ (2a + b + c) | 47.1% | |||
- E7
=(B2+C3+D4)/E5Diagonal ÷ total - E8
=(E2*B5+E3*C5+E4*D5)/E5^2Sum of row total × column total ÷ total² - E9
=(E7-E8)/(1-E8)Cohen's kappa - E19
=(E17-E18)/(1-E18)Kappa with a rare label
Synthetic labels for illustration. Kappa: Cohen (1960). Interpretation bands: Landis and Koch (1977).
The rare-label trap
The second table in the example has two annotators flag 200 support tickets as Toxic or OK. They agree on 191, which is 95.5%, better than the first pair. But only 9 and 8 tickets were flagged toxic, so chance agreement is already 91.9% and kappa falls to 0.447, only moderate. The real problem is visible in the rare class: of the 13 tickets either annotator flagged, they agreed on only 4. The positive agreement on the rare label, 2a ÷ (2a + b + c), is 8 ÷ 17 = 47.1%. With imbalanced labels, report kappa together with the agreement on the rare class.
More than two raters, ordered labels and other variants
Cohen's kappa compares exactly two raters. For three or more raters who each label every item, use Fleiss' kappa, which works from the number of raters choosing each category per item. For ordered categories such as a 1–5 rating, a weighted kappa gives partial credit to near misses, with linear or quadratic weights. For the internal consistency of scale items rather than raters, see Cronbach's alpha in Excel, and for comparing a model's labels with the truth rather than two humans, use a confusion matrix with precision and recall.
Common mistakes
- Reporting only percent agreement. 95.5% agreement can hide a kappa of 0.447 when one label is rare.
- Using the grand total instead of its square for pe. Chance agreement divides the sum of row × column totals by N², not N.
- Mismatching row and column totals. Multiply each category's row total by the column total of the same category. SUMPRODUCT of a vertical and a horizontal range does not do that, so write the products out or use MMULT.
- Using Cohen's kappa for three or more raters. Averaging pairwise kappas is not the same as Fleiss' kappa.
- Treating the bands as a test. 0.61 is not significantly better than 0.59. Report kappa with the sample size, or a confidence interval for important decisions.
- Ignoring ordered categories. On a 1–5 scale, a 4 against a 5 is a smaller disagreement than a 1 against a 5. Use a weighted kappa there.
Frequently asked questions
How do you calculate Cohen's kappa?
Build the agreement table of the two raters, compute the observed agreement p_o as the diagonal divided by the total, compute the chance agreement p_e as the sum of row total times column total for each category divided by the total squared, then kappa = (p_o - p_e) / (1 - p_e).
What is a good Cohen's kappa value?
On the Landis and Koch scale, 0.61 to 0.80 is substantial and above 0.80 almost perfect agreement. Many annotation teams aim for at least 0.7 to 0.8 before using labels, but the right threshold depends on the task and on how costly label errors are.
Does Excel have a Cohen's kappa function?
No. Excel has no built-in kappa function, but three formulas are enough: observed agreement, chance agreement and (p_o - p_e) / (1 - p_e). COUNTIFS builds the agreement table from two columns of labels.
Why is kappa low when agreement is high?
When one category dominates, two raters agree often by chance alone, so chance agreement is high and little room is left above it. Disagreements on the rare category then weigh heavily, which is why kappa drops.