Inter-rater reliability
Inter-rater reliability is the degree to which independent coders or raters, judging the same material, reach the same conclusions. In qualitative research it shows whether a codebook is applied consistently, usually reported as Cohen's kappa.
How it works
Two or more coders apply the same codebook to the same sample of data, independently. Their agreement is then measured. Simple percent agreement overstates reliability, because some agreement happens by chance, so Jacob Cohen proposed kappa in 1960, which subtracts the agreement expected by chance.
Kappa runs from below 0 to 1: 0 means agreement no better than chance and 1 means perfect agreement. A widely used scale reads 0.41 to 0.60 as moderate, 0.61 to 0.80 as substantial and above 0.80 as almost perfect. When agreement is low, the team revises the code definitions that caused disagreement and tests again.
kappa = (Po - Pe) / (1 - Pe), where Po is observed agreement and Pe is agreement expected by chance
A worked example
Say two analysts at a telecom code the same 200 support-call snippets as "billing" or "not billing". They agree on 170, so Po = 0.85. From how often each used each code, chance agreement Pe = 0.50. Kappa = (0.85 - 0.50) / (1 - 0.50) = 0.70, substantial agreement.
Related
Automate this in Pulse
A second coder checking agreement can open the same Pulse transcripts the first coder used, and both judge identical text instead of each other's notes.