Panel quality
Panel quality is how trustworthy the data from a research panel is: whether its members are real, correctly profiled, attentive and representative of the audience they are meant to stand for. It determines whether findings from panel research can be believed.
How it works
Quality has several parts. Recruitment decides who can join. Verification confirms identity and profile data. Fraud controls catch bots, duplicates and people who misrepresent themselves to qualify. Panel management sets how often members are invited, which affects fatigue and conditioning. In-survey checks flag speeding, straight-lining and nonsense open-ended answers.
An AAPOR task force on online samples set out to propose sample-quality metrics beyond completion and response rates, and to apply the AAPOR Code's reporting guidelines to online panels. Pew Research Center tested the two most common checks, a failed attention question and speeding, and found that 76 percent of bogus respondents passed both. Strong programs therefore add further checks, such as flagging non sequitur answers and people who take a survey more than once.
A worked example
Over 6 months of panel studies, a consumer goods brand's insights team logs how many completes each sample source loses to quality checks: 4 percent from source A, 6 percent from B and 21 percent from C. The team drops C. A study that needs 1,000 usable completes now buys about 1,064 from B, where C would have needed about 1,266.
Related
Automate this in Pulse
Fraud checks on each recruited participant are Pulse's answer to the first test of panel quality, whether the people answering are real.