Where sampling bias enters an interview study, and how to catch it

A director asks who the insights team talked to. The honest answer is whoever replied to a CRM email. That is where sampling bias in an interview study starts: in how you find and invite people, at five points before the first question. So check those five points before interviews begin. More interviews will not fix it.

A retention plan built on talks with happy customers can spend a year fixing the wrong reason people leave.

The checks below show, before the first interview, whether your sample can answer the question, and which group to add if it cannot.

Sampling bias starts before the first interview

Sampling bias means some people were more likely to end up in your study than others. So the findings lean toward them. It is set by who you recruit. A good interviewer cannot undo it. And more interviews from the same list only repeat the same lean, with more confidence.

Compare it with random error. That is the chance difference between one sample and the next, and it shrinks as the sample grows. Sampling bias does not shrink as the sample grows, because it comes from the method.

For illustration, a bank's card team wants to know why people stop using their card. It talks to active cardholders, then books twice as many from the same list. Every one of them still uses the card, so the people who stopped are still missing.

This post covers who ends up in the sample. It does not repeat interviewer bias or analysis bias, which are covered in the guide to other kinds of bias in user research.

Five steps decide who you hear from

Sampling bias gets into an interview study at five steps: the list you recruit from, the screener and fraud checks, the invitation and reward, the time and format, and who finishes. Each step can only shrink the group the step before it let through. So a group lost early stays lost.

1. Your recruit source decides who can take part

Your sample frame is the list you recruit from. Anyone the list cannot reach is missing from the start. Researchers call this gap undercoverage.

For illustration, a CRM export of customers who agreed to marketing email leaves out those who canceled, went quiet or opted out. For a question about why people leave, those are often the ones with the answer.

Hand-picked lists have the same problem. When account managers choose who gets invited, they tend to pick their friendliest accounts. Those are the clients least likely to complain.

Recruiting in public is a recruit source with its own risk. NORC at the University of Chicago, in its evaluation of commonly used recruitment approaches for qualitative research, warns that targeted-intercept recruitment, stopping people in public, risks selection bias. The people available and willing to stop in a public place may differ in a steady way from the people who are not. So the intercept sample leans toward them before anyone is screened.

In B2B studies, many teams use snowball sampling, where each person you interview suggests the next. That keeps the sample inside one circle of contacts.

2. Screeners and fraud checks remove some groups faster

A screener is the short set of questions that decides who qualifies. Every screener removes some groups faster than others. Strict rules shrink the pool. Open questions that reward smooth talkers favor people used to being interviewed.

Fraud checks screen out people whose answers are not genuine, whom Pew Research Center calls bogus respondents. Pew's survey work shows these checks do not fall evenly.

Pew studied opt-in samples: people who sign themselves up online, instead of being picked at random. In an August 2026 methods appendix, Pew wrote that bogus respondents in these samples disproportionately claim to be young or Hispanic. In its 2023 benchmarking study, that gave larger errors for those groups than for the population as a whole.

In a May 2026 Q&A, Pew said opt-in polls have given wrong data for young adults and for rare behaviors, such as military service.

In Pew Research Center's August 2026 test of three screening methods for online opt-in polls, trap questions and prescreening were especially likely to flag respondents claiming to be ages 18 to 29 or Hispanic. Those are the groups earlier Pew studies found most affected by bogus responding. A trap question is built to catch people who are not reading. Prescreening means questions asked before the survey starts.

So the flags may catch real fraud, or remove real people from those groups. A count of removals cannot tell you which.

The same logic holds for an interview panel. For illustration, if a bank's screener removes far more young applicants than older ones, report that split and find out why.

3. The invitation and reward decide who says yes

People choose whether your study is worth their time. That choice is called self-selection. When people answer an open call, such as a link in a newsletter, it is also called voluntary response. The invitation sets the price of saying yes.

A gift card that matters to a retail customer means little to a procurement director. So reply rates tilt by group before anyone is screened.

4. The time and format shut out busy people

A one-hour video call during office hours suits people with desk jobs and open calendars. It loses frontline staff, shift workers and senior executives. Group sessions add their own filter, covered in the post on the customers focus groups never hear from.

5. Some groups drop out before the end

Some people book a slot and never finish. A group with a large share of invites and a small share of finished interviews is the one your findings will underplay.

How an insurer could miss its broker clients

Illustrative example: a large insurer's insights team runs a study on lapses, meaning policies that customers let end at renewal. It serves two units: retail, which sells direct, and broker, which sells through brokers. The question is why people lapse and where to spend the retention budget. The answer needs both channels, and the default recruit would miss one.

Each of the team's four recruit options supports its own finding.

Recruit choice	Who it reaches	What it can show	What it cannot show
CRM list of customers who accept email	Retail buyers who opted in	Why recent retail buyers left	Anything about opted-out or broker clients
Lapse records	People who lapsed in both channels	Whether reasons differ by channel	Reasons from people who refuse contact
Customers referred by brokers	The clients brokers put forward	The reasons brokers hear	A fair picture, since brokers pick who speaks
Interviews by link, taken in the evening	People who cannot join a daytime call	Reasons from people with fixed work hours	Reasons from people who never open the link

Retention money is split by channel, so a one-channel sample cannot guide it. The team makes the lapse records its main source, cleared with the privacy team. Broker referrals stay out, because brokers would pick who speaks.

The screener had a strict rule on how recent the lapse was. The team loosens it and sets a quota, a minimum number of interviews, for older lapses. The daytime call becomes a link anyone can take in the evening.

With both channels, the team can show which reasons came from broker clients and which from retail. The budget can follow that split. If both give the same reasons, that is a finding too: one program can serve both.

How to catch sampling bias before you report

Catch sampling bias on paper first. Then watch for it during fieldwork, the weeks when interviews run. Last, check the finished sample before anyone presents. The checklist below pairs each step with a check before fieldwork and a check during fieldwork. The numbered steps then put the checks in order.

Step	What the bias looks like	Check before fieldwork	Check during fieldwork
Recruit source	A group the list misses	Write down who the list leaves out	Compare finished interviews with the list
Screener	Rules drop a group	Test it on people whose group you know	Count screen-outs by group
Fraud checks	Flags pile up in one group	Set a rule for reviewing flags	Report removals by group
Reward and format	Some groups say no more often	Fit both to each group	Track refusals by group
Who finishes	One group drops off	Set a minimum per group	Check finished interviews daily

1. Write down who your list cannot reach

Write three things in the study plan: who you want to learn about, the list you will recruit from, and the groups that list misses. A gap you name before fieldwork can still be closed with a second source.

2. Ask the stakeholder who is missing

Show your main stakeholder the list, the gaps and the minimum per group. Ask which customer they would expect to disagree with the findings. If that customer is not in the plan, add a source that reaches them.

3. Count removals, refusals and finished interviews by group

Set a minimum number of finished interviews for each group that matters to the decision. Then count screen-outs, fraud flags, refusals and finished interviews by group each day. A group that falls behind needs more invites, or another reward or format, while the study is open.

4. Read every theme by group before you share it

After fieldwork, note where the final sample differs from your list. Say so in the readout, the report that takes findings to the people deciding. Trace each theme to its quotes and note which groups they came from. A theme that rests on one group is a finding about that group.

5. Re-test a surprising finding on a new sample

Before a surprising finding shapes a decision, test it on a second sample. Pew Research Center's probability-sample re-test shows how. To check a December 2023 opt-in poll on Holocaust denial among young Americans, Pew repeated five questions from that poll on a probability-based survey of 5,140 U.S. adults conducted Jan. 16-21, 2024. In a probability-based survey, everyone has a known chance of being picked.

For an interview study, run a short round from another source. Or send a survey to a random draw from your list.

If a group that matters to the decision is missing from the finished interviews, recruit it before the readout. Until then, present only what the sample you have can support.

Where Pulse fits

Two of the five steps are about reach: who can be invited and who can take part. Pulse is built for both. It invites a company's own customers from a list, or recruits people to the profile the brief describes. People then take AI-moderated voice interviews by link, with no call to schedule.

That widens who can say yes. A shift worker or a broker's client can take the interview on a phone in the evening.

Pulse then groups what people said into themes, with every quote traced to its transcript. A researcher can see which groups each theme rests on before the readout.

To see a study traced from theme to quote, book a demo.

Questions readers ask

1. Is random sampling free of bias?

Random sampling takes the recruiter's choice out of who gets invited. It does not fix a list that leaves people out. Nor does it stop some invited people from saying no more often than others. A random draw from an incomplete list is a fair sample of the wrong group.

2. What is the most biased sampling method?

No method is the most biased in every study, since bias depends on who you want to reach. Convenience recruiting is usually the hardest to defend. That means interviewing whoever is easiest to reach, such as the first people to reply to a broad email. Nothing controls who answers.

3. How do you correct sampling bias after fieldwork?

In a survey, weighting adjusts for groups with too few or too many answers. It counts some answers more and others less, as long as each group has enough answers to start with. A small interview study is different. A missing group has no answers to count more.

So the fix is to recruit that group, run the extra interviews and report which findings rest on which groups.