Skip to content

How to evaluate an AI moderator

By Khyati Ojha · · 10 min read

How to evaluate an AI moderator: what it does in an interview, how to brief it, and how to read its first transcripts before the full study runs.

How to evaluate an AI moderator

The first interviews finished overnight. You work at a meal-kit company, and an AI moderator spoke with trial customers who cancelled early. Now you're reading what it asked. Did it ask the right follow-up questions?

An AI moderator is software that runs a research interview by voice, video or chat. It asks the planned questions and adds follow-ups based on each answer.

How to evaluate an AI moderator comes down to two things. Give it a clear brief. Then read its first transcripts against a short checklist. Fix the brief where it slipped. Only then open the study to all.

Most of this work happens before the full study starts. That's when a change costs least.

The short version

  • In research, an AI moderator runs interviews. In social media, "AI moderation" means removing posts, which is a different job.
  • A good AI moderator asks for a real example when an answer is vague, and never hints at the answer.
  • Write the brief for software. Say what you want to learn, and which words get a follow-up.
  • Run a small first batch, and read each transcript before you open the study to all.
  • Before you choose a tool, ask for a sample transcript and check it with the same list.

What is an AI moderator?

Say the meal-kit team wants to know why trial customers cancel early. The AI talks to each of them when it suits them. It follows the same guide every time. It asks follow-up questions when an answer needs more detail. Each interview ends as a transcript, the full written record of what was said.

Search for "AI moderator" and you'll find a second meaning too. So it helps to tell the two apart.

Meaning What it moderates Who uses it What it does
AI moderator in research A one-to-one talk with a customer Research and product teams Asks questions and follows up
AI content moderation Posts, comments and images Social networks Removes posts that break the rules

Content moderation works at a very different size. The European Commission reports that platforms logged more than 9 billion moderation decisions in the first half of 2025. Of those, 99% were made under the platforms' own rules. ITIF also looked at one day of the EU's records, 31 July 2024. Machines did 97% of the flagging. Everything below is about the research meaning.

Inside one interview, the AI goes through the same parts a good human would. It opens by saying who is running the study, and asks for consent. A short warm-up follows, such as "How did you first hear about the meal kits?" Then comes the main question: "Tell me about the week you decided to cancel."

The follow-ups are where an interview does its real work. Three moves matter most.

1. It asks for a real example when an answer is vague

A customer says, "It got expensive." A good follow-up asks about the last real moment: "Think about the last box you got. What made it feel expensive that week?" A real moment is easier to act on than a general feeling.

2. It climbs from a feature to why it mattered

Someone says, "The recipes took too long." The next question asks why that mattered: "What was going on the evenings you cooked them?" This is called laddering. It means moving step by step from what someone did to why it mattered to them.

3. It repeats a strong word back

When someone says "annoying," the AI can say it back: "You said annoying. What happened?" People tend to explain more when they hear their own word.

A good AI also never does some things. It never leads, as in "Was it the price?" That hands the person an answer. It never asks two questions at once. And it stops digging once a point is answered.

When the interview ends, the team gets a transcript and a recording. On video, some tools also pick up tone of voice and facial expressions. To see a full interview from start to end, walk through a full AI interview.

Flow of one interview from consent, warm-up and core question to follow-ups and transcript, with three follow-up moves: ask for the last real moment, ask why it mattered, and repeat a strong word back
The parts of one AI-moderated interview, and three follow-up moves that turn a short answer into a usable one.

AI or human moderator: which should you use?

It comes down to how many talks you need, and what kind. Say you need dozens of talks this week, in a few languages, with the same questions each time. An AI moderator fits well. But say your team needs people to build ideas together in a room. Then a live session run by your own researchers fits better.

Cost and speed are part of the choice too. An AI moderator runs many interviews at the same time, with no calls to book. So a study can end much sooner. Its cost grows with the number of interviews, how long each one is and how people are found and paid.

This table starts from common situations and says which kind of interview tends to fit.

Situation Better fit Why
Many interviews needed in a week AI moderator Interviews run at once, with no calls to book
Many countries and languages AI moderator All hear the same questions from one guide
Feedback on an idea, a price or an ad AI moderator Follow-ups dig into each reaction
Health, debt and other touchy topics Either, with care Plan how people can stop and where to get help
No clear question yet A few talks led by your team first Shape the question, then let an AI ask it widely
People building on each other's ideas A live group run by your team The value is in how people react to each other

Research on content moderation gives a useful comparison. A 2024 study of AI moderation tools found they were more accurate than older methods. But a 2024 paper argues accuracy alone is the wrong test. Easy cases need to be right and fast. Hard cases need a clear reason behind each call. The same idea works for interviews: match the kind of moderator to the kind of talk.

For more on the trade-offs, read about AI-moderated versus human-moderated interviews.

How to brief an AI moderator

An AI moderator sticks to the brief. So the brief has to spell out what you want to learn, and how to react to each answer. The researcher writes or approves the brief. They check each question before launch, and they decide what the findings mean.

A good brief has six parts. Each one tells the software something a person would just guess.

  1. What you want to learn, in one sentence.
  2. The questions it must ask, in order.
  3. Follow-up triggers: vague words, strong feelings and answers that contradict each other.
  4. Words to avoid, such as leading phrases or brand names.
  5. A rough time limit for each part of the interview.
  6. The screener, a short quiz that decides who takes part.

Most of this is already in a good discussion guide, the list of questions an interview works through. The difference is that you write down the follow-up triggers and the words to avoid. A human interviewer would keep those in their head.

Here's one meal-kit question written two ways.

A weak version: "Why did you cancel? Was it the price or the recipes?" It asks two questions at once. It also offers two answers, so most people just pick one.

A strong version: "Tell me about the week you decided to cancel. What was going on?" Then add a follow-up trigger to the brief: "If they say expensive, ask what they compared the price with." The question stays open. And the follow-up is ready when the vague word shows up.

How to evaluate an AI moderator's work

You'd listen to a new interviewer's first few talks before trusting them with a full study. Do the same for an AI. Run a small first batch. Read each transcript against a checklist. Trace the first themes back to quotes. Then fix the brief before you open the study to all.

1. Read the first five transcripts against a checklist

Each row starts with something you might see in a transcript. The last column says what to change in the brief.

What you see What went wrong What to change in the brief
"Was it the price?" A leading follow-up Add it to words to avoid; ask "What made you stop?" instead
"It got expensive," then a new topic A vague answer let through Add "expensive" as a trigger
Three follow-ups on one answered point Too much digging Set a limit for that topic
A long talk about delivery times Drifting off the guide Mark which topics are in
The same question asked twice Repeating itself Look for two questions that ask the same thing
"I loved it," then "I cancelled" A clash left alone Add clashing answers as a trigger

Say the first meal-kit transcripts let "too expensive" pass again and again, with no follow-up. You add it as a trigger and run five more. Now the answers may explain what "expensive" meant. For one person, it's the price per meal compared with a takeaway. For another, it's paying for boxes they forgot to skip. Those need two different fixes. The first batch couldn't tell them apart.

2. Trace each theme back to its quotes

A theme is an idea that shows up in many interviews. For example: "cancelled after a box came in a busy week." Before anyone acts on it, open the quotes and the transcripts they came from. If you can't find clear words behind a theme, it needs more proof. This guide on how to audit each theme back to the exact quotes goes step by step.

3. Check the selection criteria for any tool

The checklist also works when you compare tools. Ask each company for a sample transcript and read it the same way.

Then check the selection criteria for an AI moderation platform. The first is the signals it reads: verbal signals (the words), tonal signals (the tone of voice) and expression signals (faces on video). The others are the languages it runs in, the interview modes it offers and how it recruits people.

Security, privacy and compliance matter most for enterprise buyers. Ask about single sign-on, data residency and GDPR. And ask whether your data is used to train AI models. The questions below cover each point.

What to check Question to ask
Follow-up quality Can we read a full sample transcript, start to finish?
Interview types Can people answer by voice, video, text chat, phone or a messaging app?
Languages Which languages can one study run in at the same time?
Signals beyond words Does it pick up tone of voice or faces? How is consent handled?
Finding people Can we invite our own customers and also find new people? Are there fraud checks?
Themes you can trace Does every theme link to its quotes and transcripts?
Security and privacy Do you offer single sign-on and data residency? Are you GDPR compliant? Do you train AI models on our data?

Single sign-on lets people log in with their work account. Data residency means choosing the region where your data is stored.

For more on choosing a tool, see this guide to choosing an AI-moderated research platform for enterprise. There's also a list of what your legal team will ask.

Four steps in a row, write the brief, run five first, read each transcript and trace three themes, then two outcomes: open the study if only small fixes are needed, or fix the brief and run five more
Check a small first batch, then decide whether to open the study or fix the brief.

So here's the rule. If five transcripts pass with only small fixes to the brief, open the study to all. If the same slip shows up in most of them, fix the brief and run five more first.

Where Pulse fits

Pulse runs AI-moderated interviews like the meal-kit study above. You set the research question, the topics, the audience and the screener. The interview is built from that brief, and your researcher checks and edits every question before launch.

You can recruit from more than 200 million people around the world, with fraud checks on every person. Or you can invite your own trial customers by uploaded list, link, QR code or WhatsApp. People answer by voice, video, text chat, phone call or WhatsApp, in 240+ languages on request.

The AI asks each person the same clear questions. It builds follow-ups on what people said earlier, and asks a second question when an answer is vague. On video, facial coding picks up expressions. On voice, tonality analysis picks up tone.

Findings come back as themes, each with the quotes behind it, and every quote links to its transcript. The report builds as interviews finish and is ready minutes after the last one. Every study stays in a library you can search across projects.

It is GDPR compliant and offers SAML and OIDC single sign-on on request. Data residency options come on the enterprise plan, and customer data is never used to train AI models.

To see it on your own study, book a demo, start on the $0 Explorer plan, or ask for a free study.

Questions readers ask

1. How much does an AI-moderated interview study cost?

It depends on how many interviews you run and how long each one is. It also depends on how people are found and paid, how many languages you need and which plan you're on. Ask each company to price the same study, so you compare like with like.

2. Which AI interview software is built for customer research, not hiring?

Look at what the software makes. Tools built for hiring score job candidates. Tools built for customer research find people through a screener. They ask follow-up questions about what people did, and turn the answers into themes with quotes. If a demo talks about candidates and job fit, it's a hiring tool.

3. Do people know they're talking to an AI moderator?

They should, and a good study tells them at the start. The consent step says who is running the research, and that an AI will ask the questions. It also says how the recording will be used and how to stop at any time.

4. Can one AI moderator run a study in several countries?

Yes. Each person hears the same questions from one guide, in their own language. So differences between countries come from the customers, not from how each interviewer asked. Have someone who knows each country read a few transcripts in that language before you compare results.

Book a demo

Book a Demo

Your details

By clicking “Book Now”, you agree Pulse may use your information as described in our Privacy Notice, including to contact you about our products and services.