When a concept test says yes and the launch still fails

Here's a familiar story, told as an illustrative example. Your team's new product idea scores well in its concept test, the steering committee says go, and a few months after launch hardly anyone is buying. Concept test results record what people say when a yes costs them nothing. So treat a strong score as a ceiling on demand, and check what buyers would pay and stop using before you decide.

Below, you'll find why that gap opens, six checks to run before a go decision, and what to do if a launch has already missed.

The short version

• A yes in a concept test is free to give, so the score is the most demand you can hope for.
• Five gaps push scores up: no cost, a polished description, no rival in view, the wrong people answering, and a product or market that moved.
• Say go when top answers come with a real use, a named replacement and a price the buyer has seen.
• If a launch has already missed, interview the people who passed and find which assumption broke.

What a concept test really measures

A concept test shows people a description of a product and asks how likely they are to buy it. A launch asks for much more. People have to find it, pay for it, set it up and stop using what they had.

It's a bit like asking friends whether they'd come to your party. Saying yes over coffee is easy. Turning up on a rainy Tuesday is a different decision.

Many concept tests use a five-point purchase-intent scale. That's a question with answers running from "definitely would buy" to "definitely would not buy". The headline number is often the top-box score, which is the share of people who picked the top answer. Some teams add the top two answers together.

Either way, the number sums up a reaction on a day when nothing was asked of anyone. Read it as a forecast and the cost lands later, when build and launch budgets go to a product the market didn't want in that form.

Why a yes doesn't turn into a sale

A yes in a concept test is easy to give, and five gaps push the score above what the market will do.

Saying yes costs nothing

A purchase-intent question asks for no money, no time and no risk. Ask the same person to pay and to spend an afternoon on setup, and you're asking a different question.

The description beats the product

The concept board is the description or mock-up people react to. It shows the benefit and leaves out the price, the setup steps and the flaws. The product that launches has all three.

There's no rival in view

In the market, people weigh a new product against what they use now, or against doing nothing. Most concept tests leave that choice out, and how much they leave out depends on the test design.

The wrong people answered

Panel members, meaning people who sign up to take surveys, are often the easiest to recruit. So are fans of the category. If the people who pay, approve or switch are a small part of the sample, you get a precise reading of the wrong group.

The product or the market moved

What launches often differs from what was tested. A feature gets cut, the price goes up or the sales channel changes. The need can shift too. When that happens, the old score describes a different product.

Four test designs and the blind spot of each

The four common concept test designs are monadic, sequential, proto-monadic and comparison. Each one answers a slightly different question, and each has a blind spot that can lift the score above what buyers will do.

Test design	How it works	Its blind spot
Monadic	Each person sees one idea on its own and rates it	Nothing to compare against, so the idea is never weighed against what the buyer uses today
Sequential	Each person sees several ideas, one after another	Later ideas are judged against earlier ones in the test, not against the market, and the order shifts the scores
Proto-monadic	One idea alone first, then a choice between ideas	The choice is only between new ideas, so the current tool and doing nothing are still missing
Comparison	The idea sits beside another option and people pick one	Closest to a real choice, but only if the other option is what buyers actually use today

A comparison test against today's option comes closest to the decision a buyer makes after launch. The guide to how concept tests are designed covers how to set up each format.

Bigger samples won't close the gap

A bigger sample makes the same skewed number more precise. A z-test between two concept scores checks whether the difference between them is more than chance. It can show that one idea beats another and still rank them on the wrong question.

More scale points or a stricter top-box cut change the number without changing what you asked. Retesting with the same panel repeats the same gaps. All of these measure what people say more sharply. None of them measures what people do.

Reading concept test results before you say go

Before you say go, test the yes. The six checks below turn a score into evidence a steering committee can weigh against the forecast.

1. Read the reasons behind each top answer

A top answer with no concrete use is a soft yes. For illustration, here's a soft yes beside a firm one:

Soft yes	Firm yes
"It looks useful."	"I'd use it for month-end reporting with my two analysts."

Code the reasons, which means sorting each answer into a labelled group. Then count how many name a real use.

2. Ask what they'd stop using

Ask which product, process or budget the new one would replace. Someone who can't name one hasn't pictured switching yet.

For illustration, "How likely are you to buy this?" invites a polite yes. "What would you stop paying for to make room for this?" asks the person to picture the switch.

3. Ask when they last bought something similar

Ask when they last bought or tried something like it, and what happened next. What people did before is firmer ground than what they say they'll do.

4. Show the price and the effort

Put the price and the setup effort into the concept, or test them as a separate step before launch. An idea that keeps its score with its costs in view is a stronger bet.

5. Read the real buyers first

Read the results for the segment that matches the real buyer before you read the total. A segment is a group of respondents who share a trait, such as their job or what they buy today. Then read the people who chose the middle answers. Their doubts are the ones the top box hides.

6. Score a known concept alongside

Include a past concept whose launch you know, or a benchmark concept. A score means more next to one your team has already seen play out in the market.

Five patterns that should make you pause

These patterns in concept test results hint at a gap between the test and the market. None proves the launch will miss, but each tells you which check to run first. Use the diagnostic table below when the results come back.

Pattern in the results	What it often signals	What to check before launch
High intent, vague reasons	People like the idea but have no use for it yet	Code the reasons and count concrete uses
High intent, nothing named to replace	Interest without a switch	Ask what it replaces and which budget pays
Strong in the total, weak among category buyers	The wrong people carried the score	Read by buyer segment and recruit more real buyers
People like it but can't say what it does	Appeal without understanding	Fix the description and test understanding again
Strong score on a concept shown without a price	Demand measured without its main cost	Test with the price and effort in view

The table has no cut-offs or benchmark figures on purpose. A threshold borrowed from another category's tests would give a false sense of precision. Your own past concepts, with launches you know, are the reference point to use.

Here's a simple go rule. Say go when most top answers come with a real use, a named replacement and a price the buyer has seen. If any of the three is missing, treat the score as a ceiling and size the forecast to match.

When a well-tested feature goes unused

Illustrative example: a large retail bank tests a budgeting feature for its mobile app. The top-box score comes back high, the steering committee approves it, and after launch few customers turn it on. The team must decide whether to fix the feature, promote it harder or stop spending on it.

The team has four ways to look into it.

Option	What it can tell the team	What it can't tell the team
Re-run the concept test with a bigger sample	Whether the stated interest still holds	Why customers didn't turn the feature on
Read the app's usage data	Where people stopped, if they started	Why the rest never started
Survey customers who saw the launch	How many pick each listed reason	Reasons the team didn't think to list
Interview customers who saw it and didn't use it	Which assumption broke, in their own words	How common each reason is across all customers

The team needs the reason before it can pick a fix, so the interviews come first. It invites three groups:

• customers who opened the feature once and stopped
• customers who saw the launch message and passed
• heavy app users who ignored it

Each interview walks through the moment the person decided, and the concept board stays out until the end. That way, people describe their own decision instead of reacting to the idea again.

Next, the team marks each assumption behind the concept as held, broken or untested. The list covers the need, the price, the switch, the channel and the product as built. It shares the list with the committee that approved the launch and stores it where the next concept brief will start.

The next decision depends on which assumption broke. Suppose many customers say they already budget in a spreadsheet they trust. Then the switch broke, and a bigger campaign won't fix it. A better next step is a version that imports the spreadsheet, plus a survey to size how many customers budget that way. If most people never saw the feature, the channel broke, and promotion is the right fix.

Where Pulse fits after a launch misses

Pulse runs AI-moderated voice interviews with your own customers or with recruited participants. Each interview follows the same guide and asks a follow-up when an answer is vague. Each theme links to the quotes behind it. For the bank, that means every broken assumption arrives with the customer words that show it.

Each study is kept in a searchable insight repository, so the next concept brief can start from what this launch taught. Your researcher still decides which assumptions to test and what the committee should do next. Pulse runs the interviews and the first pass at the themes, so their time goes to the decision.

To see how a post-launch study like this runs, book a demo.

Questions readers ask

1. When does concept testing make sense?

Run a concept test before you commit build or media budget, and use it to choose between ideas. It works best for ranking ideas against each other and for collecting the reasons behind people's reactions. Treat the score as a ceiling on demand. Add behavior and replacement questions before it feeds a forecast.

2. Can you trust purchase-intent questions?

Trust them more for ranking ideas than for forecasting demand. A question like "If the product were available today, how likely would you be to buy it?" measures interest that costs nothing to state. The answers get more useful beside what the person would stop using, when they last bought something similar, and their reasons in their own words.

3. When should you interview people who passed?

Plan those interviews before launch and run them soon after, while people still remember the moment they decided. The longer you wait, the more people rebuild their reasons from memory, and a tidier story is less useful to you. Recruit from the groups the concept test said would buy.