Technology
Concept testing with a research agent is when an AI agent runs a concept test end to end with real target respondents. The agent drafts the study plan, screener and discussion guide, recruits respondents, moderates the interviews, checks response quality, and drafts a go, refine or no-go recommendation. The research team approves the decision rule, the study design, the full launch and the final call.
Summary:
|
Concept testing with a research agent is a way of running a concept test where an AI agent does most of the work and your team approves the decisions that matter. The agent drafts the study plan, screener and discussion guide, recruits real target respondents, moderates the interviews, checks response quality and drafts a go, refine or no-go recommendation. The team approves four gates: the decision rule, the study design, the move to full fieldwork and the final call.
Two things this is not. It is not a desk-research agent that searches the web and summarizes what it finds, the kind covered in our complete guide to AI research agents. And it is not a panel of synthetic respondents answering on behalf of real people. This is primary research with real category buyers.
The stakes are high. McKinsey research found that more than 50% of product launches fail to hit their business targets, even though new products generate more than 25% of total revenue and profits across industries. A better concept decision earlier in the process is one of the cheapest ways to improve those odds.
To make the process concrete, this guide follows one example from start to finish: a D2C beverage brand testing two positionings for the same new drink. All numbers in the example are illustrative.
What a Research Agent Does, and What the Team Approves
It helps to separate three approaches that are often confused. The difference comes down to who designs the study, who answers it and who decides.
Research agent | AI moderator | Synthetic respondents | |
Who designs the study | Agent drafts, team approves | Researcher designs | Researcher designs |
Who answers | Real respondents | Real respondents | AI-generated personas |
Who decides | Team, using the agent's draft memo | Researcher | Researcher |
Best for | End-to-end studies with human sign-off | Running interviews at scale | Early idea screening |
A research agent covers the whole study. An AI moderator covers the interviews inside it. For a deeper look at how agents work in this field, see our overview of AI agents in consumer research.
The Ten Steps and Four Gates
Step | What the agent does | What the team approves |
1. Brief intake | Restates the decision, flags gaps | The brief |
2. Action standard | Proposes go/refine/no-go criteria | Gate 1: the decision rule |
3. Study plan | Proposes design, cells, quotas and sample size | Included in Gate 2 |
4. Screener | Drafts qualifying and terminate logic | Gate 2: screener, guide and stimulus |
5. Guide and stimulus | Drafts questions and probe rules | Included in Gate 2 |
6. Recruiting | Sets quotas and tracks fill | Panel and incentive choices |
7. Pilot | Runs 5-10 interviews, reports issues | Gate 3: move to full fieldwork |
8. Fieldwork and QA | Moderates, flags and replaces bad responses | Spot checks |
9. Synthesis | Scores cells, codes themes, links quotes | Review of evidence |
10. Decision memo | Drafts the recommendation | Gate 4: the final call |
Agentic AI is still early, which is why these gates matter. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, and estimates that only about 130 of the thousands of agentic AI vendors are real. Human-in-the-loop research keeps the agent's speed while keeping accountability with people.
A Worked Example: One Concept, from Brief to Decision
The example brand is a D2C beverage company launching a lightly sparkling drink with natural botanicals. It wants to know which positioning to take to market:
Cell A, "Calm focus": a drink that helps you stay clear-headed through the afternoon.
Cell B, "Natural lift": a drink that gives you gentle energy without a crash.
Each respondent sees only one positioning. Every number below is illustrative, not from a live study.
Step 1: The Brief the Team Writes
The team writes five lines:
Decision: Choose one positioning for launch, or send both back for rework.
Concept: Same drink, two positionings (Calm focus, Natural lift).
Audience: US adults 25-45 who buy functional or better-for-you drinks at least monthly.
Deadline: Decision needed in 10 working days.
If the answer is "no": Rework the positioning and retest before the retailer pitch.
The agent restates the decision in one sentence and flags what is missing. In this case, it asked for the planned price point and whether any competitor products should be named. The team added a price of $3.49 per can and chose not to name competitors. A clear brief makes everything downstream easier, as our guide to an effective research brief for AI agents explains.
Step 2: The Action Standard, Agreed Before Fieldwork (Gate 1)
The action standard is the decision rule, agreed before anyone sees a result. The agent proposed, and the team approved:
Outcome | Criteria (all must hold for "Go") |
Go | Top-2-box purchase intent of 45% or more (the brand's norm for launches), at least 70% correct playback of the core benefit, and no single barrier theme raised by more than 25% of respondents |
Refine | Purchase intent meets the norm, but playback or barrier criteria fail |
No-go | Purchase intent below the norm |
Setting the rule first guards against hindsight bias. Once results arrive, it is tempting to move the bar to fit the winner. A rule agreed in advance stops that.
The standard also avoids relying on purchase intent alone. People often overstate how likely they are to buy, so comprehension and barriers act as checks on a high intent score. Our existing AI moderated concept testing guide covers why purchase intent should never be the only gate.
Step 3: The Study Plan the Agent Proposes
The agent proposed a monadic design: two cells, 150 respondents per cell, with quotas on age, gender and region. Monadic means each person sees only one concept, which avoids order effects and comparison bias when the two positionings are genuinely different directions.
Sample size drives confidence. Nielsen Norman Group recommends 40 participants for most quantitative studies, which gives a 15% margin of error at 95% confidence for a binary metric. A go/no-go call needs tighter numbers, so the agent proposed 150 per cell. At that size, the margin of error on a percentage is roughly plus or minus 8 points at 95% confidence, using the standard formula for a proportion near 50%.
The team approved the plan with one change: adding a quota so at least a third of each cell were current energy-drink buyers.
Step 4: The Screener (Gate 2)
The agent drafted six screener questions:
Category use: Which of these drinks have you bought in the past month? (Must select functional or better-for-you drinks.)
Buying role: Who usually buys drinks for your household? (Must be sole or shared buyer.)
Recency: When did you last buy a functional drink? (Must be within 30 days.)
Industry rejection: Do you or anyone in your household work in beverages, marketing or market research? (Terminate if yes.)
Red herring: Which of these brands have you bought? (Includes one made-up brand. Terminate if selected.)
Attention check: Please select "Sometimes" for this question. (Terminate if missed.)
Anyone who fails the category, role, recency, industry, red-herring or attention criteria is screened out immediately. The team tightened question 3 from 60 days to 30 days to focus on active buyers.
Step 5: The Discussion Guide and the Agent's Probe Rules
The agent drafted the guide in this order: first impression, playback ("In your own words, what is this drink offering?"), appeal, believability, deal-breakers, price and purchase intent. Purchase intent comes last so earlier questions do not anchor it.
It also set probe rules. Allowed probes ask for detail ("What makes you say that?", "Can you give an example?"). Banned probes lead the respondent, such as "Wouldn't you agree this sounds refreshing?" Leading prompts are one of the most common sources of bias in AI moderated research, so the rules are part of Gate 2.
A sample exchange shows the agent probing a vague answer:
Respondent: It's fine, I guess.
Agent: Thanks. What would need to be different for it to be more than fine for you?
Respondent: I'd want to know it's not just another caffeine drink. "Calm" sounds nice but I don't know what's in it.
The team approved the guide and stimulus, adding one probe on ingredients.
Step 6: Recruiting Real Respondents
There are two main recruiting routes:
Panel: Fast access to fresh category buyers across markets, with quotas filled by the panel provider.
Your own customer list: Useful for testing with existing buyers, but it skews toward people who already like the brand.
For a launch decision, the team chose a panel so the sample reflects the wider market, not just fans. The agent tracked quota fill daily and set incentives in line with interview length. Good participant recruitment for consumer research matters because the go/no-go call needs real category buyers, not people who rarely buy the product type.
Step 7: Pilot, Then Full Fieldwork (Gate 3)
The agent ran eight pilot interviews before launching the full study. It reported two issues:
Several respondents were unsure whether $3.49 was per can or per pack.
One question about "botanicals" confused people who did not know the term.
The team reworded the price question to show "$3.49 per 12 oz can" and replaced "botanicals" with "plant ingredients like ginger and lemon balm." With those fixes, the team approved full fieldwork.
Step 8: Quality Control During Fieldwork
Data quality is one of the biggest risks in online research. A Pew Research Center study found that in online opt-in samples, an average of 8% of adults answered "Yes" to at least 10 of 16 Yes/No questions, a sign of bogus respondents who answer without real effort. Checks during fieldwork are not optional.
The agent kept a QA log:
Flag reason | Responses flagged | Removed | Replaced |
Speeders | 9 | 9 | 9 |
Duplicate devices | 4 | 4 | 4 |
Answers contradicting the screener | 6 | 6 | 6 |
Pasted or AI-written answers | 7 | 7 | 7 |
Off-topic replies | 4 | 4 | 4 |
Total | 30 | 30 | 30 |
Of 330 completes, 30 were removed and replaced, leaving 300 valid interviews (150 per cell). A researcher also spot-checked 20 transcripts while fieldwork was running to confirm the flags were fair.
Step 9: Synthesis: Scores, Themes and the Evidence Trail
The agent scored each cell against the action standard:
Criterion | Standard | Cell A: Calm focus | Cell B: Natural lift |
Top-2-box purchase intent | 45% or more | 48% | 41% |
Correct playback of core benefit | 70% or more | 74% | 81% |
Top barrier theme | 25% or less | Price, 22% | Caffeine worries, 31% |
Result | Go | No-go on intent |
It then coded the top three themes across both cells, with the number of respondents who raised each:
Ingredient clarity (96 of 300): People wanted to know what makes it calming or energizing.
Price versus a regular soda (71 of 300): Most accepted $3.49 if benefits were clear.
Caffeine concerns (63 of 300): Mostly in Cell B, where "lift" suggested caffeine.
Every quote in the synthesis links back to its transcript, so the team can check context.
The difference in purchase intent between cells (7 points) is within the margin of error, so the agent did not claim Cell A beat Cell B head to head. It judged each cell against the action standard instead.
Where a platform adds emotion measurement, it can expose gaps between what people say and how they react. In an illustrative case, respondents in Cell B rated appeal highly, but facial coding showed flat reactions when the "no crash" line appeared. That kind of behavioral research for concept testing helps explain why stated scores and real interest diverge.
Step 10: The Decision Memo (Gate 4)
The agent drafted a one-page memo:
Decision: Go with Cell A, "Calm focus."
Three reasons: It meets all three criteria in the action standard. Playback shows people understand the benefit. Price was the top barrier but stayed under the limit.
Risks: Ingredient clarity is a recurring question. The intent lead over Cell B is within the margin of error.
What to test next: Pack copy that names key ingredients, and a price check at $3.29.
The team reviewed the evidence, agreed with the recommendation and signed off. The agent drafted the call. The team made it.
How to Audit the Agent Before You Trust the Call
An agent's output is only as good as its checks. Before you act on a recommendation:
Re-code 10-15% of transcripts by hand and compare your themes with the agent's.
Trace every quote in the memo back to its transcript.
Review follow-up probes for leading questions.
Check results by segment, not only the overall score.
Compare stated intent with behavior or emotion signals. The say-do gap is one of the most common reasons concept tests mislead.
Confirm no respondents were removed simply for disagreeing. Quality removals should follow the QA rules, not the results.
Where Synthetic Respondents Fit
Synthetic respondents can help shortlist ideas before a real test. They are fast and cheap for early screening, such as narrowing ten territories to two. They should never make the go/no-go call.
Our guide to synthetic audiences in consumer research explains when they help and where real respondents are essential.
When Not to Hand a Concept Test to an Agent
Agents are not the right fit for every concept test. Keep a human researcher in charge when you are dealing with:
Sensory or physical products that people need to touch, smell or taste.
Regulated claims, such as pharma or financial services, where every word needs legal review.
Very small B2B audiences where each respondent is hard to replace.
Sensitive topics that need human rapport.
Co-creation sessions where the goal is to build ideas with participants, not evaluate them.
Skipping proper testing has its own cost, as this look at the hidden cost of skimping on concept testing shows.
Adoption of agents is also still early. The Stanford AI Index 2026 reports that 88% of surveyed organizations used AI in 2025, but AI agent deployment was in the single digits across nearly all business functions. Expect the right use cases to expand as tools mature.
Running Agentic Concept Tests in Decode
Decode by Entropik offers AI research agents for consumer and user research. Each agent drafts the study for you to review and edit before anything runs, and every finding links back to its evidence.
For concept tests, Decode's concept validation agent, Cole, compares concepts on preference, comprehension and emotional response. By default, Cole uses a sequential monadic design, where each person sees every concept in random order, with face emotion measurement on each concept. Teams that need a purely monadic design, like the example above, can review and edit the study before it runs.
For interviews, the AI Moderator, Mira, runs hundreds of adaptive AI-moderated interviews at once in 70+ languages.
For recruiting, Decode reaches 103M+ participants in 120 countries through Cint and Dynata, or you can invite your own customers with a link. Every participant is checked before, during and after fieldwork, including duplicate detection, attention checks and flags for AI-generated responses, as described in the panel and recruitment guide.
As a consumer research platform, Decode also measures what people feel and notice, with 90%+ facial coding accuracy across 62 facial expressions and 96% eye tracking accuracy. It is SOC 2 Type II and ISO 27001 compliant.
That combination turns stated answers into deeper consumer insights behind each concept.
For a broader comparison of options, see our guide to the best consumer research platforms.
For more on concept testing methods in general, read our concept testing guide.
Conclusion
A research agent can take a concept test from brief to go/no-go in a fraction of the usual time, while your team keeps control at every gate that matters. Agree the action standard first, approve the study materials, check the pilot, audit the evidence and make the final call yourselves.
If you want to run agent-led concept tests with real respondents and multimodal evidence, explore Decode's AI research agents or request a demo.
Frequently Asked Questions
1. Can an AI research agent make the go/no-go decision on its own?
No. The agent drafts a recommendation based on the agreed action standard, but the team reviews the evidence and makes the final call. Keeping that decision with people is the point of the approval gates.
2. How many respondents does an agent-run concept test need?
It depends on how precise the decision needs to be. Around 40 participants gives a 15% margin of error for simple metrics, while go/no-go studies often use 100-200 respondents per cell for tighter estimates.
3. How long does a concept test take with a research agent?
Timelines depend on audience and sample size, but agents shorten design, moderation and synthesis significantly. Many studies move from brief to decision memo in days rather than weeks, with recruiting and fieldwork usually the longest steps.
4. How is a research agent different from an AI-moderated interview?
An AI moderator runs the interviews. A research agent runs the whole study, from brief and action standard through screener, recruiting, fieldwork, quality control, synthesis and the draft decision memo.
5. Can synthetic respondents replace real ones in concept testing?
No. Synthetic respondents can help shortlist ideas early, but the go/no-go decision should rest on real category buyers.
6. How do you know the respondents in an agent-run study are real?
Look for layered quality checks: screener traps, duplicate and device checks, attention checks, flags for pasted or AI-written answers, and human spot checks of transcripts. Ask the provider to show its QA log for every study.
Keep reading



