Synthetic user testing is a usability method in which AI personas, often run as browser agents, attempt tasks on a website or prototype in place of real participants. It can flag obvious friction quickly, but studies show AI agents predict real user actions poorly, miss problems rooted in users’ mental models, and cannot replace usability testing with people.

Summary:
|
AI personas can now click through a checkout, fill in a form, and explain what they were thinking. That makes synthetic user testing tempting for teams that need answers before the next release. The harder question is whether those answers match what real people would do.
This guide explains how synthetic usability testing works, what independent studies show it catches and misses, and where it fits next to real participants. If you run user experience testing for a living, treat it as a fast first pass and nothing more.
What Is Synthetic User Testing?
Synthetic user testing is usability testing in which AI personas or agents attempt tasks in place of real participants. You describe a user, give the agent a task such as "find a pair of running shoes under $100," and review what it does and says.
Two forms are common:
Persona feedback on screenshots: an AI persona reviews a design and comments on what it sees.
Browser agents: an LLM-driven agent clicks through a live page or prototype, one step at a time, and logs what it tried.
Both sit within the wider practice of synthetic users in UX research, where AI stands in for people when real participants are slow or costly to reach.
Synthetic User Testing vs Synthetic Monitoring
The word synthetic has a second meaning in software engineering. There, synthetic testing and synthetic monitoring mean scripted bots that check uptime and performance. Those bots do not simulate human judgment. This article covers the UX research meaning only.
How Synthetic Usability Testing Works
Most setups follow the same sequence:
Define personas with goals, context, and experience level.
Write tasks and success criteria so the agent knows what done looks like.
Let the agent attempt each task on the live site or prototype.
Collect logs and reports once the run ends.
Typical outputs include click paths, task success, the agent's step-by-step reasoning, screen recordings, and follow-up "interviews" in which you question the agent about its choices.
A well-known example is UXAgent, a framework by Lu and colleagues that pairs an LLM agent with a browser connector and can simulate thousands of users on a website. In the authors' own evaluation, five UX researchers praised the system but raised concerns about the future of LLM-agent-assisted studies. That mix of promise and caution runs through the rest of the evidence.
Clear task scripts matter more than anything else here. A vague task gives the agent room to improvise, and improvised paths tell you little about real behavior.
Synthetic vs Real Usability Testing: Key Differences
Synthetic and real tests answer different questions. The table below compares them on the points that matter most when choosing.
Dimension | Synthetic user testing | Real usability testing |
Participants | AI personas or browser agents | Real people from your audience |
Setup time | Minutes to hours | Days to weeks |
Cost | Low per run | Higher, with recruitment and incentives |
Scale | Thousands of runs possible | Typically small samples |
Realism of task behavior | Follows patterns learned by the model | Reflects real habits and shortcuts |
Think-aloud | Generated reasoning, not true thought | Spoken thoughts in the moment |
Mental models | Guessed from training data | Revealed through behavior and comments |
Emotion and frustration | Described, not felt | Observed and measured |
Attention | Predicted at best | Tracked in the moment |
Accessibility needs | Poorly represented | Tested with assistive technology users |
Evidence value | Hypotheses and early checks | Evidence for decisions |
Agents and people also fail tasks for different reasons. An agent may stall on a rigid page structure that a person would scroll past, while a person may give up on a label the agent reads perfectly well. For a refresher on the human side of the table, see our guide on how to conduct usability testing.
Can AI Personas Run a Usability Test? What the Evidence Shows
The question breaks into three tests. Do AI agents act like real users? Do they find the same problems? Do they see what users see? Each has a separate body of evidence.
Predicting What Real Users Do Next
A 2026 ACL paper by Lu and colleagues, "Can LLM Agents Simulate Multi-Turn Human Behavior?", tested agents against 31,865 real shopping sessions containing 230,965 user actions. The task was simple to state: given a session so far, predict the user's next action.
The best prompt-only models got it right about 12% of the time. DeepSeek-R1 reached 11.86% and Claude 3.5 Sonnet v2 reached 11.69%. A model fine-tuned on real click data did better at 17.26%, which is an improvement but still leaves more than 80% of actions wrong.
The paper also described how the agents went astray. Prompt-only models bought too often, overused filters, and tended to stick with their first search keyword instead of revising it the way people do. Those habits matter for usability work, because search revision and abandonment are exactly where real friction shows up.
Agents of this kind also differ from the respondents used in surveys, a distinction explained in our comparison of AI research agents and synthetic respondents.
Finding the Same Issues as Real Participants
UX agency UX Studio compared ChatGPT feedback against real usability tests on three prototypes: a website, a compliance web app, and a children's robotics mobile app. The AI's feedback was generic and rarely specific.
It also missed interaction problems. The majority of real participants could not find the rename and delete options in one prototype, and the AI never flagged the issue, even when asked to think like a user. It also contradicted itself, praising one site's branding as appealing and then calling the same page overcrowded.
Nielsen Norman Group saw a related pattern. Its researchers found that synthetic users seem to care about everything: asked what makes an online course engaging, the synthetic user listed seven factors without ranking them. They also noted that synthetic users claimed to finish all their courses, while real learners often did not.
How Simulated Users Misjudge Task Difficulty
Simulated users also misjudge how hard tasks are. In a 2026 ACL paper, "Lost in Simulation," Seshadri and colleagues compared LLM-simulated users with real participants from the United States, India, Kenya, and Nigeria on retail tasks. The study evaluated an AI agent rather than a website, but the lesson carries over to simulated testers.
Simulated users underestimated success on the hardest tasks, where U.S. participants succeeded 30.8% of the time. They overestimated it on moderate tasks, where real success was 39.0%. Swapping only the model that played the user shifted measured success by nearly 9 percentage points.
In plain terms, a synthetic result depends on which model you picked and which tasks you ran. That is a weak foundation for a benchmark.
Reading Task Completion From Synthetic Users
Task success, time on task, and error rate are the core of usability metrics. Synthetic runs produce all three, which makes them look like human numbers. They are not.
Agents do not get tired, distracted, or confused the way people do. An agent that finishes a task in 40 seconds tells you nothing about how long a hurried shopper would take.
Use agent results to compare designs against each other. If Variant A takes the agent twice as many steps as Variant B, that is a useful signal. Do not use them to set real-world targets or to report completion rates to stakeholders.
Predicted Attention vs Real Eye Tracking
Predictive eye tracking uses models to generate attention heatmaps from a screenshot, with no participants at all. It is fast, and it can be a reasonable proxy when people view a design freely. Usability tests are task-based, which is where the proxy weakens.
A study from TU Wien measured how well a saliency model matched real eye tracking from 47 people viewing 30 data visualizations. The model's fit was AUC .738 in free viewing but dropped to about .69 to .71 once people had tasks, while agreement between real viewers in the task conditions was about .81. The test used charts, not web pages, but the direction of the result is clear.
The authors also reported that solving a task produces more consistent viewing patterns than exploring freely. Tasks change how people look, so a prediction made without a task misses the behavior a usability test cares about.
Research on web pages points the same way. A study on task-driven webpage saliency found that, given a task, human attention is strongly correlated with semantic parts of a page such as images, buttons, and input boxes.
For a usability test, that means real eye tracking matters, because tasks change where people look.
Our guide to eye tracking in usability testing explains what it reveals.
Webcam-based tracking also makes real gaze data easier to collect at scale. See the comparison of webcam vs hardware eye tracking to choose a method.
What Synthetic Users Cannot Test
Some problems only real people can surface. Three groups stand out.
Mental models and expectations: where people think a feature should be, and how they react when it is not. Frustration, trust, and hesitation belong here too.
Accessibility needs: screen readers, keyboard navigation, magnification, and people with low digital literacy. Our guide to accessibility testing covers why these need real users.
Real context of use: distractions, real accounts and data, and mobile conditions such as glare, one-handed use, and poor signal.
Products with AI features add another layer, since their behavior changes between runs. Teams in that position can read about AI usability testing for AI-powered product teams.
A synthetic pass can still help here as a pilot. It just cannot stand in for the people who will actually use the product.
When Synthetic User Testing Is Useful
Within its limits, a synthetic pass saves time. The best uses come before a real study, when the goal is to clear away obvious problems.
Pre-study checks for broken flows, dead ends, and unclear labels, alongside a heuristic evaluation by a human reviewer.
Wording checks on how copy and instructions read.
Design comparisons, where many variants are shortlisted before any go to real users.
Task piloting, to catch confusing task wording before participants see it.
Regression checks on key flows after each release.
For early design work, pair these with prototype testing once a shortlist exists.
Synthetic runs are cheap enough to repeat after every release, which is their main advantage over real studies.
Those repeated checks fit well with unmoderated usability testing, where steady, scripted tasks are the norm.
A Hybrid Workflow: Synthetic Pass, Then Real Participants
The most defensible approach combines both. Run the synthetic pass first, fix what it finds, then bring in real people.
Run a synthetic pass on the current design.
Fix obvious issues such as dead ends and unclear labels.
Run AI-moderated or moderated tests with real users.
Compare what each method found.
Record the gaps, meaning issues only real users found.
Keep that record. Over time it shows which kinds of problems your synthetic passes miss and calibrates how much weight to give them. For the real-user stage, our guide to AI-moderated usability testing shows how to run interviews and tasks at scale.
Comparing the two sets of findings also follows the logic in our guide to synthetic data vs real data, where simulated results are always checked against field evidence.
Running a Synthetic Usability Test Responsibly
A few habits keep a synthetic test honest.
Write clear tasks and success criteria so agents are not improvising.
Base personas on real user research. Stanford researchers found that agents built from two-hour interviews with 1,052 people matched survey answers 85% as accurately as the participants matched their own answers two weeks later, and beat agents built from demographics alone.
Run each task many times and across different models, and record the model version.
Label every output as synthetic.
Never report agent task success as real user performance.
These steps matter because of the model-to-model swings described earlier. A result you cannot reproduce is a result you cannot defend.
Measuring What Real Users Do, See, and Feel in a Usability Test
Real participants show where they look during a task, where they hesitate, and how they react. A synthetic pass can only guess at these.
Facial emotion AI captures frustration and confusion as they happen, with 90%+ facial coding accuracy at Decode.
Pair it with eye gaze tracking, which Decode reports at 96% accuracy, and you can see both what people look at and how they feel about it.
Together, these signals show which friction points matter and which only look like problems on a log. Decode supports 70+ languages, so the same test can run across markets.
A user testing software platform that combines these signals lets you validate a synthetic pass in one place.
If you are weighing vendors, our guide to user experience testing platforms compares the main options.
Frequently Asked Questions
1. What is synthetic user testing?
Synthetic user testing is a usability method in which AI personas, often run as browser agents, attempt tasks on a website or prototype in place of real participants. It flags obvious friction quickly but does not replace testing with people.
2. Can AI personas replace usability testing with real users?
No. Studies show AI agents predict real user actions poorly, miss problems rooted in mental models, and misjudge task difficulty. They work best as a first pass before real participants.
3. How accurate are AI agents at predicting what users will do on a website?
Not very. On 31,865 real shopping sessions, the best prompt-only models predicted the next action correctly about 12% of the time, and a fine-tuned model reached 17.26%.
4. What is the difference between synthetic user testing and AI-moderated usability testing?
Synthetic user testing replaces the participants with AI agents. AI-moderated usability testing keeps real participants and uses AI to run the session, ask follow-up questions, and analyze the results.
5. Is predictive eye tracking as accurate as real eye tracking?
Not for task-based viewing. In one study, a saliency model's fit fell from AUC .738 in free viewing to about .69 to .71 during tasks, while agreement between real viewers was about .81.
6. What can synthetic user testing catch before a real usability study?
It can catch broken flows, dead ends, unclear labels, and confusing task wording. It can also help shortlist design variants before they reach real users.
7. How do you measure task completion with synthetic users?
Run each task many times across models, then compare designs against each other. Do not treat the numbers as human benchmarks, because agents do not tire, get distracted, or misread the way people do.
8. Which usability problems do synthetic users miss?
They miss problems rooted in mental models, frustration, trust, and hesitation, along with accessibility needs and real context of use. They also tend to miss interaction problems that real participants hit, such as options users cannot find.
Confirm Synthetic Findings With Real Users
A synthetic pass can clear the obvious problems. Real users show whether the fixes work, where they look, and how they feel.
Decode is trusted by 150+ global brands and backed by 17 patents, with 96% eye tracking accuracy, 90%+ facial coding accuracy, and 70+ languages supported.


