Market research is the job people most want to hand to AI, and the one where a bad answer is hardest to spot. A wrong fact in a blog post gets corrected. A wrong read on what customers want gets funded.
The good news is that the boundary here isn’t a matter of opinion. Two peer-reviewed findings map it precisely, and once you understand them you can tell in about ten seconds whether a given research question is safe to point AI at.
Five different jobs wearing one name
“Market research” covers work with wildly different exposure to AI failure. Splitting it up is most of the battle.
| Job | What it answers | AI’s role |
|---|---|---|
| Desk research | What’s already published about this market | Strong — with source verification |
| Competitor analysis | What others are doing and charging | Strong — the data is public |
| Customer understanding | Why people buy, what frustrates them | Strong on existing text, weak at inventing it |
| Concept and message testing | Which of these five ideas is worth pursuing | Useful for screening, unsafe for deciding |
| Sizing and pricing | How big is this, what will people pay | Weakest — and where the confident fabrications live |
Notice the pattern. AI is strong where the information already exists in writing and weak where it has to stand in for a human who was never asked.
The two findings that set the boundary
The academic idea behind synthetic respondents is older than the hype. Argyle and colleagues, publishing in Political Analysis in 2023, described “silicon sampling” and the property they called algorithmic fidelity — a properly conditioned model reproduces the attitude patterns of demographic groups with surprising accuracy.
Then two 2024 results defined the edges, and both are correct at the same time.
Finding one, the encouraging half. Stanford work by Park and colleagues found that agents built from rich real interviews could reproduce a specific person’s survey answers at roughly 83–86% of the reliability with which that same person reproduced their own answers two weeks later. That’s a striking result. Note the condition attached to it: built from rich real interviews with that person.
Finding two, the one that matters more. Bisbee and colleagues found that when synthetic respondents substitute for a genuine representative survey, they hit the averages but their variance collapses — and around 48% of regression coefficients deviated significantly from the human data, in some cases flipping sign (summarised here). More recent work presented at the ACM Web Conference in April 2026 continues to probe how reliably persona-conditioned models stand in for survey respondents (paper).
Put those together and you get the single most useful rule in this article.
The variance test
If your decision depends on the average, AI can help. If it depends on the spread, it can’t.
Ask what would change your mind. “What do most people think of this name” is an average question. “Is there a segment that hates this name enough to churn” is a variance question — and variance is exactly what synthetic respondents flatten.
This matters because most of the valuable findings in market research are variance findings. The segment nobody targeted. The objection that only appears in one region. The 8% who would pay triple. Averages rarely tell you anything you didn’t already suspect; disagreement is where the money is.
Three related biases are worth knowing by name. Synthetic panels are backward-looking — good at matching known patterns, poor on genuinely novel concepts, and weak on sensitive or thinly-documented topics. They skew agreeable, under-representing extreme and negative sentiment. And they tend to overstate willingness to pay, because nobody’s money is actually leaving the room.
That last one should stop anyone using AI to validate a price.
If you’re buying a synthetic research tool anyway
The industry’s own position is unusually consistent: ESOMAR, GreenBook, NIQ, Kantar and the large consultancies all frame synthetic research as something to explore and augment with, then validate against real humans before deciding.
ESOMAR’s 2025 code update explicitly distinguishes a real person from a synthetic, virtual or digitally created persona, and requires disclosure of synthetic data and AI use in research along with human oversight of the outputs. ESOMAR also publishes 20 Questions to Help Buyers of AI-Based Services, which is free, short, and a better vendor filter than any comparison page.
Two questions from it are worth asking before anything else:
- “Which validation method do you use?” The answer you want mentions Train Synthetic, Test Real, or holdout validation against a real sample. A vendor who can’t name a method is selling plausible text.
- “What is the output grounded in?” Real first-party research, behavioural data and profiles is a different product from a model told to act like a 34-year-old buyer. Both get called synthetic personas.
And a disclosure point that applies even to informal use: if a synthetic finding reaches a client, an investor or a board pack, say it’s synthetic. Presenting model output as fieldwork is the kind of shortcut that ends careers rather than projects.
Where AI genuinely earns its keep
Here’s the part the synthetic-respondent debate crowds out. Most market research a small business actually needs isn’t survey work at all, and AI is very good at the rest of it.
| Task | Why AI works here |
|---|---|
| Mining reviews and support tickets | Thousands of real customer sentences already exist. Nothing is invented — it’s summarised |
| Coding interview transcripts | Theme extraction across long qualitative data, with a human checking the code frame |
| Competitor positioning and pricing sweeps | Public information, tedious to gather, easy to verify |
| Drafting the questions | Spotting leading, double-barrelled and ambiguous wording before you field anything |
| Triaging desk research | Deciding which forty sources are worth opening |
Review mining deserves the top slot. Your competitors’ one- and two-star reviews are the cheapest customer research in existence — unprompted, specific, written by people with no incentive to be polite, and available in volume. It’s the opposite of a synthetic panel: real variance, real complaints, zero cost. The tooling for this overlaps heavily with AI social listening and sentiment analysis, and the competitor side is covered in analysing a competitor’s content strategy with AI.
The judgement of what to do with the findings stays yours. On why that boundary holds for anything multi-step, see what AI agents can and can’t do.
The market-size trap
Ask any assistant how big your market is and you will get a number. It will have a dollar figure, a growth rate, a compound annual growth rate to one decimal place, and often a plausible-sounding source.
It may be assembled from nothing. This is the highest-consequence failure mode in AI-assisted market research, because sizing numbers get copied into business plans and investor decks where nobody checks them again.
Four checks before any market figure survives:
- Name the source, then open it. Not “according to industry reports.” A specific publisher, a specific title, a specific year. If you can’t find the document, the number doesn’t exist.
- Find the methodology. Was it modelled, surveyed, or extrapolated from a related market? A figure without a stated method is marketing.
- Check the date and the geography. Stale figures and wrong-region figures are the two most common quiet errors.
- Prefer a smaller number you can defend. Bottom-up sizing from your own reachable customers beats a top-down billion-dollar figure in every conversation that matters.
The general failure pattern — a real source cited for a claim it doesn’t make, a fabricated citation, a number with false precision — is catalogued in fact-checking AI-generated content. Sizing figures are where it costs most, which is why writing a business plan with AI needs the same discipline.
A workflow with no research budget
For a small business, in order:
- Read your own data first. Support tickets, search queries on your site, sales objections, refund reasons. You already own the most relevant dataset in existence and almost nobody reads it.
- Mine competitor reviews. Pull the negative ones at volume, have AI cluster the complaints, read the clusters yourself.
- Map the competitors. Positioning, price points, what they promise, who they ignore. Public, verifiable, tedious — ideal AI work.
- Talk to ten real customers. Ten conversations beat any synthetic panel, and AI can draft the questions and code the transcripts afterwards. This step has no substitute.
- Use AI to screen, humans to decide. Narrow twenty concepts to five with AI. Test the five on people.
- Size bottom-up. Reachable customers times realistic conversion times price. Slower, smaller, defensible.
Step four is the one people skip because it’s the only one that requires courage rather than tooling. It’s also the only one that reliably produces something a competitor doesn’t already know.
If your research feeds a store, the structured-data angle in AI ecommerce tools that increase sales is the natural next step, and for research-grade search specifically, using Perplexity as a research assistant covers the sourcing workflow. The wider stack is in the best AI tools for small business owners.
Frequently asked questions
Can AI replace surveys and focus groups?
No, and the research is fairly clear about why. Synthetic respondents reproduce averages reasonably well but collapse variance, with roughly half of statistical relationships shifting significantly in one 2024 study — and variance is where most useful findings live. Industry bodies including ESOMAR frame synthetic research as exploration to be validated against real humans, not a replacement.
When are synthetic respondents actually appropriate?
Early-stage screening: narrowing many concepts, messages or copy variants down to a few worth testing properly. They’re weakest on genuinely novel concepts, sensitive topics, thinly-documented audiences and anything involving price, since willingness to pay is systematically overstated when no real money is involved.
What should I ask a synthetic research vendor?
Which validation method they use — you want to hear Train Synthetic, Test Real, or holdout validation against a real sample — and what the output is grounded in. ESOMAR’s free “20 Questions to Help Buyers of AI-Based Services” is a better filter than any vendor comparison page.
Is it safe to use AI for market sizing?
Only with verification. Sizing is where AI produces its most confident fabrications, complete with growth rates and plausible-looking sources. Open every cited document, check the methodology, date and geography, and prefer a bottom-up estimate from your own reachable customers over any top-down figure you can’t trace.
What’s the highest-return AI use in market research?
Mining real text you already have or can access — your support tickets, site search queries and competitors’ negative reviews. It’s genuine customer language at volume with nothing invented, which is the exact opposite of the synthetic-panel risk profile, and it costs nothing.
Do I have to disclose that research used AI?
If the work falls under professional research codes, yes — ESOMAR’s updated code requires disclosure of synthetic data and AI use plus human oversight of outputs. Even informally, any synthetic finding that reaches a client, investor or board should be labelled as such. Presenting model output as fieldwork is a serious credibility risk.
