Technology
Sampling bias occurs when the selection of research participants systematically favors certain members of a population, resulting in an unrepresentative sample. Common types include undercoverage bias, self-selection bias, convenience sampling bias, and survivorship bias. Researchers can reduce sampling bias through appropriate sampling frames, random selection, stratified sampling, and monitoring participant representation and nonresponse.
Summary:
|
A survey can have thousands of respondents, clean data and sharp charts, and still be wrong. If the people in the sample differ systematically from the people you want to understand, every conclusion inherits that distortion.
That is the problem of sampling bias. It affects public opinion polls, clinical trials, customer satisfaction surveys, UX studies and the datasets used to train AI models. This guide explains what sampling bias is, what causes it, the most common types, how it differs from related biases and how to avoid it.
What Is Sampling Bias?
Sampling bias is a systematic distortion caused by the way participants or observations are selected for a study. It happens when some members of the target population are more or less likely to be included than others, so the sample no longer represents the population.
The result is an unrepresentative sample. Any estimate drawn from it, such as average satisfaction, purchase intent or brand awareness, can be consistently too high or too low.
It helps to separate sampling bias from random sampling variation. Every sample differs slightly from the population by chance, and those differences shrink as samples get larger. Sampling bias is different because it pushes results in the same direction every time. Collecting more data from the same flawed process simply gives you a more precise wrong answer.
What Causes Sampling Bias in Research?
Sampling bias usually comes from the design of the study rather than from participants themselves. The most common sources of bias in sampling are:
Incomplete sampling frames: The list used to select participants leaves out part of the population.
Nonrandom recruitment: Participants are chosen because they are easy to reach, not because they are representative.
Unequal inclusion probabilities: Some groups have a higher chance of selection than others, and the analysis does not correct for it.
Self-selection: People decide for themselves whether to take part.
Inaccessible groups: Some people are hard to reach through the chosen channels.
Differential nonresponse: Certain groups are less likely to respond once invited.
Recruitment procedures matter as much as sample size. Good participant recruitment for consumer research starts with a clear definition of who should be in the study and how they will be reached.
Common Types of Sampling Bias
Each type of sampling bias arises in a different way, but all of them make the sample less representative.
1. Undercoverage Bias
Undercoverage bias occurs when some members of the target population are excluded or inadequately represented by the sampling frame. They never had a chance to be selected.
The classic example is an online-only survey that excludes people without internet access. That group is still large. According to ITU's Facts and Figures 2026, 2 billion people remain offline, and internet use stands at 93% in high-income countries compared with just 32% in low-income countries. A study of a global or lower-income market that relies only on online recruitment will miss many people it claims to describe.
How to prevent it: Improve the sampling frame and use multiple recruitment channels, such as phone, in-person and mobile, to reach groups that a single channel misses.
2. Self-Selection Bias
Self-selection bias, also called volunteer bias, occurs when people who choose to participate differ systematically from those who do not. Volunteers often have stronger opinions, more free time or more interest in the topic.
Voluntary online polls and open customer feedback forms are common examples. The people who bother to respond are often the happiest or the angriest customers, not the typical ones.
How to prevent it: Use controlled recruitment and probability sampling where possible, and monitor who is participating so you can spot imbalances early.
3. Convenience Sampling Bias
Convenience sampling bias occurs when participants are selected mainly because they are easy to access. Examples include surveying only nearby customers, students on a single campus or a company's own employees.
Convenience sampling is fast and cheap, which makes it useful for early exploration. The risk is limited representation, which makes findings hard to generalize to the wider population.
4. Survivorship Bias
Survivorship bias occurs when analysis focuses only on observations that remain visible after a selection process, while ignoring those that did not make it through.
A common business example is studying only retained customers to understand loyalty while excluding customers who have already churned. The people who left are often the ones with the most useful information about what went wrong.
How to prevent it: Include relevant discontinued, unsuccessful or missing cases, such as churned customers, failed products or abandoned journeys.
5. Nonresponse Bias
Nonresponse bias occurs when people who respond differ from those who do not in ways that matter to the research outcome. For example, dissatisfied customers may be less likely to complete a satisfaction survey, which inflates the results.
Low response rates make this risk harder to ignore. Pew Research Center reported that typical response rates for its telephone surveys fell to 7% in 2017 and 6% in 2018, after holding around 9% for several years. A low response rate does not automatically mean a biased result, but it means the people who answer may differ from the people who do not.
How to prevent it: Send follow-ups, monitor response rates by subgroup and apply appropriate statistical adjustments such as weighting.
6. Sampling Frame Bias
Sampling frame bias, also called coverage bias, occurs when the list used to draw a sample is inaccurate or incomplete. Outdated customer databases, missing contact details and incomplete geographic coverage are common causes.
How to prevent it: Validate the frame before fieldwork, update databases regularly and check population coverage against known figures.
Undercoverage Bias vs Nonresponse Bias
Undercoverage and nonresponse are often confused because both leave groups out of the final data. The difference is when the exclusion happens.
Undercoverage bias | Nonresponse bias | |
When it happens | Before selection | After selection |
Cause | Eligible people are missing from the sampling frame | Selected people do not provide usable responses |
Example | Online survey excludes people without internet access | Unhappy customers ignore a satisfaction survey invitation |
How to identify | Compare the frame with population data | Compare respondents with nonrespondents and track response rates |
How to correct | Improve the frame and add recruitment channels | Follow-ups, incentives and weighting |
Sampling Bias vs Undercoverage Bias
Sampling bias is the broader problem of systematic sample distortion. Undercoverage bias is one specific mechanism that can cause it.
Consider an online survey about grocery habits that excludes people who do not use the internet. The undercoverage is the missing offline segment. The sampling bias is the overall result: estimates that reflect more digitally active shoppers than the real population.
Sampling Bias vs Selection Bias
The two terms overlap, and they are often used interchangeably.
Sampling bias concerns systematic distortion in who ends up in a research sample.
Selection bias is a broader concept that covers any systematic inclusion or exclusion of observations, including in observational studies, experiments and data analysis.
In participant recruitment, they describe much the same problem. Selection bias can also arise later, for example when researchers exclude certain cases during analysis. This guide to selection bias covers the wider concept in more detail.
Sampling Bias vs Response Bias
Sampling bias affects who is included in the sample. Response bias affects how accurately participants answer.
An unrepresentative recruitment process that over-samples loyal customers is sampling bias. A representative sample that gives socially desirable answers, such as overstating healthy habits or recycling, has response bias. Cultural norms can also shape answers, a challenge explored in this guide to cultural response bias.
A study can suffer from both at once, which is why sample design and question design both need attention. Many of the same mental shortcuts appear in product research too, as this overview of cognitive biases in user research shows.
Sampling Bias vs Sampling Error
Sampling bias is systematic. Sampling error is the random difference between a sample and the population that arises by chance.
Increasing sample size reduces random sampling error, but it does not eliminate systematic bias. A biased process at 10,000 respondents is still biased. The answer is sample design: probability sampling gives every member of the population a known chance of selection, which is the foundation for unbiased estimates.
Real-World Examples of Sampling Bias
Market Research
A brand that surveys only members of its loyalty program to estimate overall brand satisfaction will almost certainly overstate it. Loyal customers are, by definition, the ones most likely to be satisfied.
UX Research
A product team that tests a redesign only with experienced power users may miss the problems new users face. The sample should reflect the full range of people who use, or will use, the product.
Healthcare
Clinical trials are a high-stakes example. A study of 341 pivotal trials supporting US FDA drug approvals between 2017 and 2023 found that only 6% enrolled participants in line with the distribution of the four largest racial and ethnic groups in the US population. When trial samples do not match patient populations, differences in how treatments work across groups can go undetected.
Public Opinion
Online polls are especially vulnerable. A Pew Research Center study found that online opt-in samples averaged 5.8 percentage points of error across 28 benchmarks, about twice the 2.6 points of probability-based panels. Errors were even larger for 18- to 29-year-olds and Hispanic adults.
How to Avoid Sampling Bias in Research
You cannot remove every source of bias, but you can reduce it significantly with good design.
Define the target population clearly. Be specific about who the research should represent.
Use an appropriate sampling frame. Check that the list or panel covers the whole population.
Apply probability sampling. Random selection gives every member a known chance of inclusion.
Stratify where it matters. Divide the population into meaningful groups and sample within each.
Use suitable recruitment methods. Combine channels to reach groups that one channel misses.
Set quotas carefully. Quota sampling can balance key groups when probability sampling is not practical.
Monitor representation and nonresponse. Track demographics and response rates throughout fieldwork, not just at the end.
How Stratified Sampling Helps Reduce Sampling Bias
Stratified sampling divides a population into relevant strata based on demographic, geographic or behavioral characteristics, then randomly selects participants within each stratum. This guarantees that each important subgroup is represented in planned proportions.
For example, a brand studying shoppers across regions might stratify by region and age group, then sample randomly within each cell. The benefits of stratified sampling include more precise estimates and reliable subgroup comparisons.
Stratification has limits. It cannot fix an incomplete frame, it depends on choosing the right strata and it does not prevent nonresponse within each stratum.
How to Identify Sampling Bias in Survey Results
Even with a careful design, it is worth checking results for signs of bias.
Compare against benchmarks: Check sample demographics against census data or other known population figures.
Analyze response rates: Look at response rates by subgroup and recruitment channel.
Check subgroup representation: Look for groups that are much smaller or larger than expected.
Apply weighting: Adjust results so the sample matches known population proportions.
Run sensitivity analysis: Test how much conclusions change under different weighting assumptions.
Weighting helps, but it cannot fully correct for groups that are missing entirely. Report remaining uncertainty honestly.
What Is Sampling Bias in AI and Machine Learning?
Sampling bias also affects AI. When training or evaluation datasets do not represent the population a model will serve, performance can vary sharply across groups.
An MIT study of three commercial facial-analysis programs found gender classification error rates of 0.8% for light-skinned men but up to 34.7% for dark-skinned women. One benchmark dataset used to assess a system's performance was more than 77% male and more than 83% white.
Historical data collection limits create similar problems, because past data often reflects who was easiest to measure. Representative data collection, subgroup evaluation and regular dataset audits help reduce the risk. The same principles apply to newer research methods, as this guide to bias in AI moderated research explains.
Impact of Sampling Bias on Market and Consumer Research
In market and consumer research, sampling bias can distort the metrics that drive major decisions.
Customer satisfaction: Over-sampling engaged customers inflates scores and hides churn risks.
Brand perception: Recruiting only existing buyers misses how non-customers see the brand.
Advertising research: Testing ads with an unrepresentative audience can make weak creative look strong, or the reverse.
The risk is drawing broad conclusions from narrowly recruited groups. Reliable consumer insights depend on samples that reflect the market you actually want to win.
This applies to behavioral research too. When teams measure emotional response to advertising with facial coding or eye tracking, the findings are only as representative as the people being measured.
This is where a consumer research platform helps. It can combine targeted recruitment with surveys, interviews and behavioral measurement, so teams study both what consumers say and how they respond.
Decode supports research in 70+ languages and is used by 150+ global brands.
Its panel integrations help teams reach the audiences their studies need.
For a wider comparison of options, see our guide to the best consumer research platforms.
Conclusion
Sampling bias is one of the most common and least visible threats to research quality. It does not announce itself in the data, and a larger sample will not fix it.
The best defense is good design: a clear target population, a complete sampling frame, probability or stratified sampling, careful recruitment and ongoing monitoring of who is, and is not, in your data. If you want to study consumer feedback and behavioral responses with the right audiences, explore Decode or talk to an expert.
Frequently Asked Questions
1. What is sampling bias in research?
Sampling bias is a systematic error that occurs when the method used to select participants makes some members of the population more or less likely to be included. The resulting sample does not represent the population, so findings can be consistently wrong.
2. What are the most common types of sampling bias?
The most common types are undercoverage bias, self-selection bias, convenience sampling bias, survivorship bias, nonresponse bias and sampling frame bias.
3. What is undercoverage bias with an example?
Undercoverage bias occurs when part of the target population is missing from the sampling frame. For example, an online-only survey of a whole country excludes people without internet access.
4. What is the difference between undercoverage and nonresponse bias?
Undercoverage happens before selection, when eligible people are missing from the sampling frame. Nonresponse happens after selection, when invited people do not respond or provide unusable answers.
5. How is sampling bias different from selection bias?
Sampling bias concerns who ends up in a research sample. Selection bias is broader and covers any systematic inclusion or exclusion of observations, including during analysis. In participant recruitment, the two largely overlap.
6. What is the difference between sampling bias and response bias?
Sampling bias affects who is included in a study. Response bias affects how accurately included participants answer, for example by giving socially desirable responses.
7. How can researchers prevent sampling bias?
Define the target population clearly, use a complete sampling frame, apply probability or stratified sampling, combine recruitment channels, monitor representation and nonresponse, and compare results against known benchmarks.
8. Can increasing sample size eliminate sampling bias?
No. A larger sample reduces random sampling error, but it does not fix systematic bias. If the selection process is biased, more data simply produces a more precise wrong answer.
Related Topic:
Keep reading



