Create Representative Samples for Stronger Survey Data
Good Survey Research Design Starts with Strong Sampling Strategy
In a perfect world, a survey research project could study all the members of a target universe. Generally, this is neither practical nor affordable. Instead, samples of the larger population (universe) are generated - the sample is the base from which assumptions are made about the target universe. Further, the sample is constructed by using techniques and strategies that contribute to a valid and reliable study. Traditional market research is based on the idea that a sample - a representative group of respondents -- can be identified and accessed.
Representative Samples in Survey Research
In market research, the term representative sample refers to:
- The selection of a few consumers who match members of a target universe of consumers. An example of a target universe could be owners and users of SmartPhones, ages 20 to 30.
- The match between the sample and the universe must be strong for all attributes anticipated to be influential on survey outcomes.
- One example of a sample-to-universe match could be the selection of consumers for a perfume designed by a young, female celebrity. In this instance, attributes anticipated to be influential in survey outcomes would be Female, 18-28 years old, entertainment-savvy.
- A secondary set of attributes might be: Urban-dwelling, enrolled in college, residing on the east coast or west coast, discretionary income (income levels).
- The proportions of members to whom relevant characteristics can be attributed in a sample must closely approximate the proportions of members in the targeted universe of consumers.
- For example, if the consumer universe contains business people, college students, and senior citizens, a representative sample could not be built from agreeable students in the university bookstore on Wednesdays afternoons.
- Access to survey participants can be difficult. It is one of the main reasons why professional panels of consumers are often used in survey initiatives.
- Another effective strategy is to use a stratified random sampling procedure that assists a researcher to tease out data about sub-groups.
Sample Selection in Survey Research
Members of a sample are selected in a number of ways that are intended to reduce bias. It means the probability of generating valid research conclusions is increased, and the conclusions can be generalized to the target universe.
Survey samples are preferably selected through a randomizing process. For instance, if sample members are selected from a database, every third member in the database listing might be selected. Occasionally, members of a sample may need to be assigned rather than randomly selected. It is not a preferred approach as, even under the best conditions, surveys are subject to sample-based inaccuracies that have everything to do with chance and nothing to do with research design. Let's look at a list of sources of error, modified from voter telephone polling issues identified by Experimental Resources.
This list includes possible sources of inaccuracies across survey design, survey implementation, and analysis of survey data:
- Incomplete information about members of a database result in important variables being left out of the sample
- Sample members who were selected are unwilling to participate in the survey.
- Sample members who decline to participate in the study are different with regard to an important variable in the study than those sample members who agree to participate.
- Survey respondents provide false or incomplete responses to survey questions.
The items in this list, again modified from the telephone polling list by Experimental Resources, are related to survey design.
- A randomization process was used but - by chance -- it picks up too many outliers.*
- The questions on the survey are worded poorly and confuse the respondents.
- The order of the questions on the survey unduly influences the responses of subsequent questions.
- Survey responses are subjected to weighting or grouping that distorts the data.
Once a market researcher is reasonably comfortable that a sample is representative of the target population in his survey research, attention can shift to consideration of sample size and confidence intervals.
Experiment Resources is an interesting website created by psychology researchers who were trying to figure out how to calculate and remove outliers.