Sample size does not always matter

If you have ever taken a statistics class, you know that in order to use a subset of a population to infer conclusions about the entire population, your subset needs to be a representative sample. A representative sample means that the characteristics of the sample closely parallels the full population. This ideal sample has three very basic requirements. 

1. The sample size has to be large enough that the mean (average) falls within a normal distribution (Admission: this is way oversimplified, but you can learn more here.)
2. The sample has to be gathered completely at random. Interviewing pedestrians on a busy street is random but sending an email blast to your contacts list is not.
3. The sample population has to have the same demographic makeup as the the general population. For example, if the general population of gamers is 70% 20-somethings, the sample should  also have 70% 20-somethings

This requirement of needing a sample makes many companies reluctant to engage in survey research because they are concerned that their conclusions will be derived from a non-representative sample. Aside from the oft overlooked fact that all research and feedback is useful even when conclusions are not statistically accurate, this concern may still be somewhat unfounded. In truth, not all surveys actually require an effective sample in order to draw inferences from the data.

Often times the results can be considered significant even if the sample is less than representative. To illustrate this concept: Your startup is doing research about which mobile operating systems are the most popular in Singapore, but was only able to get 100 smartphone users to answer their survey. According to SurveyMonkey’s sample size calculator (Disclosure: I work for SurveyMonkey) an accurate sample size of the smartphone users in Singapore requires 324 respondents at the 85% confidence level with a 4% margin of error. You might be unable to draw conclusions about the popularity of Android vs iOs, but if there was only 1 user of the Yun OS (Alibaba’s operating system) and 3 users of Blackberry, the startup can confidently conclude that Blackberry is more popular than Yun OS. Even if the sample was extended to the full 324 respondents, based on computing the standard score (learn about z-score here), Yun OS will always be a lot less than Blackberry.

The concern of not having an accurate sample size should never discourage you from initiating survey research. However, science aside there a number of times where even from the outset, a survey should be conducted knowing that the sample will not be representative of the general population. Here are three popular examples:

Customer feedback. Many of your customers might never bother to answer a customer survey, and without costly incentives it could be quite difficult for you to achieve an accurate sample size from your user base. Nonetheless, you should not be deterred from gathering and analyzing feedback. Negative feedback should never be disregarded even if it is from the handful of people that could be considered as complainers. This complaining segment could just be vocalizing what other customers are feeling. Hearing these complaints can alert you to underlying issues before they get out of control. On the optimistic side, effusive praise could lead you to modify your product in a way that satisfies even more of your customers.

Market research. This is one of those areas that when making big investment decisions really might benefit from an accurate sample; however, an inaccurate sample can still be extremely useful. Numerous times there may be significant (see above for what this means) differences in the responses that negates the need to even have the right sample. In addition, if you ask open ended questions – questions where people just answer with comment boxes –  you could gather valuable feedback that could aid in your product/company development.

Content marketing. Very few people will question the validity of the underlying sample size for a data-driven piece of content before deciding to share it on social media. If your goal is to create a piece of content that is a magnet for SEO links and social shares, you don’t really need the most scientific piece of content. A claim that 92% of a 400 person sample use Facebook on the toilet is going to get lots of social shares regardless of whether the sample was truly representative.

Representative samples aren’t bulletproof

Finally, you should be warned that a sample size is only part of what makes a survey scientific. Even when an accurate sample is used, you can still draw the wrong conclusions if there is bias or faulty methodology at play. Notoriously, all the major polling organizations with their sufficiently large sample sizes completely blew last May’s UK election while SurveyMonkey’s somewhat non-random sample size predicted the outcome almost perfectly. Even the most ideal sample size with lowest possible margin of error is absolutely worthless if it is not representative of the desired population. Keep your focus on gathering a representative sample for research and worry about whether your data is truly sound enough to make a tough decision once you have responses to analyze.  Don’t get stuck in the analysis paralysis of sample size calculations; save those brain cycles for when there is data to analyze.