Editor’s Note: This is the ninth in a series of posts on common Survey writing mistakes. Click here to see the previous item and stay tuned for more!
MISTAKE #9:Â YOUÂ THINK SAMPLE SIZE MATTERS
Yes, I have just committed Survey heresy. I can already hear the statisticians, data scientists, and demographic pollsters screaming. “What the heck are you talking about?” they holler from on high. “OF COURSE sample size matters! How can you get statistically significant results if you don’t have the right sample size? How do you know your margin of error? What about confidence intervals and second-order relationships and…” And on, and on, and on.
Before they start lighting matches, allow me to clarify. For quantitative opinion polls constructed by professional Survey writers, I agree – sample size is important. Seriously, who would believe a presidential poll of 25 people? (Hehehe…) It’s just that most Surveys are not written by professionals, are not intended for mass distribution, and are so fraught with mistakes that all a bigger sample size will do is yield more bad data.
As my boss at Applied Materials used to say to me early in my career,
It is better to be approximately right than exactly wrong.
In other words, do not waste a lot of time, money, and effort trying to achieve perfection when in most situations what matters most is that you get the right idea (a concept that has life implications far beyond how many responses to seek out for a Survey!). This idea is especially true for qualitative expertise Surveys in which the professional credentials of the respondents carry tremendous weight. Quality matters far more than quantity.
We live in a data-obsessed era, so perhaps we shouldn’t be blamed for thinking that more is better. Unfortunately, in the quixotic quest for ever-greater resolution we often miss the bigger picture. We forget about WHY we are running a Survey in the first place: to get an understanding of how things are (more or less) at a particular moment in time. If you approach the problem with the understanding that things can and will change rapidly, your focus will shift from precision to approximation, from definitive answers to sentiment and directionality.
Would you rather know that based on a poll of 1,000 nurses with a margin of error of plus or minus two percent 62.8%  prefer a particular model of ER bed or that most of a sample of 25 nurses from large hospitals who are on their facilities’ procurement committees prefer a different model? I would argue that the sentiments captured in the latter, targeted study matter far, far more than those of the former. The expense and effort required to gather those 1,000 untargeted opinions would be better spent crafting really great questions for the targeted group that dive deep into the WHY in addition to the what.
Surveys offer a great way to get a lot of insights quickly, but the notion that sample size is of paramount importance is just plain wrong. Instead of more precision, more sample usually begets more bad data. Write great questions and pose them to the right people, and you’ll almost always find yourself in the admirable position of being approximately right instead of exactly wrong.
