Life never seems to hand you quite as much data as you’d like to make a decision. So what do you do when that happens? Whether it is planning the viability of future programs, or reporting existing successes, at some point you are going to need to do some extrapolating. So how do you extrapolate in a way that makes for effective decisions? How do you avoid the pitfalls of extrapolation?
Population-based extrapolations
People are unique and unpredictable. Unfortunately that makes for difficult public planning. Ultimately, you are going to have to predict widespread behavior based on a limited data pool.
One of the most frequent, and I think dangerous, forms of extrapolation is the “friends” extrapolation. “All of my friends behave in a certain way or believe a certain thing so I will extrapolate that everyone, or at least the majority of people, behave or believe the same way.” Be very careful about doing such extrapolations. Rather than finding them to be effectively predictive, you are likely to find that your friends are not as diverse as you think they are. If you asked a technophile in the 1980’s, based on an extrapolation of their technophile friends, Betamax tapes were going to be the wave of the future. The annals of history now laugh at such predictions.
The more formal form of population extrapolation is surveying. I would caution you to proceed carefully though. A poorly done survey is not nearly as scientific and predictive as you might want to believe. Early in my career, I had the good fortune of working one year with a doctoral student in statistics. I got to see what a “real” survey looked like. Quite frankly, it was overwhelming, with layers upon layers of redundant questions, all asked slightly differently to ensure you were getting a consistent answer. It has left me completely underwhelmed by, and cynical, of every survey I have seen since. I wish I could say that this underwhelming-survey issue was an issue limited to over-eager first year students, but it seems to pervade our society. How many headlines have you seen projecting national trends based on a survey of 100 people from across the country – if by “across the country” you mean that the survey randomly called 20 people each in New York, LA, Dallas, Chicago, and Indiana (Indiana always seems to get included to prove that your survey has captured the pulse of the “heartland”)? The bigger problem revolves around how many times that survey is quoted by people who use its conclusion to bolster their own argument.
Even among the better surveys I have seen recently, it seems that the only takeaway that most surveyors have gotten from their statistics class is that you should strive to get at least 10% of the population to participate for the survey to be statistically relevant. And even in most of those, the “at least” part seems to have been missed and the 10% of the population seems to be the upper end of the goal range not the bottom (“I got almost 10% of the population to reply”).
Here’s the problem: If you don’t do a survey right, a survey is no more valuable than saying “I had a conversation with a group of friends last night at the (insert favorite hang-out spot; mall, pub, diner, gym) and we agreed that..” The opinion of a handful of people is in reality no more than that, the opinion of a handful of people. If you don’t have a statistically relevant sample size and a good survey, don’t assume that you can extrapolate information for your whole population.
I’m not suggesting not to do surveys, or not to talk to friends. I think that both can be incredibly valuable in revealing issues that you might not have considered. As an example, don’t be surprised if you find yourself having this sort of exchange with yourself after viewing the results: “how can they say they don’t know where the recycling site is? It’s been in the same place in that building for over a decade? Oh. Wait a minute. It doesn’t matter that the site has been in the same place for a decade, they’ve only been in the building for a few weeks. I guess we must have some sort of information and signage issue in that building. Let me go see what it is.”
Surveys and conversations can be incredibly revealing and valuable. Just don’t count on them to be more revealing than they actually are.
Roger Guzowski
