Uncertainty Wednesday: Fooled by Small Numbers

The last couple of Uncertainty Wednesdays examined fallacies based on statistical phenomena such as imperfect correlation and the base rate. Another one of these has to do with small numbers. A famous example of this fallacy occurred when a study showed that among the best performing schools in the country there was a high percentage of small schools. This resulted in a push for making schools smaller.

A lot of money and effort was expended on this push for smaller schools. That is, until someone decided to also study the worst performing schools. And again, it turned out that there was a high percentage of small schools among them. Ouch! So apparently being a small school either gives you really good or really bad performance.

In reality much of this is simply was the result of small numbers. When you have a small number of students, you have an even smaller number of teachers. And so it is much easier to wind up either with really bad teachers or with really good teachers. As a school gets larger, it is much more likely to have some good and some bad teachers. Hence it is much more likely to be of average performance.

This small numbers effect is, like the previous fallacies, incredibly widespread. In companies think about the performance of departments (small ones versus large ones). For governments consider small cities or countries compared to large ones. And so on.