Margins of error

I suspect most people who read this blog know all this already but I’ve met the same misunderstanding at work recently and also in the context of the opinion polls around the POTUS election. So here is a simplified explanation.

Imagine I have a great big jar of jelly beans, which are the favoured confectionary of probability explanations. There are exactly 500 red jelly beans and 500 blue jelly beans and nothing else – no Jill Stien jelly beans or exotic Even McMulberry flavours. A jelly bean pollster doesn’t know this, though. The pollster wants to estimate the proportion of red and blue jelly beans in the jar BUT is only allowed to look at some of the jelly beans.

The pollster grabs a handful of jelly beans from the jar and looks at the relative proportion of jelly beans. Naturally, I don’t want the pollster to do this very often because they’ll put their germ-ridden hands all over my beautiful jelly beans. So pollster only has this handful to look at. They have to make a key assumption – that the jelly beans were well mixed so that their handful is a random pick of jelly beans in the jar.

The pollster looks at the proportion of red to blue jelly beans. Let’s say they have 5 red and 8 blue jelly beans. The pollster says that the proportion of red to blue is 38% to 62% BUT they also report a margin of error that is quite large. They can’t be sure this figure is right because they know they may have been unlucky. With only 13 jelly beans in their handful, it isn’t wholly impossible that they could pick out nothing but blue jelly beans if the true proportion was 50-50. Now note if they did pick out nothing but blue, this could happen by chance.

Margins of error address only this aspect of errors in polling – that the proportion in the sample was to some extent an ‘unlucky’ pick. Both the reported figure and the margin of error BOTH assume that the picking was done correctly. In our jelly bean example the assumption that the beans were well mixed together.

Now it so happens that I didn’t mix the jelly beans well (although the pollster can’t tell)*. There are actually MORE red towards the top and fewer red towards the bottom of the jar. So the pollster’s assumption was wrong. A clever pollster might try to find ways to deal with this methodologically (e.g. by grabbing beans from both the top and the bottom) but the principle still applies: the reported estimate and the margin of error assume that the sampling methodology was valid. The margin of error doesn’t (and can’t) account for the probability of what in common parlance would be called an ‘error’ (i.e. a mistake).

Advertisements

4 comments

  1. thephantom182

    http://phantomsoapbox.blogspot.ca/2016/11/best-article-yet-on-us-election-polling.html

    A journalist took the unprecedented step of actually interviewing the guys who called the election accurately. There were some. What they had to say about it is very interesting. It doesn’t much resemble your notions above, though. It seems many polling companies are not very good at their jobs, and pretty much go through the motions. Getting it right is less important than staying in the pack with the other companies.

    Another issue pre-election that I mentioned and you rather rudely scoffed at, poll fixing. As in, polls deliberately bent or ditched to favor one candidate. IPSOS actually spiked one that favored Trump. A poll conducted in September was held back and released November 16th. Shenanigans.

    Like

  2. supplanter

    @Camestros: Clinton will end up winning the national popular vote by 1.5 – 2%, which is within the margin of error IIRC. In fact, it’s a smaller error than 2012, where the final polls underestimated Obama’s margin by about 3-4 points. It’s also just about bang-on the national exit polls.

    The one state exit poll I’ve perused so far, Wisconsin, is also roughly in line with Trump’s victory there. (Which is why is discount the likelihood of vote-hacking by Russia or whomever.) I expect to find that is also true of the other battleground-state exits.

    The only polling oddity is that the final preference polls in the Rust Belt swing states all looked more favorable to Clinton than the final tallies (and exit polls) turned out to be. Here I think 538 comes off remarkably well. Their relatively higher chance of a Trump upset was based on the cross-correlation of the vote between states – that states tend to move together, so if one state moved Trump’s way, other states were likely to also. I’m also mindful of an article I read on Monday or Tuesday morning that week which quoted some Clinton campaign folks as saying, “Actually, we’re kind of flying blind in Pennsylvania. We have very little good current data on where the state is right now.” Given the huge election-eve rally they staged in Philly, you kind of suspect their internal polling indicated there was a risk.

    I’m just not convinced there’s a huge mystery to explain here.

    Like