This is an edited version of three Twitter rants from yesterday. It started as an off-cuff reaction but I was too far into it before I thought that it should be a blog-post rather than Tweets.
Stephen Pinker tweeted out a very weird bit of science theatre created by Michael Shermer.
Pinker has enough critical thinking skills that he should look at it with hefty scepticism…but obviously isn’t. It’s pretend science, using play-acting at science to refute what is obvious and ignores the core issues.
The “survey” by Michael Shermer (which should be a red flag in itself) was sent to 34 notable people associated with the label “Intellectual Dark Web” and asked where they stand on a number of issues. The survey was anonymous, so the views identified in the survey can’t be matched to the individuals asked. https://www.skeptic.com/reading_room/preliminary-empirical-study-shedding-light-on-intellectual-dark-web/
Each and every one of the people surveyed is a public figure who have made multiple public statements about politics and social issues. I don’t need an anonymous survey to find out what Andy Ngo or Sam Harris thinks, I can go and read what they say. And it is what they SAY that matters and what defines the IDW term not what they might privately think. If Sam Harris thinks he has warm & fuzzy liberal beliefs that’s nice but the whole point of the “dark web” label was the contrarian issues he promotes. Maybe Ben Shapiro secretly believes Global Warming is real and climate change is caused by humans. I don’t know but what matters is he propagandises the opposite. If an anonymous survey of the 34 “Intelectual Dark” Webbers reveals that their underlying views are more centrist and mainstream then that is not evidence that the public perception of their public positions is wrong. Rather it confirms a key point about the IDW.
The fundamental issues with the disparate group lumped together as the Intellectual Dark Web is that they are DISINGENUOUS about their politics. It’s not news that Jordan Peterson thinks of himself as moderate and reasonable. We knew that already. It doesn’t change that he (and Harris & Shapiro & Ngo & Quillette) frame and enable a perspective that bolsters the far right. The whole “we are the reasonable ones” is part of the schtick of the IDW. That they’ll boost that in an anonymous survey is, frankly, wank.
Let’s be sceptical as I’m sure Dr Pinker and Shermer would want us to be. Let’s take one conclusion Pinker raises from the survey: The members of the IDW are “concerned w climate” Let’s look at the survey: The survey agrees: “67% strongly agreed that global warming is caused by human actions (no one strongly disagreed)” So their you go! Hoorah! No, no let us be sceptical first. If this was GENUINELY true would it not be easily observed?
To the empircism-mobile! Here’s the output of the Quillete Climate tag https://quillette.com/tag/climate/ zoiks! A hefty TWO article, one concern trolling Greta Thunberg and the other saying people shouldn’t be mean to capitalism. Yes, Quillette is just one source but it is one that connects Steven Pinker on the one hand (who we can observe genuinely does advocate for action on Global Warming) with Andy Ngo on the other hand (who genuinely does have connections with the alt-right and violent far right groups) via Claire Lehmann (Quillette’s founder, fan of Pinker and one time boss of Ngo).
Yes, Steven Pinker himself has a better record on the of global warming but the issue he raised was to look collectively at the IDW and their media-organs. Broadly this is not a group trying to do very much about helping with the issue. And wow, think of the actually good the IDW could achieve given their actual audience. Whatever they may think of themselves, collectively they do have the ear of many on the right – exactly where climate change denial and bad science on the topic is endemic. You’d think these out spoken people might be busy being outspoken on a potential planet wide disaster.
It gets worse. The actual sample was only 18 not 34 people. Nearly half of the 34 didn’t answer. So when the survey says “67%” (the percentage favouring gun control and which believes global warming is real) actually means “12 people” That’s actually both more plausible and more wretched. Even if we accept that 12 of those IDWs think climate change is real, it says almost nothing about the group. Any one member of the original 34 people is a hefty 3% of the population being sampled and hence missing any one of them can have a large impact on the results. This is particularly true given that we already know that the label of “Intellectual Dark Web” is being attached to a group with a very broad range of views on many topics.
Shermer is assuming non-response to the survey is random across the traits being surveyed (i.e the 18 is a random sample of the 34). There is no reason to believe that and really anybody who is wants to seriously call themselves a sceptic should dismiss any general conclusion from the survey without substantial additional supporting evidence.
Indeed there’s good reason to assume that the 18 who responded is not a good random sample of the 34, just on the nature of the numbers. It is very hard with small numbers in a survey for the sample to be representative because one person makes a big difference. Shermer hides that by quoting percentages rather than raw totals but with small number percentages hide how few people he’s talking about. It’s not invalid to look at proportions with small sample sizes, sometimes that is all you have but there’s a point where 12 out of 18 is more informative than 67%.
We can illustrate the issue with the women who were surveyed. Of the 34 named people in survey associated with the “Intellectual Dark Web” 8 (24%) are women. In the survey 3 (17%) are women. So are the IDW 17% women (generalising from survey) or 24%? Obviously 24% is the correct figure but 17% is the equivalent of the the kind of survey conclusions Shermer presents. In fact any one woman listed is 13% of the IDW women, so one more woman answering makes a huge different to sub-sample of women. Any one person is 6% of the whole sample of 18 people!
Circling back to 67% claim. Again assuming everybody who responded is being honest (which I doubt) the survey actually found that 12 people of the 34 who were asked believed in gun control and the same number believed that global warming was real (which I’ll add isn’t saying much, some prominent sceptics will say global warming is real, just as many anti-vaccination campaigners will say they support vaccinations – it is the ‘but’ that follows where the issues lie). That might mean 67% or there about of 34 believe in gun control but a safer conclusion is no less than 35% do (12/34) and no more than 82% (28/34). Given how granular this data is, hoping the estimate is in the middle isn’t supported.
This is why I call it theatre. It is the wrong methodology applied badly. It illustrates methodological snobbery. Synthesising the complex views of a small group of people is exactly where qualitative methods work better. It is a domain where you need to put on your humanities hat and apply those humanities skills. Shermer is using sciencey film-flam by presenting a pointlessly anonymous survey and presenting the results as percentages as if there were proportions of the whole group.
Don’t get me wrong I absolutely LOVE applying basic quantitive methods to things and place where they don’t always make sense. It’s very much my hobby but even on this less than 100% serious blog I’d throw more caveats at better numbers than Shermer is using.
