A bad survey about the ‘Intellectual Dark Web’

This is an edited version of three Twitter rants from yesterday. It started as an off-cuff reaction but I was too far into it before I thought that it should be a blog-post rather than Tweets.

Stephen Pinker tweeted out a very weird bit of science theatre created by Michael Shermer.

Pinker has enough critical thinking skills that he should look at it with hefty scepticism…but obviously isn’t. It’s pretend science, using play-acting at science to refute what is obvious and ignores the core issues.

The “survey” by Michael Shermer (which should be a red flag in itself) was sent to 34 notable people associated with the label “Intellectual Dark Web” and asked where they stand on a number of issues. The survey was anonymous, so the views identified in the survey can’t be matched to the individuals asked. https://www.skeptic.com/reading_room/preliminary-empirical-study-shedding-light-on-intellectual-dark-web/

Each and every one of the people surveyed is a public figure who have made multiple public statements about politics and social issues. I don’t need an anonymous survey to find out what Andy Ngo or Sam Harris thinks, I can go and read what they say. And it is what they SAY that matters and what defines the IDW term not what they might privately think. If Sam Harris thinks he has warm & fuzzy liberal beliefs that’s nice but the whole point of the “dark web” label was the contrarian issues he promotes. Maybe Ben Shapiro secretly believes Global Warming is real and climate change is caused by humans. I don’t know but what matters is he propagandises the opposite. If an anonymous survey of the 34 “Intelectual Dark” Webbers reveals that their underlying views are more centrist and mainstream then that is not evidence that the public perception of their public positions is wrong. Rather it confirms a key point about the IDW.

The fundamental issues with the disparate group lumped together as the Intellectual Dark Web is that they are DISINGENUOUS about their politics. It’s not news that Jordan Peterson thinks of himself as moderate and reasonable. We knew that already. It doesn’t change that he (and Harris & Shapiro & Ngo & Quillette) frame and enable a perspective that bolsters the far right. The whole “we are the reasonable ones” is part of the schtick of the IDW. That they’ll boost that in an anonymous survey is, frankly, wank.

Let’s be sceptical as I’m sure Dr Pinker and Shermer would want us to be. Let’s take one conclusion Pinker raises from the survey: The members of the IDW are “concerned w climate” Let’s look at the survey: The survey agrees: “67% strongly agreed that global warming is caused by human actions (no one strongly disagreed)” So their you go! Hoorah! No, no let us be sceptical first. If this was GENUINELY true would it not be easily observed?

To the empircism-mobile! Here’s the output of the Quillete Climate tag https://quillette.com/tag/climate/ zoiks! A hefty TWO article, one concern trolling Greta Thunberg and the other saying people shouldn’t be mean to capitalism. Yes, Quillette is just one source but it is one that connects Steven Pinker on the one hand (who we can observe genuinely does advocate for action on Global Warming) with Andy Ngo on the other hand (who genuinely does have connections with the alt-right and violent far right groups) via Claire Lehmann (Quillette’s founder, fan of Pinker and one time boss of Ngo).

Yes, Steven Pinker himself has a better record on the of global warming but the issue he raised was to look collectively at the IDW and their media-organs. Broadly this is not a group trying to do very much about helping with the issue. And wow, think of the actually good the IDW could achieve given their actual audience. Whatever they may think of themselves, collectively they do have the ear of many on the right – exactly where climate change denial and bad science on the topic is endemic. You’d think these out spoken people might be busy being outspoken on a potential planet wide disaster.

It gets worse. The actual sample was only 18 not 34 people. Nearly half of the 34 didn’t answer. So when the survey says “67%” (the percentage favouring gun control and which believes global warming is real) actually means “12 people” That’s actually both more plausible and more wretched. Even if we accept that 12 of those IDWs think climate change is real, it says almost nothing about the group. Any one member of the original 34 people is a hefty 3% of the population being sampled and hence missing any one of them can have a large impact on the results. This is particularly true given that we already know that the label of “Intellectual Dark Web” is being attached to a group with a very broad range of views on many topics.

Shermer is assuming non-response to the survey is random across the traits being surveyed (i.e the 18 is a random sample of the 34). There is no reason to believe that and really anybody who is wants to seriously call themselves a sceptic should dismiss any general conclusion from the survey without substantial additional supporting evidence.

Indeed there’s good reason to assume that the 18 who responded is not a good random sample of the 34, just on the nature of the numbers. It is very hard with small numbers in a survey for the sample to be representative because one person makes a big difference. Shermer hides that by quoting percentages rather than raw totals but with small number percentages hide how few people he’s talking about. It’s not invalid to look at proportions with small sample sizes, sometimes that is all you have but there’s a point where 12 out of 18 is more informative than 67%.

We can illustrate the issue with the women who were surveyed. Of the 34 named people in survey associated with the “Intellectual Dark Web” 8 (24%) are women. In the survey 3 (17%) are women. So are the IDW 17% women (generalising from survey) or 24%? Obviously 24% is the correct figure but 17% is the equivalent of the the kind of survey conclusions Shermer presents. In fact any one woman listed is 13% of the IDW women, so one more woman answering makes a huge different to sub-sample of women. Any one person is 6% of the whole sample of 18 people!

Circling back to 67% claim. Again assuming everybody who responded is being honest (which I doubt) the survey actually found that 12 people of the 34 who were asked believed in gun control and the same number believed that global warming was real (which I’ll add isn’t saying much, some prominent sceptics will say global warming is real, just as many anti-vaccination campaigners will say they support vaccinations – it is the ‘but’ that follows where the issues lie). That might mean 67% or there about of 34 believe in gun control but a safer conclusion is no less than 35% do (12/34) and no more than 82% (28/34). Given how granular this data is, hoping the estimate is in the middle isn’t supported.

This is why I call it theatre. It is the wrong methodology applied badly. It illustrates methodological snobbery. Synthesising the complex views of a small group of people is exactly where qualitative methods work better. It is a domain where you need to put on your humanities hat and apply those humanities skills. Shermer is using sciencey film-flam by presenting a pointlessly anonymous survey and presenting the results as percentages as if there were proportions of the whole group.

Don’t get me wrong I absolutely LOVE applying basic quantitive methods to things and place where they don’t always make sense. It’s very much my hobby but even on this less than 100% serious blog I’d throw more caveats at better numbers than Shermer is using.

13 thoughts on “A bad survey about the ‘Intellectual Dark Web’

  1. Even if we dismiss the methodological issues with the survey, and assume that the result is accurate and representative for these IDW luminaries, the numbers does not support the summary in Pinker’s tweet.

    Considering the issue at hand, and how unanimous the actual climate science is on this, 67% is a low number, not a high one. If 67% agrees that humans causes global warming, we’re left with 33% who disagree – i.e. who dismiss climate science. That fits poorly with Pinker’s attempt to describe the IDW as relatively moderate thinkers who have become victims of an illiberal “hard left” elite.

    If e.g Sam Harris was presented with a poll saying that 67% of Muslims condemn terrorism, his conclusion wouldn’t be “Muslims are concerned with terrorism”. He’d write an angry article about how not enough Muslims condemn terrorism. I would agree with him on that particular point – but I think the same logic applies here. When 1 out of 3 of these supposed “intellectuals” and sceptics dismiss climate science, the IDW as a group is not “concerned with climate” – they’re less concerned with climate than rational people ought to be.

    Liked by 2 people

  2. The very first sentence rings alarm bells – contrasting themselves with the “academic hard left”, as if there’s no-one else.

    Liked by 2 people

  3. What’s fascinating about this is that they want so desperately to say that they have liberal values — the same values they describe as a cancer and terrorism that is taking over and destroying our society. The whole schtick is that the IDW are “rebels” against this tyranny and here they are flogging a weird survey about how they actually embrace and believe in the “tyranny.”

    I mean, I get that it’s supposed to make them look moderate, reasonable and mainstream anti-equality views, but apparently they think they do that by actually embracing equality views, which they are acknowledging are popular and reasonable but at the same time have built their whole brands on denouncing. It’s that peculiar, completely contradictory thing that they do, trying to keep that good, meritorious rep in the face of a majority, younger population that is turning against their main advocacies. Where values are simply a sham that you take on and off like a coat for them, whatever will make money and give you a sense of power.

    It does indicate that they are losing ground (despite the joys of the White House,) in the culture and as such are retreating to show that they fit in. While still reassuring their followers that they are total rebel badasses who will take the feminists and uppity black people down so that they make money. It’s the erosion progress of equal civil rights efforts.

    Liked by 2 people

  4. 100% of haunted umbrellas surveyed think that the “IDW” are intellectually dishonest idiots who hold contrarian positions even when it flies in the face of scientific evidence, consensus, agreed-upon facts and human decency.

    That they might claim otherwise in anonymous surveys just displays their dishonesty.

    Liked by 3 people

  5. I know this isn’t really the point, but Pinker claiming the IDW label is “facetious” is kind of silly. It is their own term, and many of those involved actively identify with it. Whereas he obviously has no problem throwing around terms like “the academic hard left”.

    Liked by 4 people

  6. Having considered this back and forth, I suspect it’s intended as a “redeeming” piece, but I’d argue it’s either (a) irrelevant or (b) condemning.

    (a) has already been discussed in the post, so let me focus on (b).

    The people polled are, on the whole, presenting as “not all all liberal”, “global warming is either not a thing, or not human-induced”, and a variety of things. Essentially, we can assume, for profit. This would, I argue fall in the “not a net good”. And this proves they[‘re doing it without (on the balance) having the beliefs they bolster by their actions. Making it a “doing net ungood purely for profit”. That probably falls closer to the “evil” end of the “paragon to evil” scale.

    Liked by 1 person

  7. Leaving aside the whole issue of non-ignorable non-response, on issues like these the intersection of opinions is often as important for defining ‘not evil’ as a position on single issues. For example, it’s great that you accept anthropogenic climate change, but if your proposed solution is to wipe out half the human race, ideally starting with brown people, then you are not a good person.

    Liked by 2 people

Comments are closed.