Reading Bad Science So You Don’t Have To

Yesterday I made a mistake. I was aware of a kerfuffle around the publication in the notable open journal PLOS One of an article on the dubious notion of “Rapid Onset Gender Dysphoria”. The mistake was not listening sufficiently when lots of people said the article was very, very bad and thinking “Yeah, but how bad could it be if it was published somewhere non-obscure?” A pernicious thought that I’m very glad I didn’t express out loud because then I went and read the article…

There’s a bad, bad mental habit of discounting objections to arguments that you haven’t paid attention to when those arguments come from people you perceive as being in some way partisan on an issue, even if you yourself are partisan on the issue. It is the insidious sibling of false balance and ‘both sides’ that assumes that criticism from ‘your’ side must be at least a bit exaggerated. So I’m starting with a mea-culpa: it wasn’t that I didn’t believe the critics of the article, just that I assumed they must be exaggerating its badness at least a little. It’s a bias of arrogance that assumes that because somebody feels passionate about something that their statements aren’t wholly reliable (arrogance because it’s not a rule you then apply to yourself).

Anyway, enough beating myself up. Some background.

The rights of transgender people have become an increasingly virulent political battlefield following a pattern that we’ve seen many times before: a group of people who have been systematically marginalised ask for what is little more than basic human dignity only to be met with a counter-reaction that is deeply confronting. The pushback from the Christian Right and the Alt-Right is one thing but the vehement reaction from some people in the centre and the left can also be terrible.

Rapid Onset Gender Dysphoria” is a concept that has been floating around various anti-transgender rights groupings on the internet. It is essentially a pseudo-scientific term in the sense that it takes a basic (and false) claim and dresses it up in quasi-scientific terms. The claim is that many teenagers are claiming to have gender dysphoria because it is trendy or because of peer pressure. The concept has rested mainly on increased visibility of transgender people in society and structurally is no different than similar claims made about people being gay or lesbian. As social stigma is reduced and as people find more open social support, more people will be public about core aspects of themselves. (see also this link https://medium.com/@juliaserano/everything-you-need-to-know-about-rapid-onset-gender-dysphoria-1940b8afdeba )

However, there are people who are keen for “Rapid Onset Gender Dysphoria” to be a thing, that is somehow a “real” medical condition that should be recognised as a false-positive when a young person claims to be transgender. Essentially it is a way of trying to medicalise the argument that “it is just a phase”.

Into this space has come a paper published in the journal PLOS One entitled “Rapid-onset gender dysphoria in adolescents and young adults: A study of parental reports” by Lisa Littman. It is currently available here https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0202330 and it really is a very bad study.

There is a whole series of posts starting here that pull the study to bits in various ways that are worth reading and in particular this post: https://genderanalysis.net/2018/08/meet-the-unbiased-reliable-not-at-all-transphobic-parents-from-the-rapid-onset-gender-dysphoria-study/ gets to the heart of what is so very wrong with the paper.

The paper claims to be a kind of exploratory study of “rapid onset gender dysphoria” as a hypothesis and uses a report from parents to suggest that such a phenomenon exists as an actual condition. The paper then speculates on causes and recommends some actions by medical practitioners. However, the paper is not just methodologically flawed, it is actually an unwitting study of something else altogether.

The study involved posting a questionnaire on several parent websites but those websites were websites were “rapid onset gender dysphoria” was already being discussed and which included activism around the concept. The survey posted itself included leading question such as:

“How many of your children have experienced a sudden or rapid onset of gender dysphoria, which began after puberty?”

In the survey instrument that is the first question after the basic demographic questions. Put another way, the study assumed the existence of the phenomenon and then surveyed people likely to also believe in the existence of the phenomenon and then reported positive results as confirmation that the phenomenon existed.

Put another way, it would be like surveying people on an internet forum where people discussed and shared studies of UFO sightings, asked them “How many UFOs have you seen?” and then credulously used the number of positive reports as a lead into to discuss whether the UFOs were from Mars or another galaxy. Too silly an example? Perhaps, but here is a different analogy. The study is very like surveying internet forums known for GamerGate activism and asking them “How many cases of bad ethics have you seen in games journalism?” and then concluding that the volume of responses showed something meaningful about how reviews of video games function.

Essentially it is a very complex form of question-begging.

There are some formal ways that the paper adopts valid methods. There is a survey and the responses were collected and (I assume) tabulated correctly. It’s not necessarily invalid to collect data from parents to possibly identify a medical condition in their children. However, the basic structure of what was done fundamentally changes the meaning of all the results from what was intended.

To salvage the work done and turn the data into something meaningful would require recognising that it answers no medical questions at all. Instead, what has been done is a survey into *beliefs* and specifically beliefs among a community of people i.e. the study is sociological and the topic is “really” how a pseudoscientific idea becomes entrenched in some internet communities. Ironically, the study focuses on concepts such as “social contagion” and yet somehow misses that the study itself reveals how ideas spread due to peer pressure and apparent topicality.

What lessons are there to be learned? Well for me I should have not wasted my time reading a paper that everybody had already told me was bad. It was bad, bad not just interestingly flawed or mistaken.

Advertisements

6 thoughts on “Reading Bad Science So You Don’t Have To

  1. Peer review fails every now and again – PLoS One published (and later retracted) a severely flawed paper on eukaryote phylogeny last year.
    PLoS One may be more vulnerable that some other journals – it publishes lots of articles, on all fields, so there’s a higher risk of getting a peer reviewer with Dunning-Kruger syndrome (not qualified to review the paper, but not aware of their lack of qualification), exacerbated by a policy of publishing low impact studies (if I recall correctly, if you want to publish a negative result PLoS One is a sensible place to submit).

    Like

Comments are closed.