This is a vague follow-up to this post: https://camestrosfelapton.wordpress.com/2022/09/11/does-the-genre-you-read-alter-your-view-of-the-world/
As I said the other day, the study reads to me that it reflects people’s opinions of themselves. It didn’t (couldn’t) look at actual reading behaviour and didn’t look at actual cognitive preferences, so what they ended up with is a conclusion more like people who see themselves as people who read serious literature also see themselves as more thoughtful.
Assuming I had time and money to look at this further, what would I do?
One thing that struck me is that the researchers intentionally improved the overall representativeness of their sample. They started with self-selecting volunteers from Amazon’s Mturk service (https://www.mturk.com/), then students at a public university and then national random samples. A nationally representative random sample is a good thing for a research project, right? Maybe not.
Consider the claim they want to test: the kind of literature you read “growing up” has an impact on how you think about the world. Lots of things may have an impact on your cognitive development, diet, your mother’s diet, poverty, lead in petrol, schooling, your parent’s schooling etc etc. The researcher made efforts to account for this statistically but that is vulnerable to multiple issues. In some ways, it is the second study (university students) that is better experimental design. With that sample, the people involved have all reached a similar level of education.
Put another way, if the hypothesis is true, then we should see it among the socially and educationally privileged. It may seem counter-intuitive to focus your research on a group that gets over-researched but with the specific hypothesis they were examining it makes a lot more sense to see if the effect can be observed in a narrow group who have had a lot of advantages.
So, they should have focused on educated people from an educated middle-class background. Demonstrate the hypothesis among that group first.
Secondly, there is a big gender question with the genres. Romance and Chick-lit are both genre categories heavily socially associated with women readers. In a society where women’s interests are still perceived as lesser or less serious, that is another confounding issue.
So the sample should have been largely women. After all, if the hypothesis is true then it should be true for women and if the hypothesis can’t be demonstrated with a women-only sample then there would be substantial reason to doubt it.
Actually tracking reading habits would be difficult, in particular, because a long-term study of reading habits itself might impact people’s reading choices. One way through this though might be to consider genre ambiguity.
In the study people just picked broad genres names. However, fiction can be ambiguous. Is Pride & Prejudice romance? Is 1984 science fiction? The researchers saw genre ambiguity as simply adding noise to the classification but there is scope here to test something else.
Consider a list of famous books that have some degree of genre ambiguity. Get your subjects to classify them into genres. If we are lucky, we will get a decent number of people who have read, say, Pride & Prejudice, some of whom see it as “literary fiction” and some of whom see it as Romance (or Chick-lit?). That wouldn’t just be a way to control for genre ambiguity but it would also allow for eliminating a counter-hypothesis.
If I’m right about what they found, then if we had two groups of people who had largely read the SAME pool of books but group A saw the books as “literary fiction” and group B saw the books as “romance” then the relationship the researchers found would still hold up but the underlying cause would not. The books hadn’t changed but the way the people thought of those books had changed, which would lend weight to the result being about self-perception rather than the books changing how your brain worked. Of course, that would be dependent on finding enough people in both groups.