Ways of fixing a study

This is a vague follow-up to this post: https://camestrosfelapton.wordpress.com/2022/09/11/does-the-genre-you-read-alter-your-view-of-the-world/

As I said the other day, the study reads to me that it reflects people’s opinions of themselves. It didn’t (couldn’t) look at actual reading behaviour and didn’t look at actual cognitive preferences, so what they ended up with is a conclusion more like people who see themselves as people who read serious literature also see themselves as more thoughtful.

Assuming I had time and money to look at this further, what would I do?

One thing that struck me is that the researchers intentionally improved the overall representativeness of their sample. They started with self-selecting volunteers from Amazon’s Mturk service (https://www.mturk.com/), then students at a public university and then national random samples. A nationally representative random sample is a good thing for a research project, right? Maybe not.

Consider the claim they want to test: the kind of literature you read “growing up” has an impact on how you think about the world. Lots of things may have an impact on your cognitive development, diet, your mother’s diet, poverty, lead in petrol, schooling, your parent’s schooling etc etc. The researcher made efforts to account for this statistically but that is vulnerable to multiple issues. In some ways, it is the second study (university students) that is better experimental design. With that sample, the people involved have all reached a similar level of education.

Put another way, if the hypothesis is true, then we should see it among the socially and educationally privileged. It may seem counter-intuitive to focus your research on a group that gets over-researched but with the specific hypothesis they were examining it makes a lot more sense to see if the effect can be observed in a narrow group who have had a lot of advantages.

So, they should have focused on educated people from an educated middle-class background. Demonstrate the hypothesis among that group first.

Secondly, there is a big gender question with the genres. Romance and Chick-lit are both genre categories heavily socially associated with women readers. In a society where women’s interests are still perceived as lesser or less serious, that is another confounding issue.

So the sample should have been largely women. After all, if the hypothesis is true then it should be true for women and if the hypothesis can’t be demonstrated with a women-only sample then there would be substantial reason to doubt it.

Actually tracking reading habits would be difficult, in particular, because a long-term study of reading habits itself might impact people’s reading choices. One way through this though might be to consider genre ambiguity.

In the study people just picked broad genres names. However, fiction can be ambiguous. Is Pride & Prejudice romance? Is 1984 science fiction? The researchers saw genre ambiguity as simply adding noise to the classification but there is scope here to test something else.

Consider a list of famous books that have some degree of genre ambiguity. Get your subjects to classify them into genres. If we are lucky, we will get a decent number of people who have read, say, Pride & Prejudice, some of whom see it as “literary fiction” and some of whom see it as Romance (or Chick-lit?). That wouldn’t just be a way to control for genre ambiguity but it would also allow for eliminating a counter-hypothesis.

If I’m right about what they found, then if we had two groups of people who had largely read the SAME pool of books but group A saw the books as “literary fiction” and group B saw the books as “romance” then the relationship the researchers found would still hold up but the underlying cause would not. The books hadn’t changed but the way the people thought of those books had changed, which would lend weight to the result being about self-perception rather than the books changing how your brain worked. Of course, that would be dependent on finding enough people in both groups.


11 responses to “Ways of fixing a study”

  1. I think that Literary Fiction should be left out as a category entirely. Anything published as that will also fall under one of the other categories.

    Liked by 1 person

    • Well yes but then the whole point of the study vanishes. Another approach would be for them actually get to grips with what they claim is going on with “literary fiction”. If there is some element of fiction that leads to this shift in view then “isolate” it in some sense. Come up with a measure of story complexity?


      • then the whole point of the study vanishes.

        I don’t think that is necessarily the case. This is merely an anecdote, not data, but:

        I was raised in a household with a parent who is extremely racist, sexist, homophobic, ableist, and classist (and not shy about making derogatory statements about POC, women, LGBTQ, the ADA, and poor and homeless people — oh, and also environmentalism and the EPA). I was also growing up in a small rural town in an area where a large percentage of the population hold those views.

        Yet growing up, I did not imprint on those views. Instead, from an early age, I felt that such comments and attitudes were unfair and wrong. I have spent a considerable amount of time thinking about why that might be, and the only answer I can come up with is that I was an avid reader, especially of SFF, and that exposed me to the wider world and more egalitarian viewpoints.

        Did the mysteries and gothics I also was reading contribute to that? Probably — but those genres don’t place nearly as much emphasis on egalitarianism, environmentalism, and looking out for each other as SFF does.

        Of course, this is all looking through the retrospectroscope, and I have no way of really knowing why I rejected the views on which the vast majority of my classmates imprinted.

        Liked by 1 person

  2. You could compare literary fiction readers among men to those who mainly read male-signified ‘trashy’ fiction – cowboy books or spy thrillers for example (no offence to those who like those genres – I am picking on the reading preferences of my dad who is a 100% pure pulp reader).

    I must say I also struggle to distinguish between literary fiction and ‘classics’ – is it just a matter of age or is one the new term for the other? If I read a lot of classics as a child does this count as reading lit-fic?

    Liked by 2 people

    • And spy fiction and cowboy fiction have the same ambiguity with “literature” – is Graham Greene a spy writer or a literary guy? How about Cormac McCarthy – cowboy writer or literati?

      Liked by 2 people

  3. Watching the researchers make genre assignments that English professors still debate vehemently is all I need to question the study.

    (Well, that and the fact that I’m an English professor at an R1 public flagship but nearly all of my pleasure reading is SF, fantasy, and popular science. In spite of all that “trash” reading, I somehow manage to stay thoughtful about the “literary” fiction I teach.)

    Liked by 2 people

  4. Asking people in their 50’s about what sort of books they read in high school is not going to give very accurate answers. That’s also an argument for selecting people fresh out of adolescence – university students – instead of a representational sample of the population at large.

    If you have a group in the same age range, you could probably set up a list of ~100 books and have it be a reasonable selection of classics and books that were bestsellers when the group were teens. Then you can ask “which of these had you read by the time you graduated high school”, and get a decent picture of people’s reading habits. It’s also possible to include options for “read it before junior high”, “in junior high”, “in high school”, if you want that granularity.

    Liked by 2 people

  5. The main Problem is that they have wastly different books in the group and it is for everyone different what they get from a work.
    There is a cool quote about 2 books that can chance a person forever, and both Tolkien and Ayn Rand are in the same group here.
    There is a big different if I read Larry or The Broken Earthtrilogy.

    Liked by 1 person

  6. “So the sample should have been largely women. After all, if the hypothesis is true then it should be true for women and if the hypothesis can’t be demonstrated with a women-only sample then there would be substantial reason to doubt it.”

    Yeah, there’s a problem with that idea right there — women are a marginalized group. As such, we’re taught and expected to have a more open, equalitarian and complex view of others. We are taught to treat others as our superiors or equals and to analyze others in depth for understanding, be concerned for their feelings, try to caretake and service others, etc. It’s not across the board. White women, for instance, may treat BIPOC, especially WOC, as inferior, be unconcerned with understanding them, retreat into stereotypes, etc. — there are cross-intersections and of course, many differences among individual women, wealth classes and so on. But in general, women are culturally treated as inferior and legally second class and expected as such to watch others, appease others and care for others’ feelings — enough to skew the data as a variable.

    Additionally, women in most cultures that allow women to read are much less socially stigmatized for reading for non-work/school purposes than men. At least 70% or more of the fiction reading audience in English language publishing are women and any sub-genre will have at least 50% women if not more. And while women’s opinions of thrillers, SFF, serious non-fiction, business and politics — things coded as important and worthy and thus men’s areas — are considered inferior, they aren’t usually stigmatized for reading in those areas anymore (though there is still some serious stigmatization of women fans of all forms of SFF.) They certainly aren’t usually discouraged from reading classic literature or current fiction designated literary. The majority of members of book clubs, for instance, are women.

    Whereas, men are readers of self-help, romance and women’s fiction (of which “chick lit” — modern comedies with women protagonists — is a sub-genre) but are still regularly stigmatized for reading such material openly and often discouraged away from it entirely. Boys are still discouraged by adults from reading women authors and stories with girl protagonists and that discouragement usually continues into adulthood for most of them. It’s changing but very slowly. And men in general are discouraged from reading for pleasure or information outside work/school as indulgent and frivolous — a perennial problem for book publishers, especially in fiction, and one of the reasons (the others being sexism) why they prioritize and center men readers. Men are less likely to consider fiction works they like literary unless they prize being highly educated as a status– which many men are discouraged from favoring. And men who do prize being highly educated as a status also tend to prize elitism as a value — whether they are liberal or conservative politically — and so are less likely to fit the profile the scientists were asserting for those who read literary fiction.

    So if you had a sample that was mainly women, whether or not they chose literary classifications for what they read, you would very likely find that they are more open to others, diversity of culture, recognizing of differences, etc., just because they are women and socialized to have those views. If you split up the women and the men into two groups, the women will “win” for those values, especially if it’s a group that is not majority white cishet women. The best you could probably do is have two groups of women, those who self-identify as literary readers and those who self-identify as non-literary readers, and see what you get from each group. The group of women who then self-identify as literary readers might very well show up as less open to diversity, etc. than the ones who do not, because there is a slight status attachment to being a reader of literary fiction, though fiction readers in general do not attach status to buying and reading individual titles.

    And you’d need to exclude non-fiction works from the list because non-fiction — which are all works of reference information (even the essays and memoirs) — rarely ever have a literary designation applied to them, are deeply different areas of interest and likely skewed the results of the study further. Men are socialized/encouraged again to read non-fiction if they are going to read, so the fiction results in that study were probably majority women.

    The main problem is simply that literary is a completely subjective term, a term of praise for written works, mainly fiction or non-fiction story telling, and one laden with cultural influences and social status markers that change over time and by location. Too many variables to try to correlate it with political-social values, at least in the way that they are attempting.

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: