How many finalists? Crunching continued…

This is a follow up to the earlier post. Read that post first for background and the data I’m looking at.

I’ve looked at 2018 Hugo data for both stages:

  • The nomination stage by EPH
  • The final voting stage by IRV

My impression was that there are some changes in the ranking between the two but not so many as to cast doubt on the nomination process itself nor so few changes as to make the final voting stage redundant. It looks like things are pretty much in a sweet spot:

  • final winners are often the top finalists — which implies there’s not a mismatch between how people nominate and how they vote (or between the people voting at each stage etc)
  • low ranked finalists often do better in the final voting — which implies that there is a lot of value in a two stage process.

To show that here is a graph of how the rankings compare between EPH stage 1 and IRV stage 2 of the Hugo voting process:

The width of a blob indicates the frequency of that pair of ranks. For example there were 9 cases of 1st rank EPH coming 1st in the final stage and 10 cases of 4th ranked finalist coming 3rd in the final stage. I’m not sure if a simple linear regression is appropriate with this data but Excel tells me that the first stage voting accounts for about 25% of the variance in the second stage ranks.

However, can we look at this data and say how long the finalist list should be? Are there ENOUGH finalists? Should there be a list of 7 or 8? Putting administrative and practical limits aside I think we can examine this question with the data.

Obviously, I’m only looking at one year, so any conclusions are tentative and limited. I could look further but recent data is weird due to Puppy activities and there have been rule changes since. So, I’m sticking with 2018 (also I’m lazy).

One graph I drew was to look at the distribution of the differences in rank between the two stages.

Again we can see that no change (zero on the x-axis) is common but that bigger changes in rank happen. Unfortunately, we really can’t take this as being true of every ranking. Obviously rank 6 finalists can only either stay the same of go upwards.

A different way of thinking about the issue would be to consider what would happen with different number of finalists. For example, what if in 2018 there was only 1 finalist per category? Yes, that’s silly be we can work out that of the 15 categories I looked at, 9 would have the same winner as what actually happened and that 6 wouldn’t. 1 finalist would contain 60% of the actual winners.

  • 1 finalist: 9 or 60% of winners
  • 2 finalists: 12 or 80% of winners
  • 3 finalists: 14 or 93% of winners
  • 4 finalists: 14 or 93% of winners (i.e. no extra winners)
  • 5 finalists: 15 or 100% of winners
  • 6 finalists: 15 or 100% of winners

So for most categories 3 finalists would just about do. Adding finalists after 3 brings only small gains but 2018 still need 5 finalists to capture all the eventual winners.

Now, obviously, if we added more finalists people’s choices and the voting would change but we can see from the trend that the gains trail off quickly after 3 finalists.

So is five enough? Five clearly works but that’s actually an argument for having six finalists if you want to be confident you’ve got all the plausible contenders. As we definitely got one fifth ranked finalist winning a category (Rebecca Roanhorse in the Campbell Award) there’s maybe a 6% chance of rank 5 finalist winning (one winner out of 15).

Add in the possibility of one finalist being in some way dodgy or have cheated etc then 6 is a safe contingency. Does the same argument not work for 7 or 8 finalists? No, because we can see that the gains trail off rapidly after 3 finalists. Five is probably enough, six is almost certainly enough.

Crunching reform or rollback

There is an on-going discussion at File770 on the 5/6 Hugo nomination rule:

While the Sad and Rabid Puppies slates were filling up most of the slots on the 2015 and 2016 Hugo ballots, majorities at the Worldcon business meetings passed and ratified several rules changes that made it much more difficult for that to keep on happening. The success of these majorities has tended to overshadow how many fans did not want any changes made – no matter how often Vox Day dictated what made the ballot – or else did not want these particularchanges made. And there are business meeting regulars who evidently feel now is the time to start turning back the clock.  
Here’s a matched set of proposals to end the “5 and 6” part of the Hugo nomination reforms. If you are going to the Dublin 2019 business meeting, you will have to decide whether the claims made about convenience and efficiency warrant undoing the protective rules put on the books just a few years ago.

http://file770.com/reform-or-rollback/

The proposal states that:

“The losers will be those who had placed sixth in recent years. There is only one case of a sixth-placed finalist at nominations stage going on to win the Hugo in the last three years (the rather odd situation of Best Fan Artist in 2017, where two finalists were disqualified). On the other hand, a reduced pool of finalists increases the cachet of being among that number.”

I have some doubts about this point. Firstly, 2017 and 2018 isn’t a lot to go on and 2017 still had some residual Rabid Puppy action and hence isn’t a great example for 6th places. We really only have 2018 as ‘regular’ year of the two big voting reforms EPH and 5/6.

I won’t rehash all the arguments from the File770 discussion (at least not yet) but I did want to look at the specific issue of how likely is it that a 6th place nominee might win the Hugo in their category.

Obviously, there are zero examples of this from 2018 but it would be wrong to infer that the answer is therefore zero chance. Instead, I decided to look at how ranks change between the EPH nomination stage and the instant run-off voting (IRV) final stage.

To do that I looked at the nomination rank (EPH) and final rank (IRV) of Hugo and Campbell nominees from 2018. I discarded categories which had declined nominations because I felt they might have weird impacts. Here’s an example of the Novel data:

IRV EPH Dif Mag Finalist Category
1 1 0 0 The Stone Sky Novel
5 2 -3 3 Raven Stratagem Novel
4 3 -1 1 Six Wakes Novel
3 4 1 1 Provenance Novel
2 5 3 3 The Collapsing Empire Novel
6 6 0 0 New York 2140 Novel

In the example: IRV column shows the rank of the work through the elimination process; EPH shows the nomination rank; Dif is EPH minus IRV (negative means the work was less popular in the 2nd stage); Mag is the magnitude of the change regardless of direction.

The average difference has to come to zero (everything balances out) but the (mean) average of the magnitude comes to 1.27 i.e. on average finalists shift about one place from first round to second round. Of the 90 finalists listed 25 had no change, 34 changed by 1 (i.e. the modal change), 19 by 2, 7 by 3, 4 by 4 and only 1 by 5. That last change was a drop from 1 to 6 rather than a rise but does demonstrate the scale of possible change.

How about 6th placers in general? The magnitude of the shift for those ranked 6th in nominations was 1.2 but that was also the average of the difference (i.e. with direction). Of course, if you are in 6th place you can’t get a negative change in your rank in the second stage because you can’t get lower (assuming you don’t get No Award of course and I didn’t model that).

Of the 15 6th placers I looked at, 5 didn’t shift at all, 4 shifted up by 1, 5 shifted by 2 and 1 shifted by 4 (Sheila Williams in Best Editor Short).

I’ll put all the numbers after the fold but I think the figures point to it being unlikely in general that a 6th placer will go on to win in the second round but not so unlikely that we won’t see it every so often.

Data in EPH rank descending order after the fold.

Continue reading “Crunching reform or rollback”

How To Edit This Year’s Hugo Novel Finalists Into One Giant Voltron Like Book

Obviously, the best way of approaching the six Hugo finalists is as seperate books with their own distinct plots and characters. However, imagine the book world was under attack by some giant monster and all the books had to team up to fight it, how would that work exactly?

A reader who has considered the matter carefully might say “What are you talking about? Did the fumes from cleaning the cat’s catnip vomit go to your head?” The answer to those rhetorical questions are “see above” and “yes” and “why does the room keep wobbling like that?”

Where to start?

Luckily both New York 2142 and Six Wakes have connections to New York in the nearish future. The Collapsing Empire, Provenance and Raven Stratagem all have space empires in different phases of development. The Stone Sky has to team up with the whole of the Broken Earth series first but its post-apocalyptic back story puts the story way into the future.

So for a sequential order of a giant history of humanity try this order:

1. New York 2142 – in the near future humanity struggles with the impact of climate change

2. Six Wakes – shortly after which humanity develops a unique ‘cloning technology’ and begins space exploration

3. The Collapsing Empire – humanity has continued to colonise space but it’s method of interstellar travel has restricted its capacity to grow

4. Provenance – (somehow) humanity has found a way past the technological limits of space travel and now is spread all over. A remnant of a galactic empire exists that makes use of the ‘cloning technology’ (see 2.) but that’s not important for this book.

5. Raven Stratagem – technology has advanced even further to a peak of reality bending. Unfortunately toxic empires have grown even stronger.

6. The Stone Sky – we return to Earth were hubris and human experimentation have left the planet as a tectonic mess

So what is a fan writer anyway?

I appear, for once, to have some sort of verifiable credential on the topic about which I’m writing but I also know that I don’t have a good grasp of the long history/tradition of fanwriting. What is a fan writer, what constitutes fan writing, is fan writing a genre of writing or does it come down to (lack of) money or to the channel of distribution? Also, what does the sport of rugby have to do with this?

The official Hugo rules actually say very little:

Any person whose writing has appeared in semiprozines or fanzines or in generally available electronic media during the previous calendar year.

It doesn’t say “non-professional” or amateur. There is an implication from other categories with “fan” in it that implies non-professional is a key element of the term “fan” but the inclusion of semiprozines belies that implication. A semiprozine may pay its contributors and hence according to the rules a fan writer could be paid for their writing.

But those are the rules and I don’t regard formal rules as a good source of meaning. Rules are useful for delimiting meaning when we need sharp categories to settle disputes but they aren’t the ultimate source of meaning — if they were we would have no way of judging whether we have written god rules or not.

I think there are a number of ways people parse “fan writer”:

  1. Writing by anybody in the role of being a fan
  2. Writing by a fan rather than a professional writer
  3. A person who writes for fandom
  4. A person who writes fannish things i.e. within the genre of fanwriting

John Scalzi’s Whatever blog or George RR Martin’s Not a Blog hit three of those four but not the second. I see some concern about fan writing including the blogs or other channels of professional writers but I think those two examples strongly hit other (perhaps more vague) senses of fan writing. Both Scalzi and Martin are heavily involved in fandom. However, the professional aspect is a sticking point for some. I’ll come back to that.

A different issue I’ve seen discussed is what people expect from fanwriting. For example, I’ve not seen anybody explicitly say that fiction does not count as fanwriting but when looking at examples of what people expect from fanwriting, fiction doesn’t count. Fanfiction in particular, despite clearly being non-professional and for fans and of fandom(s) is not typically recognized as fanwriting in the Hugo awards. More recently I saw, frankly puzzling, notion that fan-orientated journalism/news-curation etc such as Mike Glyer’s work at File770 isn’t fanwriting, whereas I’d see that as canonically fanwriting (i.e. if I draw a big loop around what might be fanwriting, that is safely in the heartland and away from the outer borders).

I think people have developed an expectation of fanwriting being non-fiction writing in an essay format of matters to do with science fiction/fantasy and in particular:

  • reviews
  • criticism
  • discussion of the state of the genre

Which, OK, can all be fanwriting but not definitively. I wouldn’t think of Damien Walters’s Guardian columns on SF/F as being fanwriting even though his columns there often hit at least two of the possible meaning I listed above. I note also, as The Guardian isn’t behind a paywall, those columns count as generally available electronic media.

Not-paid-for makes for simple criteria (although as I note, not actually in the Hugo rules) but I don’t like that. I’m unambiguously an amateur and not just in the sense of not getting paid but also in the sense that this blog is purely a hobby (and also in the sense that I make no effort to create a polished product!) But amateur status is the essence of privilege — this is a hobby I can afford to have in terms of time and money and security. I don’t have a lingering student debt, I don’t have to work a second job, I don’t have childcare commitments (and when I did I had no time for anything fannish). Trying to use not-being-paid as a criterion becomes terribly exclusionary as well as hard to police in the era of Paetrons, Kickstarters, etc.*

Likewise, the purity of motive in evaluating fan writing can become a pernicious form of gatekeeping. I don’t disagree with a point Patrick Neilsen Hayden is reported as making that fanwriting is a genre in itself and not a junior level of professional writing but it is also NOT-not a “junior varsity” either. In other words, fanwriting can be its own very loose genre AND also be an entry point. Again this mirrors an aspect of fan-fiction — it is a genre or species of writing in itself that requires its own skills some of which are not easily transferred to or from professional fiction writing but it is also a space where potential professional writers learn their craft or find their talent.

Looking at the different types of writing done by this years Hugo nominees for Best Fanwriter, I think the mix was pretty good. It shouldn’t only be essayists and reviewers but it certainly should include them and include people whose working may be partly monetized (either as gigs at paying sources or via online crowd funding options). As a category I think it is in an ‘if it ain’t broke don’t fix it’ state — a circular definition that fanwriting is what fans see as fanwriting works fine for the Hugos crowdsourced eligibility mechanism. My main dislike of “Best Fanwriter” as a category is that it focuses on the person rather than the work but “Best Fanwriting” would bring all the definitional issues above to the forefront and I suspect the category would collapse under its taxonomic weight.

Alternatives to fanwriting or additions to the category, might include:

  • A Best Related Work – Short Form as a space to reward essays, reports, speeches etc
  • A micro-ficition award
  • A fan-ficiton award

But, I can’t say I find any of those particularly compelling as ideas.

Wait! I promised some point about rugby at the beginning! Actually I know very little about rugby despite growing up in a family with deep connections to Rugby League and also being brought up in one of the strongest centres of Rugby League in the world and somehow now living as far away as possible from where I grew up and STILL live in a place that is mad for Rugby League. Anyway, Rugby League split from Rugby Union as codes of football because working class players needed some pay to keep playing. There’s a metaphor there about priviliege, the use of amateur status as a means of social exclusion and possibly something about cabbage ears. Luckily I don’t have an editor to tell me I need a better metaphor and a snappier conclusion.

Picking Through Hugo Numbers

Some stray observations from here: https://www.worldcon76.org/images/publications/2018DetailedResults.pdf

No Award

The spectral monster that both haunts and protects the Hugo Awards haven’t gone, it merely manifests less strongly.  No Award didn’t win anything (despite the claim that categories were ‘burnt to the ground’ in 2015) but still go some votes. Four ways of looking at it:

  1. The number of initial votes
  2. The final number of distributed votes after preferences
  3. The initial rank out of 7 it got
  4. The final rank out of 7 it got after distributed preferences
  • Novel: Initial# 42  Initial Rank 7 Final# 134  Final Rank 7
  • Novella: Initial# 40  Initial Rank 7 Final# 160  Final Rank 7
  • Novelette: Initial# 48  Initial Rank 7 Final# 177  Final Rank 7
  • Short Story: Initial# 44  Initial Rank 7 Final# 167  Final Rank 7
  • Series: Initial# 103  Initial Rank 7 Final# 175  Final Rank 7
  • Related: Initial# 45  Initial Rank 7 Final# 135  Final Rank 7
  • Graphic: Initial# 73  Initial Rank 7 Final# 131  Final Rank 7
  • BDP-Long: Initial# 19  Initial Rank 7 Final# 160  Final Rank 7
  • BDP-Short: Initial# 48  Initial Rank 7 Final# 162  Final Rank 7
  • Editor-Long: Initial# 72  Initial Rank 7 Final# 138  Final Rank 7
  • Editor-Long: Initial# 76  Initial Rank 7 Final# 110  Final Rank 7
  • Prof-Artist: Initial# 51  Initial Rank 7 Final# 85  Final Rank 7
  • Semi-pro: Initial# 70  Initial Rank 7 Final# 93  Final Rank 7
  • Fanzine: Initial# 81  Initial Rank 5 Final# 201  Final Rank 7
  • Fancast: Initial# 90  Initial Rank 6 Final# 115  Final Rank 7
  • Fanwriter: Initial# 87  Initial Rank 6 Final# 149  Final Rank 7
  • Fan artist: Initial# 56  Initial Rank 7 Final# 105  Final Rank 7

In all categories the impact of No Award was minimal. I’ve highlighted categories with above-average levels of No Award. There is a constant baseline of fortyish 1st preference votes for No Award in every category. That increase with less voted on categories, which has a double effect on the proportion of votes that go to No Award. However, after preferences, No Award doesn’t make many inroads.

Two notable categories are Best Series, which got the most ‘nopes’ from voters (but still not many) and Best Dramatic Presentation – Long Form which only got 19. Obviously there is a bit of a vote against Best Series (which I can sympathise with) but its not big and had zero impact.

EPH

EPH distributes nominations in a special way that should reduce the impact of a bloc of voters.

For Best Novel EPH had little impact on the top four nominees. However, it played a significant role in positions 5 and 6. Raw votes for the contenders for positions 5 and 6 were:

  • Collapsing Empire – 134
  • New York 2124 – 128
  • The Stars are Legion – 137
  • Autonomous – 136

Kameron Hurely’s and Annalee Newitz’s books would have been finalists in the old system. Interestingly the final comparison for sixth place ended up being between Raven Stratagem and  The Stars are Legion. It’s an interesting outcome but I think it shows EPH doing what it claims it would do – leading to a set of finalists drawn from a broader base of nominations.

Novella was more conventional, the top 6 raw vote winners were the finalists. Novelette had a raw vote draw for sixth place which was resolved by EPH. Best Series was a bit of a mess due to withdrawls and inelligibility. Other categories were largely dull aside from:

  • Professional Artist, were again there was some EPH action for 5th and 6th place.
  • Fanzine, were BlackGate was unlucky not to get a sixth place position.
  • Fan Artist, again some EPH shuffling of sixth place.

 

 

Post Hugo Post

So first off, thank you to everybody who voted for me. It really was special having Robert Silverberg present the awards. Sarah Gailey was a very deserving winner. I had a respectable showing but I guess the most elegant outcome would have been to have lost to No Award :).

Results here http://www.thehugoawards.org/hugo-history/2018-hugo-awards/ and here http://file770.com/2018-hugo-winners/ and breakdown of results are here https://www.worldcon76.org/images/publications/2018DetailedResults.pdf

There were plenty of surprises but my only disappointment was that The Divine Cities didn’t win Best Series. Bujold is hard to beat though!

I wasn’t surprised, on reflection, that Sana Takeda won best professional artist — Monstress is exquisite and was my top pick for the graphic story — but I was surprised that Victo Ngai didn’t do better.

The did-Uncanny-win-twice thing happened, once for Best Editors and once as Best Semi-prozine. It’s a bit like Best Film and Best Director at the Oscars — it’s not an award for the same thing but winning either has a strong implication that they should win both. The two awards have too much in common currently and it is Best Editor that needs to change. Of course, both Uncanny and the Thomas×2 were deserving winners and I don’t want to take the shine off their rockets.

Sad that Mike Glyer couldn’t make the awards. He also withdrew File770 from further nominations, which was very magnanimous. Winning a Hugo this time in the first puppy free year since 2012 was important.

I haven’t had time to delve into the nomination stats much yet. Interesting to see how EPH works. As a rough rule of thumb, categories where the red highlighted nominees run consistently stepwise along the bottom, are categories were I think EPH probably had little impact. Where there is a bit of a ‘shelf’ with red text over black, it’s more interesting.

In other non-news it looks like the Castalia House blog has started functioning again – which is dissapointing:)

 

Hugo Wild Guesses

It’s more fun to watch a competition with some bets on the action. I’ve no actual money on the results but if I was gambling these are my guesses. Note that these are not neccesarily what I voted #1 on my ballot or what I think *should* win, just what I’m guessing will win.

Hugo ceremony starts at 8 pm Pacific Daylight Time which is 1 pm (Monday) Sydney Time and 4 am (Monday) London.  Coverage details are here http://www.thehugoawards.org/2018/08/2018-hugo-ceremony-coverage-plans/

Best Novel

  • The Stone Sky, by N.K. Jemisin (Orbit) – it’s a solid set of nominees but no book really stands out from the set like the Broken Earth trilogy does.

Best Novella

  • All Systems Red, by Martha Wells (Tor.com Publishing) – I think this is almost the clear winner on lots of levels but “And Then There Were (N-One),” by Sarah Pinsker might win instead.

Best Novelette

  • Don’t know.

Best Short Story

  • “Carnival Nine,” by Caroline M. Yoachim (Beneath Ceaseless Skies, May 2017) but “Sun, Moon, Dust” by Ursula Vernon, and “Welcome to your Authentic Indian Experience™,” by Rebecca Roanhorse are about as likely I think.

Best Series

  • The Divine Cities, by Robert Jackson Bennett (Broadway) but InCryptid, by Seanan McGuire has the advantage of  McGuire’s on-going popularity with Hugo voters and a strong fanbase.

Best Related Work

  • No Time to Spare: Thinking About What Matters, by Ursula K. Le Guin (Houghton Mifflin Harcourt) but maybe  A Lit Fuse: The Provocative Life of Harlan Ellison, by Nat Segaloff

Best Graphic Story

  • Monstress, Volume 2: The Blood, written by Marjorie M. Liu, illustrated by Sana Takeda (Image Comics)

Best Dramatic Presentation, Long Form

  • Too hard to guess. It will be easy to pick a ‘why X won’ post-hoc in this category but at this point I’ve no idea.

Best Dramatic Presentation, Short Form

  • Black Mirror: “USS Callister,” written by William Bridges and Charlie Brooker, directed by Toby Haynes (House of Tomorrow). I wasn’t as enamoured of this as others were but it was a clever episode and Trek-nostalgia will give it an extra boost.

Best Editor, Short Form

  • Lynne M. Thomas & Michael Damian Thomas – Another good year for Uncanny

Best Editor, Long Form

  • Don’t know.

Best Professional Artist

  • Victo Ngai – spectacular covers

Best Semiprozine

  • Uncanny Magazine, edited by Lynne M. Thomas & Michael Damian Thomas, Michi Trota, and Julia Rios; podcast produced by Erika Ensign & Steven Schapansky

Best Fanzine

  • Don’t know – I really don’t know. I wouldn’t be surprised by any of the finalists winning.

Best Fancast

  • Don’t know

Best Fan Writer

  • Ha, ha, not saying 🙂 [not likely to be me though]

Best Fan Artist

  • Likhain (M. Sereno)

Lodestar Award for Best Young Adult Book

  • Summer in Orcus, written by T. Kingfisher (Ursula Vernon), illustrated by Lauren Henderson (Sofawolf Press)

John W. Campbell Award for Best New Writer

  • Not sure but I think Jeannette Ng or Rebecca Roanhorse