The EPH Analysis

An analysis of proposed new Hugo voting rules is out. It’s disappointing to some but I think it validates the change to EPH.

The story so far:

In response to the Sad Puppy/Rabid Puppy slate of the 2015 Hugo Awards, a voting system called E Pluribus Hugo was proposed and passed at the 2015 Worldcon Business meeting. The system used a process of weightings and elimination rounds to make the nomination process have more proportionality without changing the basic mechanics of how people nominate things.

Much thought and tinkering was put into EPH but what it lacked was real data. EPH should make the list of finalists more proportional to the underlying groupings of voters. However, that meant that the impact of EPH couldn’t really be known without knowing to what extent Hugo voters clustered around choices anyway. Without slates, do Hugo voters form natural groupings (perhaps along sub-genres or sub-fandoms) or are they just a noisy mess of stuff? Without real data there is no way of knowing.

While EPH was passed at the 2015 Worldcon Business meeting, it requires ratification this year to come into effect. As part of that process an analysis of the 2015 and 2014 nomination ballots has been done and the results are just out…

What it all means…

I don’t know. No, that isn’t a useful reaction. OK, I’ll try again.

Below is a list of possible talking points, reactions, counter-argument things. I made them up. They don’t necessarily reflect actual people’s views (I’ll say when it does). Bold represent a possible reaction (not mine) and not bold is my response.

I’m also a hostage to fortune because more results are coming – post the Hugo ceremony, data on the 2016 nominations will come out and who knows what that will show.

For a different take try Nicholas Whyte http://nwhyte.livejournal.com/2707679.html?utm_source=twsharing&utm_medium=social

2015 results show that EPH doesn’t fix the slate problem!

No one thing can fix that problem. However, in most categories, at least one additional non-slated works made it onto the ballot with EPH. That means, probably, instead of No Award winning several categories in 2015, a worthy finalist would have won instead.

EPH+No Award together produce a strong disincentive to puppy-style slates. Slate voting will produce legitimate votes and so is bound to have some impact. The combination of EPH and No Award means that a slate will find it hard to sweep a category and win a Hugo.

EPH doesn’t stop those slate-inclined who just want to get to be a finalist and don’t care about winning!

True, but that was a given. Get enough votes and you get to be a finalist. EPH does demonstrably reduce the chance of that succeeding for a slate of nominees but it doesn’t do anything about a single nominee. Again, get enough votes and you get to be a finalist. The only guaranteed way of stopping that is to create a wholly different kind of award.

There is a non-puppy related change in 2015 Best Graphic Story!

That is interesting. With EPH instead of Sex Criminal 1 getting nominated Schlock Mercenary gets to be a finalist.

Sex Criminals got 60 noms in total and Schlock Mercenary got 51. However, Sex Criminals must have been more clustered with other nominees (such as Saga?) and hence lost out a bit to Schlock Mercenary.

With only one slate nominee, this was an interesting category. I liked Sex Criminals, but I think this is a positive demonstration of EPH. It should result in more variety of nominees without slates.

They didn’t include Best Dramatic Presentation!

The reason the report gives is this:

In testing, it was identified that the results in two categories (Dramatic Presentation, Long and Short Form) were usually producing results with many nominators submitting matching entries to other nominators. This was more due to the smaller pool to nominate from compared to other categories than any external coordination of nominating ballots. As such, we decided to produce results with these categories excluded, as changes in the dramatic presentation categories aren’t as useful for gauging if EPH is acting as appropriate where desired as the other categories would be.

That seems silly to me. There are lots of reasons to expect more organic coordination of ballots in these categories, and seeing how EPH works in that circumstance is useful as a way of comparison.

I hope they change their minds at some point.

A single coordinated minority of less than 20% would still average controlling over 80% of the ballot!

Aside from the exclamation mark, that is a direct quote from the report. This appears to be true but controlling only 80% of the ballot is enough to kill Puppy-style slates without having No Award win multiple categories.

Killing the incentive to use the 2015 Puppy slate tactic is what EPH needs to do. It will do that.

EPH+ would be better!

Probably yes, but I don’t know if other side effects (see below) would be worse.

2015 Puppy-style slates are last year’s problem. EPH doesn’t deal with THIS year’s problem!

True. However, the structural weakness of the Hugo voting system exists regardless and the cat is out of the bag. Others can try to game the Hugo Awards in the same way and perhaps more covertly.

As for the griefing style tactics of Vox Day, I think that needs a qualitatively different approach but that is an argument for another day.

EPH knocks out a potential winner in 2014!

There are few changes to finalists with the 2014 data. I think that confirms that without slates EPH will tend to deliver similar results as the current system. However, what isn’t guaranteed is that the results must be exactly the same.

In 2014 three results are notable.

  • Firstly Best Editor Short has a swap of finalists in the last spot – Sheila Wiliams (86) swaps with Bryan Thomas Schmidt (80).
  • Pro artists also has some swaps, partly because in 2014 a tie for fifth place meant 6 nominees. Essentially four artists with 50, 49, 49 and 48 nominations end up with a different ordering with EPH. The EPH ranking ends up as 49, 48, 49, 50 and that looks fine to me because I have that special kind of innumeracy that results from being overly numerate.
  • Fancast has the most understandable change but also the most problematic. In 2014 this was a three-way tie for last finalist at 35 nominations each. EPH breaks the tie and resolves the issue with a single nominee. Unfortunately, one of those three (SF Signal Podcast) won and would have been eliminated by EPH.

The thing is these are all pretty much very close votes with smaller numbers of voters. Anything different about 2014 would probably have resulted in different outcomes. For the Fancast result, an internet outage or a sick cat could have ended up with a different result. The least error in collating the data could have ended up with different results.

Put another way: Hugo voters did not have a clear consensus of which of these people/works should have been nominated. These cases are not good arguments against EPH.

Yes, but, but EPH+ might make that problem worse!

I’ve really no idea. I guess it might broaden what we might think of as a marginal tie and lead to more notable discrepancies between the number of nominating ballots and grabbing that last spot in the finalists. I don’t know.

The current system doesn’t avoid this issue, it really just hides it. For some categories, there are finalists who we really can’t say are substantially more nominated than others. The differences are small enough to be down to happenstance. And yes, some of those may actually end up being winners.

I think the answer is the number of nominees needs to be more flexible than just 5. However, deciding the rules on when to expand the number of nominees beyond an exact tie is unclear.

Where nominator coordination is not present, there are still significant numbers of changes not only to the long list, but to ballots where it’s not generally considered for anything untoward to have happened. Items removed from the 2014 ballot included a
winner of the Hugo. Had EPH been in place, they would not have been on the ballot.

That is a direct quote from Dave McCarty’s conclusion on the report. Sorry, but that is a flawed counterfactual. If we could somehow rewind the tape back to early 2014 and re-run the 2014 nomination ballot again, how likely is it that we’d have ended up with that exact tie that occurred? EPH changed the result because it broke a tie and the other places where there were changes were also spots with very close votes.

Almost ANY change would have meant that something slightly different would have happened! For SF Signal not have been a finalist required ONE nominator’s vote to be different

The changes to the Ballot and Long list are not easily verified and for people reviewing the detailed results at the end the only way to check that the process is working correctly would require access to secret nomination data and significant time.

That’s Dave McCarty again. Well, ANY verification of results needs access to ballots. Given Dave McC is worried about the 2014 Fancast result shifting by possibly one vote, to verify the CURRENT process would require checking that ballots had been classified correctly and counted correctly.

Assuming the underlying ballot data is correct (i.e. everybody’s nominations have been correctly collated) and in a machine readable form (e.g. a text file or spreadsheet), the EPH check takes seconds. Don’t trust the EPH program you are using? Use a different one and see if you get the same results. EPH is not hard to code, I made an Excel version that only uses standard Excel formulas and NO extra code at all.

So, yes, cleaning the nomination data and getting it all tickety-boo takes time – without a doubt BUT if we wanted to verify that the results DON’T CHANGE under the CURRENT process YOU WOULD STILL NEED TO DO THAT.

 

Advertisements

33 comments

  1. iamzenu

    EPH and 4/6 this year. EPH+ next year. Perfect is seldom required and never obtained.

    EPH needs a spokesman. I recommend Sam Biederman (Asst Commissioner, NYC, Department of Parks).

    Like

  2. Kendall

    Great write-up, @Camestros Felapton – thanks! I’m not really surprised at the hand-wringing about the results, but yes, expecting identical results, especially when the point is to improve results – and as you say, with ties and near-ties in the mix – seems silly.

    Like

    • Andrew M

      It was definitely part of the intention behind EPH that it should not significantly perturb results in non-slate situations (hence the test against 1984 data): Jameson Quinn assured me of this, and said that it could be formally stated and demonstrated (given certain assumptions, which turned out not to be perfectly true; but that was certainly the intention). If you value the Hugos because of the results they have historically produced, as many people do, this is clearly important.

      No,/i> perturbation is certainly too much to hope for, and indeed the 1984 results already showed that it wasn’t achievable; and in near-tie situations it’s clearly not achievable (though I wouldn’t call the Schlock Mercenary result a near-tie). But if there turn out to be significantly more changes than we expected, I’d say that was a problem.

      Like

      • camestrosfelapton

        The Schlock Mercenary result is interesting because it was in a slate year. Now, I know Puppy voters have mixed views on it, but it ticks some Puppy boxes. So I suspect it got a fair number of non-slate Puppy votes (i.e. some people voting the Puppy slate nominating it off-slate).

        Like

      • Kendall

        EPH doesn’t know what a slate is. It tries to improve representation so it’s more proportional (as I understand it). So to think that a non-slate year would never ever come out differently seems a bit naive, given EPH isn’t a sentient being – it’s merely a way of tallying votes.

        I obviously can value historical Hugo results (winners and/or short lists) without feeling they’re perfect, and without being bothered by minor differences. Some of which seem to be in close races anyway.

        Like

  3. Mark

    What’s particularly interesting to me is looking at the EPH points vs original nominations, and how far EPH reduces the effects of slate nominees. Looking at 2015 Best Novel we get reductions in the power of the top 5 slate noms of between 45%-70% (From Butcher and Larry down to Gannon) whereas non-slate noms see reductions of 18%-37% (from Liu Cixin down to Robert Jackson Bennet) (figures done very swiftly on calculator, may be horribly wrong!). Looking at it in this very basic way, EPH’s mechanics had at minimum twice the effect on slate noms as it did to organic noms.
    While there’s definitely a “does it do enough” problem, I think it’s clearly demonstrated that EPH is working correctly to reduce the power of slates.

    Like

  4. thephantom182

    Shorter conversation:
    Larry Correia said the WorldCon voting for the “most prestigious award in SF/F was inbred at best and crooked log-rolling at worst. So he discovered anybody can get a WorldCon membership and vote. For three years, he made a few blog posts about this curious fact on his own web site, and encouraged people to join and vote. Which they did.

    Larry Correia was solely responsible for bringing increased participation from brand new people to WorldCon, and with it, new money.

    The WorldCon Fandom response was:
    Year One: distaste. Who let these assholes in?
    Year Two: outrage! How dare these barbaric racistbigothomophobe Puppies soil our hallowed halls?!
    Year Three: Catastrophe!!!!!! Man battle stations!!! Repel boarders! Unleash the Kraken, Noah Ward!

    Year three, as predicted, WorldCon moved to change the voting rules, so as to exclude the eeeevile new participants and render their votes moot. But they will still take our money, thanks so much. Some call such things “disenfranchisement”, you boys call it “EPH”. A rose, by any other name, is still a prickle bush.

    This year of course, Year Four, has been all about the Rabids kicking your asses. More agitation, more shrieking, more stupid tantrums on Twitter, more Very Important People in SF looking like a pack of small-souled, intolerant political partisans.

    All because I and others decided to spend $40US last year and cast our one little vote each. The Hugos have gone from “Most Prestegious SF Award” to “those sjws on twitter” in four short years.

    That’s what Kate McMillan calls “showing up to riot.”

    The only question of any importance this year, is how many Hugos go to Noah The Destroyer.

    Like

    • iamzenu

      I am trying to think of a short but gentle summary. Hmmm. Let’s see…

      Larry wanted a Hugo because he said it was THE award, couldn’t get one and got pissed off because nobody wanted to hang with his crazy self. So he decided to piss in the punch bowl and then make up some self serving drivel. After all, there are some really whacked wingnuts out their and somebody will be stupid enough to buy into it. That’s his whole marketing approach. It’s niche marketing and a niche most people don’t want and those that do are worse writers then he is. He does after all write decent gun porn. Now it is starting to constrict his market so he backed out of the whole dumb dog movement and left it with his ally Vox Day who is even goofier than Larry. That’s about it.

      Loved last year Hugo presentation. Fans gave the dogs a middle finger yet again. What’s not to love about that?

      Like

      • thephantom182

        Don’t think so, Camestros. This it the telling part right here:
        “Loved last year Hugo presentation. Fans gave the dogs a middle finger yet again. What’s not to love about that?”

        That’s what I said. Thanks for the confirmation. That’s the Reality (TM) situation. The alternate universe you boys inhabit is where this is all a conspiracy.

        ‘Fans’ have been beating the ‘diversity’ drum a long time. WorldCon is supposed to represent -all- the fans. It’s a fetish, really. Y’all never shut up about the Diversity.

        Yet it has been very unreceptive to the notion that there are some fans, such as my own humble self, who felt unrepresented. So we voted, and got the middle finger. A very loud, very energetic middle finger.

        Almost as if our tastes and our views were not only unwanted, but were anathema to the reigning group. Almost as if all that ‘freedom of expression’ and ‘diversity’ stuff was a smoke screen to fool the rubes. Like you didn’t mean any of it, you just said it so you’d look good.

        Like

      • iamzenu

        @phantom, what color is the sky in your alternate reality?

        Nobody cares what you like to read or don’t like to read. It’s just that the dogs are assholes. Nobody likes crazy assholes. If you count yourself as a stupid dog and think the finger was for you, fine. If you don’t like it, stop being an asshole.

        Like

  5. JJ

    phantom182: a bunch of blathering which bears no relation to reality

    Year One: Larry Correia makes pathetic blog posts begging fans to turn out and get him a Hugo. It doesn’t work. Worldcon members’ response is to generally ignore the jerk begging for votes.
    Year Two: Larry Correia escalates his assholishness by putting together a slate of other works and still attempting to get himself a Hugo. It doesn’t work, but his followers do manage to get some mediocre crap put on the ballot. Worldcon members’ response is, okay, they’re being assholes, but let’s just try to ignore that and get on with celebrating good SFF.
    Year Three: Brad Torgersen and Vox Day in a coordinated campaign escalate the assholishness to an epic level by insulting Worldcon members repeatedly, then persuading their brainless Puppy sycophants to game a bunch of mediocre-to-execrable crap onto the Hugo ballot. Worldcon members’ response is okay, we’ve given these people a chance to behave like decent human beings and instead they just got worse, so we’re going to remove the loophole that they are shamelessly exploiting to cheat things onto the Hugo ballot — and also, we sure as hell are not going to give rockets to that crap (Where IS all this good SF by conservatives that they claim is being overlooked? Because they sure didn’t put any of it on the ballot!). Puppies’ reponse: NO FAIR! You were supposed to give awards to the crap we nominated! BT and VD knew, but didn’t warn us, that our cheating would be for naught!
    Year Four: The expected continued Puppy assholishness occurs, Worldcon members roll their eyes and get on with celebrating good SFF while continuing to fix the loophole that the Puppies are shamelessly exploiting to cheat things onto the Hugo ballot. The Puppies continue to whine like the big whiny babies that they are — all because they are spoilt children with an unjustified sense of entitlement who think that they should be allowed to dictate to other people what sort of SFF they should be allowed to like.

    Liked by 1 person

    • thephantom182

      This is another interesting alternate reality. “Brad Torgersen and Vox Day in a coordinated campaign…”

      Uh, no. Do you guys really believe that stuff? Or do you just keep saying it because the truth is too hard to take?

      Like

  6. JJ

    thephantom182: This is another interesting alternate reality. “Brad Torgersen and Vox Day in a coordinated campaign…”
    Uh, no. Do you guys really believe that stuff? Or do you just keep saying it because the truth is too hard to take?

    Actually, several Puppies themselves admitted this later on, claiming that Brad and Larry had talked VD into walking his slate back from where he had wanted it to be. Surely you’re not admitting that you actually have no idea what the Puppies were doing last year?

    Like

    • thephantom182

      “Actually, several Puppies themselves admitted this later on, claiming that Brad and Larry had talked VD into walking his slate back from where he had wanted it to be.”

      I heard that Queen Elizabeth II called Larry up and put the whole thing together with Vox and Brad T, JJ. You are so out of the loop, man.

      Like

  7. JJ

    Well, Puppies wouldn’t be bragging that Brad and Larry had talked VD into walking his slate back from where he had wanted it to be, if it wasn’t so, would they?

    Or are you admitting that Puppies are blatant liars?

    Liked by 1 person

    • thephantom182

      I think my baseless, made-up rumor about Queen Elizabeth carries exactly the same expectation of validity as that stuff you said. Some guy calls himself a Puppy and says a bunch of stuff on the web? Wow! Totally legit, bro. Ten points for research.

      Note that I’m assuming, for the sake of argument, you didn’t make yours up like I did. I’m magnanimous that way.

      There is no Sad Puppies ‘organization’, at least not one that contacted me and told me how to vote. There’s a few posts on a blog, nothing you can sign up for and belong to.

      You’re objecting to random private citizens like myself who paid money, joined WorldCon and voted. Just contemplate that for a few moments, eh? Lot of uproar over a difference of opinion. Somebody might even think there was money involved.

      Like

  8. iamzenu

    So… tonight is the night.

    One thing I have in common with Phantom is that I am not an SFF Fan (capital F). I like SFF just fine. I just don’t like it as much as most of you guys like it. Just as Phantom has his culture war that happens to cross path with Larry’s culture war front of the Hugo Awards, I have a wide interest in lit that crosses multiple genres – including SFF.

    I tuned into the Hugo Awards for the first time last year. Lost in the diminishing returns of trying to hold rational dialogue with dumb dogs, is that it was a really good show last year. I watched alone last night but tonight I think we put it on the big screen for both the wife and I. It is streamed.

    Because of some of the real fans, I got to read some really good SFF including some of those nominated this year. And some nominated in prior years. And some that were not nominated because pups pushed them off the ballot (“Lock In”, “The Martian”). So thanks Fans for letting me into your world. And for putting on a great show. And for introducing me to some good stories. Looking forward to this evening.

    After that, looking forward to the business meeting which is also streamed.

    Like

  9. Andrew M

    I am a bit worried about the Schlock Mercenary result. Sex Criminals was a new and distinctive thing, interest in which had been propagated by recommendations and fannish buzz. Schlock Mercenary was a well-established series with a following, which had been nominated several times before. I think it’s on the whole better that things of the first kind get nominated. The data we have are not enough to allow any generalisation, but if EPH regularly has effects like this, I think it would be a problem. And one can see how it might; it’s not implausible that nominators who survey the field and seek out new stuff would be more clumped than those who vote for their established favourites, though obviously less clumped than slate voters. So while EPH would lead to more variety in the sense that more groups of voters get represented, it might lead to less variety of content – because the voters who tend to converge are the ones who are actually seeking variety.

    Like

  10. Andrew M

    Well, yes, but is representing the spread of taste and choice the only thing we’re aiming at? This isn’t an actual election; it’s a process to find candidates for the final Hugo vote. The final vote is set up so as to favour things with wide appeal, rather than things which just have a lot of fans. I think it’s reasonable that the nomination round should be directed to finding plausible candidates for that – and convergence, if it’s real and not conspiratorial, would tend to point out plausible candidates. If the governing aim is to represent a greater spread of voters, that will tend to favour writers and series who have a devoted following but less broad appeal.

    I’ve always thought it important that the nomination system should not penalise convergence on excellence. I believed that EPH did not (at levels where it is likely to happen, at least: obviously if people converged with unanimity like that of a slate, they would be penalised, but that’s never going to happen). This result makes me less certain. EPH is still an improvement on the existing system in the current circumstances, but as a long-term solution I’m doubtful.

    Like

    • camestrosfelapton

      Well it isn’t the only thing we are aiming at but I think it is a good test of a system for coming up with nominations that you end up with finalists that give a strong sense of what people are reading.

      Personally, what I think would be better regardless of the voting system is that the number of nominees is connected to the range of support they collectively represent. i.e. when there is less consensus there are more finalists.

      Like

  11. thephantom182

    I see that the Mad Genius crowd has a new term for EPH: Calvin Ball! That’s what I said, but shorter. Awesome! Calvin Ball it is.

    By the way, Camestros. I’m noticing some serious rude word name-calling action here. Is it that some animals are more equal than others? One definitely gets that impression.

    Kind of like how WorldCon is happy to take my $40, but not so happy that they’ll give me a vote that’s the same as yours. Under EPH, anyway. My vote’s worth depends on what I vote for. That would be a very strong disincentive to spending money on something, leaving the gate open for a guy who can afford $8000 to buy the 200 votes he needs to win his category. Somebody with a $3 million book deal could do that every year and come out ahead, couldn’t he? Or she, no need to be sexist I suppose.

    Is a Hugo worth $8K in book sales? Might be, huh?

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s