An analysis of upsets in quizbowl

How to measure an upset?

What is an upset? Coming into 2017 DI ICT, Northwestern was not considered a top-tier contender, yet made it all the way to the final before falling to Michigan A. Was Northwestern’s run through the top bracket due to a string of upsets, or were they just underrated coming into the tournament? Given how much team strength can change tournament-to-tournament due to roster changes or intensive studying, for this analysis I think it makes sense to stick to in-tournament performance as a measure of team strength¹. That is, look at how both teams performed in the tournament by a given statistical metric, and count the match as an upset if the team who is worse in the metric won the match.

Then, which metric to use? Ideally you want the metric that most accurately measures team strength, which, in aggregate, means you want a metric that produces a low number of upsets. If a metric is very good at predicting most matches, but fails on a small number of them, it’s a good sign that those matches are upsets of some sort.

The metrics I considered using for this analysis were: points per bonus, bonuses heard per tossups heard (essentially, what percentage of tossups is a team getting), and A-value². To decide between them, I looked at how accurate they were in predicting the outcome of matches in the dataset, and picked the metric with the highest accuracy.

As you can see, A-values are more accurate than either points per bonus or bonuses heard per tossup (which makes some sense, as A-value incorporates both).

How frequent are upsets?

Not all upsets are created equal. Take the example of March Madness, where a 10-seed beating a 7-seed will barely register, but a 15-seed beating a 2-seed³ is a historic upset. So we should be asking not only “How frequent are upsets?”, but also “How frequent are upsets when teams have at least an [X] gap in team strength?”

Unsurprisingly, upset chances monotonically decrease with A-value margin⁴. You can think of A-value margins as equivalent to PPG margins against common opponents. There’s not an exact comparison to PPB margins, since doing that would require assuming the teams are equivalent in tossup skill, which is rarely the case if there’s a PPB gap.

Examples

To illustrate the degrees of upsets we’re discussing, here are some examples of games that were considered upsets at different levels of difference in team strength. The entire game dataset with team A-values, PPB, and tossup stats can be found at this link.

What about difficulty?

One of the claims I was interested in investigating coming into this piece was that harder tournaments have fewer upsets, since the increase in difficulty (and generally longer questions) allows for finer differentiation of skill level. I was skeptical, since questions are written with the potential audience in mind – sure, ACF Fall wouldn’t necessarily differentiate Stanford and Berkeley, but it does differentiate novice teams – and so looking at upsets measures only how appropriate sets are for their field, not some hypothetical other field.

Somewhat surprisingly, upset rates are very similar across all difficulty levels.

DI ICT vs. ACF Nationals

It’s long been common wisdom among involved college quizbowlers that DI ICT is more prone to upsets than ACF Nationals: shorter questions, a game clock, and a distribution heavy on geography and pop culture are the most proferred explanations. But is it actually true that ICT is more prone to upsets?

Instead, I’d offer that DI ICT does actually test skill with a similar level of effectiveness to ACF Nationals, it’s just a different set of skills. When people think of how strong a particular team is, they tend to focus on the team’s ability on an ACF distribution. That undervalues teams with strong NAQT players and can cause some matches to seem like upsets when in fact the team that won was just better at the NAQT distribution/format.

Conclusion

In the course of my analysis, I was not able to find a broad pattern in the occurrence of upsets, whether by the dimensions examined above, or year-over-year (below). This is a remarkably consistent result. I’d be interested in ideas about other ways to look for patterns in upsets or ideas about how to transform the data, but as of now the rate of occurrence of upsets seems to be a very fundamental and stable part of college quizbowl.

Addendum: Upsets by round

By request of Mike Cheyne, I also looked at whether upsets are more common in Round 1 of a tournament (very slightly!).