Rugby: The Wisdom of Crowds and Super 14 Picks

In his book, The Wisdom of Crowds, James Surowiecki argues that the the aggregation of information in groups results in better decisions than could be made by a single member of the group.  I have to admit that I haven’t read the book, yet, but I thought I would test the wisdom of rugby followers who participated in the Sky Sport Virtual Rugby game that ran for the 2009 season.

The hypothesis would be that the aggregation of picks would tend to be pretty close to actual results.  However, we need to look at the criteria that separate wise crowds from irrational crowds, as set out in the Wikipedia entry on the book and subject:

Diversity of Opinion: each person participating will tend to have private information, which will (primarily) be their own eccentric interpretation of known facts

Independence: Not so sure about this one, since I am sure that some people (e.g. me) were influenced by the views of others as reflected in the level of support for particular results, especially where they don’t have any strong eccentric interpretation of their own

Decentralization: People do specialize and draw on local knowledge, but the other side of this coin is that they may well support an outcome favouring their local side irrespective of the facts and experience

Aggregation: Sky Sport Virtual Rugby provides the mechanism for turning private judgments into a collective decision

On this basis, the elements for a wise crowd appear to be there in Virtual Rugby.

Bad judgments can result when the crowd is:

Too homogenous: if there is not sufficient diversity within a crowd – in the case of Virtual Rugby it is likely that there is too great a focus on New Zealand teams, since the majority of the participants are (I assume) New Zealanders

Too centralized: I don’t think this is a problem since there were about 122,000 participants

Too divided: there is ample scope to share information through media reporting and commenting, and through other forms of information exchange, so people can choose what information they need

Too imitative: Choices are visible, in the aggregate, which could lead people to reflect the majority view

Too emotional: Well of course, clearly there will be biases, hearts will dominate over heads, Wellingtonians will support the Hurricanes, despite the evidence

Anyway, that’s enough about the theory, what abut the analysis.

Virtual Rugby is played by making a prediction for each game in each round of the Super 14, with choices being one side or the other to win by either 12 points or less, or more than 12 points, or a draw.

For the purposes of the initial analysis, I looked only at the outcome of the game and regarded a crowd prediction as successful if the actual outcome reflected the views of the highest proportion of players.  This includes predictions of both wins and losses.

On this basis, the crowd had a 69% success rate over the 14 rounds of the 2009 Super 14 season, i.e. it got slightly better than 2 out of 3 right.  Is this good?  I would have thought that it’s not so good.

A more detailed analysis by team and country is interesting:

Bar chart of success rate of crowd picks in Virtual Rugby, by team and country

The most predictable teams tended to be those that ended up in the bottom third of the competition, i.e. they were predicted to lose most of the time.  Not unrelated, given that they had three teams in the bottom third, the crowd did best with South African sides (75%).  It did worst with New Zealand sides, which could well reflect the emotional attachment to local teams.  The most unpredictable teams were the Crusaders and Waratahs, which can possibly be explained by their changing fortunes through the competition.

What about picking winners. The analysis shows that the crowd got it right for 80% or more of the time for the Hurricanes, Stormers and Blues, but less than half the time for the Reds, while crowd opinion was definitely against the Cheetahs.  It could also mean that Hurricanes supporters always pick their team to win.

When picking losses it was a bit of a mixed bag.  The crowd got it right for the Sharks, Bulls and Lions, but was nowhere near it for the Hurricanes and Waratahs.  I think that this supports the view that supporters of these teams let their emotion cloud their judgment.

Did the crowd learn anything as the season went on?  Not sure about that, although there is some evidence to support the proposition in that for the last three rounds at least the outcomes were well-predicted.

So what does this all mean, apart from a suggestion I have too much time on my hands?  What I think it means is that the crowd that does Virtual Rugby is reasonably well-informed but is not too wise. It also means that if you want to win at Virtual Rugby, don’t follow the crowd.

Further analysis comparing the crowd verdicts to the TAB betting odds would be very interesting.