In an earlier posting, I looked at how the wisdom of crowds might apply in the case of the Sky Sport Virtual Rugby game to make picks in the Super 14 rugby competition. In that case, and looking only at outcomes, i.e. win/loss, the crowd had a success rate of 69% over the round robin stage of the competition. I suggested that this was not a great result. I might have to revisit that.
The Air New Zealand Cup round robin stage has now been completed, so I have looked at the figures for those games to see if there is any difference. In the Jimungo Virtual Rugby competition, participants pick the outcome (who will win) and a score level (12 point margin or less; moe than 12 points). A draw can also be picked. The published data include the percentage of participants who have picked each possible result, and it is this information, plus the actual outcome, that I have analysed. What I was interested to find out was the level of predictive success for both outcomes and margins, whether the crowd learned anything over the course of the competition, and whether any teams were more predictable than others.
The Air New Zealand Cup round robin stage is played by 14 teams over 13 weeks. The teams range from unions which provide a base for Super 14 teams – Auckland, Waikato, Wellington, Canterbury, Otago – to provincial unions, some of which have been struggling in financial terms, and until this year with declining attendances. An added fillip this season has been the planned restructuring of the competition for next year which is intended to reduce it to 10 teams, which has meant that those teams under threat of exclusion have had an incentive to succeed on the field and in boosting crowds. The perennial issue of whether and to what extent the All Blacks will be available for their provincial teams has also been aired again, while the continuing impact of the Ranfurly Shield cannot be ignored (except, it appears, by Wellington)
One hypothesis is that the Air New Zealand Cup, being a domestic competition, should mean that Virtual Rugby participants have more knowledge of the teams and players than may have been the case with the international Super 14. Countering this may be the greater degree of parochial attachment that such a competition provides, the relatively unknown status of many players and the extent to which removal of key players to international duties will affect team performance. Also, what should we expect in terms of crowd wisdom? Random picks should produce a 50% outcome when taken over 91 games and 61,239 participants (I have just tossed a coin 91 times and heads won only 45% but I’m not going to do it another 61,238 times), so crowd wisdom should do much better than that, but by how much?
So, what were the results. Well, the average success rate of predicting the outcome was 61.45%, which was less than the Super 14 result. For both outcome and margin, the success rate was 32.29%. Assuming that the crowd that participates in Virtual Rugby has more information than a random generator, is this sufficiently better than 50%? I don’t think so.
The chart shows the extent to which the outcome (win/loss) success rate moved over the season. The success rate got up to 90% for the penultimate round, and did improve as the season developed after an initial slump. The earlier rounds were characterised by “upsets”, and if defined as an outcome that fewer than 20% of participants picked, then there were 2 upsets in each of the first four rounds (out of 7 games in each round), as well as in rounds 7,8 and 13.
The higher success rates in the later rounds could reflect better informed participants, having more information on how the teams played, while other possible explanations could be that more All Blacks were playing for their provincial teams and strengthening the larger provinces with more favoured teams, or that the variable of the draw had come together to pit top and bottom teams against each other, with more predictable outcomes (not sure that this stands up to analysis).
The outcome and margin chart tells a similar story. In round 3 only 10% of participants got the outcome and margin right and in only one round did more than half of participants get it right.
So what about team support? Well, the participants in Virtual Rugby had mixed results. They were most successful with Counties-Manukau and Canterbury, i.e. the bottom and top teams from the round robin, while Wellington and Hawkes Bay ended up in the top 4. Participants were not so successful with the other semi-finalist, Southland, or with Bay of Plenty and Taranaki, who sprang surprises in both wins and losses. Margin predictions were most successful for Counties-Manukau again (presumably the extent of their losses), and Wellington (win margins), and least successful for Otago.
Looking at the wisdom of the crowd in picking the final order of teams in the competition (and leaving aside the complicating factor of bonus points), the crowd didn’t do too badly. A difference of 3 ranking places was the maximum error, three each out of the top 4 and bottom 4 were correctly ranked.
Five ranking levels matched, but Southland did twice as well as predicted, Tasman, Auckland also did much better, but Waikato, Otago, Bay of Plenty and North Harbour did much worse (by at least 2 places).
In a graphical presentation, this chart compares actual ranking with the ranking by the level of crowd prediction measured by the average percentage of win predictions.
So how did the crowd do? Not as well as it should have, in terms of win/loss outcomes, although it did get fairly close to the final order, with the exception of Southland. So ok on the big picture. On outcomes, there may have been a degree of home team emotion supporting some predictions, while in some cases teams just played so well or so badly that pre-game predictions became irrelevant and no amount of information or expertise would have helped. This is probably a good thing, and is certainly what has made this year’s Air New Zealand Cup a great competition. It’s not broken, so why fix it?