Rugby: The Wisdom of Crowds and the Air New Zealand Cup

29 October 2009

In an earlier posting, I looked at how the wisdom of crowds might apply in the case of the Sky Sport Virtual Rugby game to make picks in the Super 14 rugby competition.  In that case, and looking only at outcomes, i.e. win/loss, the crowd had a success rate of 69% over the round robin stage of the competition.  I suggested that this was not a great result.  I might have to revisit that.

The Air New Zealand Cup round robin stage has now been completed, so I have looked at the figures for those games to see if there is any difference.  In the Jimungo Virtual Rugby competition, participants pick the outcome (who will win) and a score level (12 point margin or less; moe than 12 points).  A draw can also be picked.  The published data include the percentage of participants who have picked each possible result, and it is this information, plus the actual outcome, that I have analysed.  What I was interested to find out was the level of predictive success for both outcomes and margins, whether the crowd learned anything over the course of the competition, and whether any teams were more predictable than others.

The Air New Zealand Cup round robin stage is played by 14 teams over 13 weeks.  The teams range from unions which provide a base for Super 14 teams – Auckland, Waikato, Wellington, Canterbury, Otago – to provincial unions, some of which have been struggling in financial terms, and until this year with declining attendances.  An added fillip this season has been the planned restructuring of the competition for next year which is intended to reduce it to 10 teams, which has meant that those teams under threat of exclusion have had an incentive to succeed on the field and in boosting crowds.  The perennial issue of whether and to what extent the All Blacks will be available for their provincial teams has also been aired again, while the continuing impact of the Ranfurly Shield cannot be ignored (except, it appears, by Wellington)

One hypothesis is that the Air New Zealand Cup, being a domestic competition, should mean that Virtual Rugby participants have more knowledge of the teams and players than may have been the case with the international Super 14.  Countering this may be the greater degree of parochial attachment that such a competition provides, the relatively unknown status of many players and the extent to which removal of key players to international duties will affect team performance.  Also, what should we expect in terms of crowd wisdom?  Random picks should produce a 50% outcome when taken over 91 games and 61,239 participants (I have just tossed a coin 91 times and heads won only 45% but I’m not going to do it another 61,238 times), so crowd wisdom should do much better than that, but by how much?

So, what were the results.  Well, the average success rate of predicting the outcome was 61.45%, which was less than the Super 14 result.  For both outcome and margin, the success rate was 32.29%. Assuming that the crowd that participates in Virtual Rugby has more information than a random generator, is this sufficiently better than 50%?  I don’t think so.

The chart shows the extent to which the outcome (win/loss) success rate moved over the season.  The success rate got up to 90% for the penultimate round, and did improve as the season developed after an initial slump.  The earlier rounds were characterised by “upsets”, and if defined as an outcome that fewer than 20% of participants picked, then there were 2 upsets in each of the first four rounds (out of 7 games in each round), as well as in rounds 7,8 and 13.

Successful Prediction of Outcomes

The higher success rates in the later rounds could reflect better informed participants, having more information on how the teams played, while other possible explanations could be that more All Blacks were playing for their provincial teams and strengthening the larger provinces with more favoured teams, or that the variable of the draw had come together to pit top and bottom teams against each other, with more predictable outcomes (not sure that this stands up to analysis).

The outcome and margin chart tells a similar story.  In round 3 only 10% of participants got the outcome and margin right and in only one round did more than half of participants get it right.

Blog2

So what about team support?  Well, the participants in Virtual Rugby had mixed results.  They were most successful with Counties-Manukau and Canterbury, i.e. the bottom and top teams from the round robin, while Wellington and Hawkes Bay ended up in the top 4.  Participants were not so successful with the other semi-finalist, Southland, or with Bay of Plenty and Taranaki, who sprang surprises in both wins and losses.  Margin predictions were most successful for Counties-Manukau again (presumably the extent of their losses), and Wellington (win margins), and least successful for Otago.

Successful prediction of margin and outcome by union

Looking at the wisdom of the crowd in picking the final order of teams in the competition (and leaving aside the complicating factor of bonus points), the crowd didn’t do too badly.  A difference of 3 ranking places was the maximum error, three each out of the top 4 and bottom 4 were correctly ranked.

Blog5

Five ranking levels matched, but Southland did twice as well as predicted, Tasman, Auckland also did much better, but Waikato, Otago, Bay of Plenty and North Harbour did much worse (by at least 2 places).

Blog4

In a graphical presentation, this chart compares actual ranking with the ranking by the level of crowd prediction measured by the average percentage of win predictions.

So how did the crowd do?  Not as well as it should have, in terms of win/loss outcomes, although it did get fairly close to the final order, with the exception of Southland.  So ok on the big picture. On outcomes, there may have been a degree of home team emotion supporting some predictions, while in some cases teams just played so well or so badly that pre-game predictions became irrelevant and no amount of information or expertise would have helped.  This is probably a good thing, and is certainly what has made this year’s Air New Zealand Cup a great competition.  It’s not broken, so why fix it?


Rugby: Winning the World Cup

22 September 2007

The 2007 Rugby World Cup is getting to an interesting stage, but is the result inevitable? What sparked this post is an article in the New Zealand Herald by Robbie Deans, Crusaders coach and formerly part of the All Black coaching team, in which he sets out the view that:

To win the World Cup, a team has to think it can. It then has to transfer this into a series of meaningful actions that grows belief.

While there will be up to eight so-called contenders who will be trying to convince themselves that they can win the tournament as we head towards the quarter-final phase, the reality is just two teams know they can win it: New Zealand and South Africa.

He then goes on to outline why this is so, i.e. that the South Africans have experience of winning, with the nucleus of the side beating New Zealand at under-21 level, plus the success of South African sides in the Super 14, plus strength of key players, plus effective leadership. He doesn’t elaborate on why New Zealand assumes it can win but I think we can take that as axiomatic – the All Blacks always assume they will win.

I agree with Robbie Deans, and I think his thesis is confirmed by the English win at the 2003 World Cup. It’s my belief that England knew they could win the Cup when they beat the All Blacks in Wellington in June 2003, even when down to thirteen men at one stage. This confidence took them through to ultimate victory in Sydney later in the year. New Zealand contributed to England’s Wellington victory through selection decisions and a lack of agility to adjust the game plan. In particular, that game marked the All Black debut of Ma’a Nonu, and the selection of a still green Rodney So’oialo at No 8. Now don’t get me wrong, Rodney is now a great player and one of the most effective All Black forwards, but in 2003 he was still learning. Similarly, Ma’a Nonu was very promising, still is, but to debut against a strong England side was a risk. The outcome was that New Zealand were reactive, played the game the way England wanted it played, and lost.

The risk that now arises, and which also derives from a New Zealand decision, is the point Robbie Deans makes about South Africa’s success in the Super 14. The winning records established by South African sides in this year’s competition can be largely attributed to the decision by New Zealand Rugby to rest key All Blacks for most of the Super 14 as part of the conditioning campaign for the World Cup (I know, I know, this could be challenged – but what are the Hurricanes without Jerry Collins and Rodney So’oialo). While the conditioning programme may well contribute to the All Blacks reaching the finals, it would be ironic if it also served as a mental conditioning for the Springboks, who should be their opponents in that final in Paris.

Today saw France defeat Ireland, and neither country looks, or plays, as though they believe they can win the World Cup. I was surprised that Robbie Deans didn’t include Australia in his list of believers, especially since they are the only side to have beaten the All Blacks this year, and also because they have the star players and are the most agile thinkers of the major contenders. While some of their star players might be just too old this time around, there are others who are still young, fresh and fast.

So things are starting to get a bit interesting, even if it is only around whether England, Ireland, Wales and France can stagger into the quarter-finals, or suffer some ignominy at the hands of more lowly-ranked nations.

One thing, though, is that this World Club has been clearly marked by the triumph of the blogs as a means of communication about the events and people’s take on them. Great engagement!