Predicting the winner of the World Cup is hard. For most people.
If only Paul were still alive. He never watched a game of football, learned how to use a spreadsheet or issued a press release about his state-of-the-art machine-learning-based forecasts, yet he was globally renowned for his preternatural ability to predict results at major international tournaments. Throughout the European Championship of 2008 and the World Cup of 2010 he was wrong on only two occasions. That Paul had only ever issued 14 predictions before his untimely death in October 2010 did not detract from his legendary status. Nor did the fact that he had tipped his native Germany 11 times. For Paul was an oracular octopus, who could prophesy the footballing future by choosing between two flag-bearing boxes containing oysters.
Paul’s secret methodology went with him to his watery grave. Attempts from zoos around the world to produce an equally gifted seer have proved futile. Eight years on, football fans could be forgiven for missing the authoritative decrees from his aquarium in Oberhausen. Ahead of this summer’s World Cup they have been inundated with forecasts from algorithm-wielding boffins, who have boasted of their sophisticated number-crunching—and yielded wildly different results. Among the banks to have produced such models, Goldman Sachs has tipped Brazil for the title, ING has plumped for Spain, UBS has picked Germany and Nomura is backing France.
Such a conflicting set of predictions will do little to convince football fans that statisticians know much about the beautiful game. In some senses, that scepticism is deserved. None of these models can account for the fact that Spain fired its manager a day before the tournament started; nor that Mo Salah, Egypt’s star player, might not be fully fit; nor that Mesut Özil and Ilkay Gündogan, two German midfielders of Turkish descent, are being booed by their own fans after posing for photos with Recep Tayyip Erdogan, Turkey’s president.
Such random quirks mean that mathematicians struggle to produce more accurate predictions than bookmakers or betting markets. Data from Betfair’s exchange, which allows punters to wager against each other, show that they have indeed been responding to news of Mr Salah’s fitness. Egypt’s implied probability of qualifying from Group A dropped from about 40% to 30% when he injured his shoulder in the Champions League final. The figure slowly climbed back to 40% as news circulated that he would likely recover in time for the tournament.
Spain’s chances have barely changed since sacking Julen Lopetegui on June 13th, when he announced that he would be taking over at Real Madrid after the tournament. Perhaps gamblers are finally accepting academic research which suggests that few managers are able to make a substantial difference to performance. Germany’s odds, meanwhile, have lengthened in the last couple of weeks. The most likely explanation, however, is not the mental state of Messrs Özil and Gündogan, but rather a couple of poor performances, including a loss to Austria and a narrow win against Saudi Arabia.
That was Die Mannschaft’s first victory in six matches, after draws against England, France and Spain and a loss to Brazil. A heavily-favoured team going through a slump is the sort of case that statisticians are best at addressing. Rather than accepting the sporting cliché that “form is temporary, but class is permanent”—a claim disproved by Hungary, which has reached the final twice but not qualified since 1986—they attempt to quantify it.
They generally do so using a modification of the Elo formula, which was first devised for chess but is now applied to many team sports. The system exchanges ratings points between opposing teams, with more points given for beating a stronger side. Teams also get extra credit for winning away from home, by a big margin or at a major tournament. Crucially, the Elo equation includes a “sensitivity” weight, which varies between sports, but dictates how much the ratings should respond to recent results. This weight is optimised to produce the best possible fit between past slumps and future recoveries.
In football, the best-known website that provides such rankings is EloRatings.net. Academics have found that it is a much better predictor of results than the official rankings kept by FIFA, the sport’s governing body, which rely on arbitrary weights that can easily be manipulated to boost a weak team up the table. (FIFA has finally acknowledged this, by announcing that it will adopt an Elo-style system after the tournament.) Most of the forecasts produced by banks and economists have used an Elo method of some sort.
Yet there is still disagreement among statisticians who use this technique. EloRatings.net aims to provide a historical record, going right back to 1872. However, the number of goals scored per game has nearly halved since the 19th century, from about 4.5 to roughly 2.5, which makes using constant weights tricky. 21st Club, a football consultancy which tracks the results of both international and club teams, has found that the most accurate way to predict modern matches is to dispense with wins and losses altogether. Instead, it focuses on goals scored and conceded, as well as the effects of red cards and unique home advantages—Bolivia has the biggest edge, since it plays at an unusually high altitude.
21st Club’s team ratings are especially generous to Brazil, which has scored 47 goals and conceded just five in the past two years. That is enough to merit a 32% probability of winning the World Cup (see chart). In contrast, the ratings give Germany just a 5% chance, after recording a negative goal difference since October last year. The gamblers seem to have less extreme opinions. Betfair’s exchange suggests that Brazil has a 19% chance of lifting the trophy, with Germany on 16%. What has caused this stark difference?
The most probable explanation is that bettors believe Germany has been underperforming relative to its extremely talented squad, whereas Brazil has been overachieving—and thus that the results of both teams will eventually converge towards the quality of their rosters. Unfortunately, measuring a player’s individual merit is devilishly hard. Some statisticians, like those at WhoScored.com, a statistics website, have attempted to do so by assigning a value to each action that a player makes with the ball, from defence-splitting passes to crunching tackles. However, the best players add lots of value without the ball, either by shepherding attackers away from goal or drawing defenders out of position with arcing runs. Besides, the detailed match data needed for such a formula are only available in a handful of domestic leagues.
An alternative approach to gauging a player’s contribution is to consider how well his team does with and without him on the pitch. This method is commonly used in baseball and basketball. The former sport is mostly a series of one-on-one confrontations between pitchers and batters, whereas the latter has frequent personnel changes within games. Replicating these calculations in football is trickier, since clubs play fewer fixtures than in American sports and substitutions are rarer. Nonetheless, injuries, squad rotation and cup competitions provide plenty of different line-ups over a season.
After building such a system, 21st Club has indeed found that Brazil and Germany are among the biggest overachievers and underperformers relative to their available players (see chart). In fact, if it were to predict the outcome of the tournament on personnel ratings rather than recent team performances, Germany’s chances would more than double to 14%, while Brazil’s would drop by more than half to 13%.
In domestic competitions, the player-based forecasts have been slightly more accurate than those derived solely from team ratings, explains Luis Usier, who designed the models. The two measures are highly correlated over a lengthy club season, but the player system has the edge because it can account for changes in selection. Neither model has quite managed to keep up with betting markets when it comes to predicting whether a match will end in a win, a draw or a loss. But the player predictions have been 99% as successful, while the team ones have trailed only slightly behind on 98%.
Yet few forecasters seem to think that judging an international team chiefly by its players is the right approach. 21st Club’s final forecast uses a 50-50 split between team and player ratings. FiveThirtyEight, a sports statistics website, has gone for a 75-25 weighting (using a similar personnel-rating method to 21st Club). Goldman Sachs has also assigned barely a quarter of its model’s predictive power to player ratings (though it has not revealed how it has calculated them).
This hedging is based on widely-held scepticism about whether national teams can convert individual brilliance into collective success. Star players who get plenty of time to gel at their clubs might only spend a handful of weeks training with their compatriots. Few countries organise more than 15 fixtures a year, most of which are friendlies. 21st Club’s data show that England, France, Spain and Belgium have shared Germany’s trait of underperforming relative to their talent. That might be a random fluctuation; or it might be due to the number of puny opponents that they have faced in qualification matches, which robs them of chances to practise in a competitive environment.
Because most statisticians have developed their player-rating systems since the last World Cup, they cannot quantify whether they are more or less effective at predicting major international tournaments than club ones. But their doubts seem to be shared by gamblers, too. The implied odds on Betfair are much closer to 21st Club’s blended forecast than to either its team-only or player-only projections.
There is one fact, however, on which FiveThirtyEight, Goldman Sachs and all three of 21st Club’s forecasts agree: gamblers are overrating Germany’s chances of lifting the title, by anywhere from three to 11 percentage points. The betting markets, of course, have the superior track record, and wagering against the Germans seems like a recipe for disaster. Die Mannschaft has reached the semi-finals in nearly three-quarters of the major tournaments in which it has appeared. Its players are famed for their composure under pressure. They have scored 17 of 18 penalties in shoot-outs at the World Cup. Perhaps punters are pricing in that steely resolve, in the knowledge that statisticians will struggle to measure it.
They should be cautious, however, of putting too much faith in German efficiency. The outcomes of penalties seem to be largely random, regardless of the shooter’s perceived skill. Lionel Messi, one of the most prolific finishers in the sport’s history, has scored just 77% of attempts during his career. That is roughly the average for all penalties in league play over the last decade. Thomas Müller, Germany’s most experienced taker, has converted 79% of his shots. The average rate for his team’s other attacking options in a shoot-out—Marco Reus, Timo Werner, Mario Gómez and Mr Özil—is 74%. Paul the octopus would certainly expect his compatriots to prevail from the spot.
Gamblers would probably feel the same. Statisticians, however, will be sitting on the fence.
This article first appeared in the Economist
