Category: Elo Ratings

Big Four Losing Streaks

Italian translation at settesei.it

This is a guest post by Peter Wetz.

Novak Djokovic’s loss against Benoit Paire in his first match at this year’s Miami Masters caused a lot of head scratching. Not only did Benoit equalize his head to head against Novak–next to Hyeon Chung he is now the only active player with a balanced record against Novak; four active players hold positive records–but this was also the Serbian’s third consecutive loss.

Novak immediately made some changes, announcing the end of his partnership with his coach Andre Agassi and part-time coach Radek Stepanek after having worked with them just a few months.

A losing streak of this length by such a dominant player must be rare, and it prompted me to look for similar instances among the big four. The following table shows all three (or more) match losing streaks of the big four after they cracked the top ten in reverse chronological order. The last column shows the Elo-based probability (Prob) of having such a streak. This is simply the product of the probabilities of losing the matches that made up the streak.

Player    Start	        End	Length	Prob
Djokovic  2018-01-15	-*	3	0.002%  (0.027%**)
Murray	  2011-01-17	03-23	4	0.02%
Murray	  2010-03-11	04-11	3	0.63%
Nadal	  2009-11-08	11-22	4	1.89%
Djokovic  2007-10-15	11-12	5	0.07%
Federer	  2002-07-08	08-19	4	0.66%

* Streak still active

** Probability when adjusting Elo ratings due to absence from the tour

The table shows that since August 2002 Roger Federer never lost more than two matches in a row. Even his four match losing streak is the second most likely due to the strong competition he had to face. In November 2009 Rafael Nadal lost four matches in a row, but with a probability far higher than the other streaks. The reason is that three of the four matches occurred at the World Tour Finals, increasing the likelihood of a loss.

A number that stands out is the probability of Novak’s current streak: 0.002%. However, this number is based on traditional Elo ratings which do not take into account player absence, for instance, due to injury. Before this season Novak took a six month break suffering from a shoulder injury.

As has already been discussed, there are ways to adjust Elo ratings for players coming back on the tour. In the case of Maria Sharapova, who stayed absent for 15 months, a 200 point drop in her first five matches after the break was more in line with her level of play than simply assuming that she remained as competitive as before. For this analysis I used a drop of 150 rating points for Novak, which results in a more realistic streak probability of  0.027%, still the second lowest in the list.

This brings us to Andy Murray‘s losing streak of 2011, which most of us probably have already forgotten. After losing the Australian Open final to Novak, Andy lost against Marcos Baghdatis (#20) in Rotterdam, Donald Young (#143) in Indian Wells, and Alex Bogomolov (#118) in Miami. This looks very similar to Novak’s current situation, but Murray bounced back to achieve a 50-9 record for the remainder of the season. It remains to be seen whether Djokovic can do the same.

—

Peter Wetz is a computer scientist interested in racket sports and data analytics based in Vienna, Austria.

Dominic Thiem, Old-School Clay Court Specialist

Italian translation at settesei.it

With a tennis calendar tilted heavily toward hard court events, we don’t see many true clay court specialists these days. The best male players who excel on clay are forced to adapt their games to hard courts, as well: Rafael Nadal has won six majors off of clay, while Pablo Carreno Busta and Diego Schwartzman have both hoisted trophies at tour level hard court events. It’s possible to play a mostly-clay schedule at the Challenger level, but it’s nearly impossible to establish yourself as an ATP regular without winning some matches on hard courts.

Dominic Thiem is capable enough on fast surfaces, but more than any other tour player, he is considerably better on the dirt than off it. In the last 52 weeks, he has won 25 of 31 matches on clay, compared to only 24 of 42 on other surfaces. Against the top ten, he is a respectable 7-9 on clay (more impressive when you consider that 12 of the 16 matches were against the Big Four, seven of them facing Nadal, and two of the others came against Stan Wawrinka), but a dismal 2-15 on hard courts. If you, like me, had settled into thinking of Thiem as a solid but not particuarly threatening member of the top ten, you probably didn’t realize quite how bad he is on hard courts–or just how good he has become on clay.

When only clay court results are taken into consideration, Thiem rates as the second-best player on the surface. According to clay court Elo, the Austrian outranks everyone on tour except for Rafa and Novak Djokovic, whose rating reflects his skill level when he last regularly played and very likely will overstate his ability when he returns. Thiem trails Nadal by about 180 points, 2410 to 2235, implying that in a head-to-head matchup, we’d except the Austrian to win only 26% of the time. But when we compare Thiem to the rest of the pack and exclude the walking wounded–Djokovic, Wawrinka, Andy Murray, and Kei Nishikori–along with clay-skipper Roger Federer, his position looks much better. The next best clay courter, Alexander Zverev, trails Thiem by about the same margin, nearly 170 points.

A clay court Elo rating over 2200 is a useful marker of elite status. In the professional era, only 29 players have reached that level, 22 of whom can count at least one Grand Slam title to their names. Among active players, only the Big Four, Nishikori, Juan Martin del Potro, David Ferrer, and Thiem belong to the club.

Where Thiem really stands out is the juxtaposition of his clay court success and his hard court mediocrity. After his title last week in Buenos Aires, his Elo rating based only on clay court results was 2234, compared to a hard court rating of 1869. The first number, as we’ve seen, is good for third overall, second if we exclude Djokovic’s increasingly stale results; the second puts him 31st on tour, behind Schwartzman, Damir Dzumhur, and Fabio Fognini.

No one active today is more of a clay specialist–in the sense that his results on clay exceed his results on hard–than Thiem. (There are some even more extreme differences between grass and either hard or clay, but the brevity of the grass season means that many of those contrasts are due only to small samples.) The ratio of Thiem’s clay court Elo rating to his hard court rating–again, 2234 to 1869–is 1.20, far beyond any of the 44 other active players with a clay court Elo rating of 1800 or higher. Simone Bolelli comes in second, at 1.12, and a handful of players, including Nadal, register at 1.10. Here is the entire top 20:

Player                 Clay Elo  Hard Elo  Ratio
Dominic Thiem              2234      1869   1.20
Simone Bolelli             1834      1634   1.12
Rafael Nadal               2410      2182   1.10
Albert Ramos               1873      1696   1.10
Federico Delbonis          1869      1696   1.10
Pablo Carreno Busta        1921      1746   1.10
Pablo Cuevas               1873      1709   1.10
Nicolas Almagro            1903      1755   1.08
Karen Khachanov            1838      1701   1.08
Leonardo Mayer             1878      1741   1.08
Aljaz Bedene               1826      1695   1.08
David Ferrer               2017      1894   1.07
Philipp Kohlschreiber      1951      1845   1.06
Stan Wawrinka              2138      2027   1.06
Martin Klizan              1800      1709   1.05
Guido Pella                1825      1744   1.05
Borna Coric                1830      1760   1.04
Fernando Verdasco          1863      1794   1.04
Alexander Zverev           2067      1997   1.04
Feliciano Lopez            1830      1772   1.03

A few decades ago, when it was possible for top players to spend more than two or three months per year racking up points on clay courts, such lopsided ratings were a bit more common. Of the 29 men who have ever topped 2200 in clay court Elo rating, 11 have at some point recorded a ratio of 1.20 or higher. That includes Nadal, whose clay rating was 20% higher than his hard court number early in 2008, and Sergi Bruguera, whose ratio topped out at 1.29. Four other major titlists–Bjorn Borg, Juan Carlos Ferrero, Thomas Muster, and Guillermo Vilas–also exceeded 1.20 at some point during their career. To put Thiem’s specialization in context, though, consider that Guillermo Coria maxed out at 1.19 and Gustavo Kuerten peaked at 1.16. Even Ferrer–the epitome of the clay court specialist to a generation of fans–never exceeded 1.15 once his clay court Elo rating had passed the 2000-point threshold.

The category into which Thiem fits most neatly–specialists who are decidedly middle-of-the-pack on hard courts–largely belongs to an earlier era. When we lower our clay court Elo standard to a career peak of 2000 points, a mark equal to about 15th on tour right now, we’re left with a group of 145 players in the professional era. Of those, 65 (45%) were at some point as lopsided as Thiem is now, with a clay-to-hard rating ratio of at least 1.20. Yet only five of those belong to active players (Nadal, Thiem, Fognini, Pablo Cuevas, and Nicolas Almagro) and two-thirds of them came before 1995.

In some cases, players with substantially better clay court results learn to compete at a higher level on faster surfaces. Thiem is 24, and Nadal had a similar specialist’s ratio at age 22. Other former greats enjoyed early success on clay and quickly figured out hard courts as well. The Austrian may prove to be a late bloomer in that regard. That’s unlikely, but when Nadal retires or (improbable as it seems) fades, Thiem is poised to rack up titles and emerge as the greatest clay court player of his generation, regardless of whether his hard court game improves.

Roger Federer’s 20th, Easiest Grand Slam Title

Italian translation at settesei.it

After Rafael Nadal’s US Open title last fall, I wrote a piece for the Economist that attempted to measure each Grand Slam title by difficulty. If you’re interested in the methodology, you can review it there. The conclusion was intriguing: Nadal’s opponents en route to his 16 major titles were considerably more difficult than the routes Roger Federer took to his first 19. By “difficulty-adjusted” Slam titles, Rafa led by a whisker, 18.8 to 18.7.

Since then, Federer won the 2018 Australian Open, incrementing his major tally by one. Even though he faced rather weak competition, surely the additional title nudged his difficulty-adjusted total above Rafa’s, right?

It did, but not by much. Adjusted for difficulty, Roger’s seven wins in Melbourne were worth only 0.42 majors. By comparison, his previous low was the 2006 Australian, worth 0.61, and Rafa’s lowest was last year’s US Open, at 0.62. Federer’s previous average was 0.98, Nadal’s was 1.18, and Rafa’s route to the 2013 French Open was worth a whopping 1.65.

Fed’s draw was historically weak. Only a handful of majors in the professional era were easier for their champion, and they all came before 1985–most of them in Melbourne, which didn’t yet attract the best talent in the world. This year’s Australian Open path to the title was even weaker when put in the context of the current decade: The average major title from 2010-17 was worth 1.23, largely because the Big Four usually needed to overcome each other.

According to surface-specific Elo, the toughest challenge Federer faced last month was Tomas Berdych, closely followed by Marin Cilic. Even after deep runs in Australia, neither player even ranks in the current Elo top ten. The algorithm that adjusts slam titles considers how the average major champion would fare against a particular set of competition; against Berdych and Cilic, that hypothetical average champ is expected to win 88% and 89% of the time, respectively. Even Nadal had to get past Juan Martin del Potro in New York last year.

Still, Federer can claim the top spot on yet another list, as his 19.1 difficulty-adjusted Grand Slam titles exceed Rafa’s 18.8 as well as the 15.3 of Novak Djokovic. It doesn’t have quite the same ring that “20 majors” does, and it’s in considerably more immediate danger. If Nadal stays healthy and wins the French Open, he is virtually guaranteed to reclaim the difficulty-adjusted crown, and by a wider margin than Roger currently holds. Roland Garros has traditionally been tough: With the exception of 2010, all of Rafa’s trophies in Paris have been tougher than average. Unlike the traditional Grand Slam tally, the difficulty-adjusted ranking could yo-yo between the two rivals for as long as they remain competitive.

Overperforming in Davis Cup

This is a guest post by Peter Wetz.

With the help of weighted surface specific Elo ratings we have a powerful new tool to measure player performance. The traditional conclusion of the tennis season, the Davis Cup final, provides us with an opportunity once again to examine which players thrive when competing for their nation and which players seem to suffer from the pressure. While we are at it, I don’t like the sound of the word offseason. After all, there are still ITF tournaments, not to mention the Australian Open Asia-Pacific Wildcard Play-offs.

As already hinted, Elo ratings have proven to represent a better picture of player quality than traditional ATP rankings. Hence, comparing expected wins based on Elo with actual wins provides us with a clearer picture of who outperforms expectations and who does not.

In this evaluation, I consider completed World Group and Group 1 Davis Cup live rubbers played since 1980. The data set contains around 5000 matches through this year’s World Group Quarterfinals, and I’ve limited my focus to players having played 15 or more matches.

Let’s first take a glance at the obvious stat, win-loss percentage. The following table shows the top ten win-loss records of all players under consideration. (The Active column denotes if the player is still an active player).

Name	        W	L	Perc	Active
Rafael Nadal	20	1	95%	1
Boris Becker	31	2	94%	0
Andy Murray	25	3	90%	1
Balazs Taroczy	23	3	89%	0
David Ferrer	20	3	87%	1
Andre Agassi	23	4	85%	0
Roger Federer	40	7	85%	1
Novak Djokovic	27	5	84%	1
Guillermo Vilas	16	3	84%	0
Andrei Medvedev	16	3	84%	0

As one would expect, the big four and other all time greats are included. However, this obviously does not tell the whole story. Rafael Nadal is expected to win most of the time and that is what he does. For such a player, it is hard to outperform expectations.

If we compute how much a player outperforms his expectations, we get a clearer picture, given we want to know who does especially well in Davis Cup. Expected wins are calculated based on a half-and-half mix of surface specific Elo and overall Elo as this, in general, provides close to the best results, as pointed out in a previous article.

The tables below show the top and bottom five among all (first table) and active (second table) players in terms of over and underperforming expected wins. It shows actual wins (W), expected wins (eW), the percentage of over or underperformance (+/-), and if a player is still active.

Name	         W	eW	+/-	active
Francisco Maciel 11	6	72%	0
Slobodan Zi'vic  20	11	72%	0
Vasek Pospisil	 9	5	71%	1
Adrian Ungur	 6	3	56%	1
Mahesh Bhupathi	 5	3	55%	0
...
Wally Masur	 7	10     -31%	0
Sebastien Lareau 7	10     -31%	0
James Blake	 7	10     -36%	0
Nicolas Kiefer	 6	10     -40%	0
Aqeel Khan	 2	4      -57%	0

Name	        W	eW	+/-	Active
Vasek Pospisil	9	5	71%	1
Adrian Ungur	6	3	56%	1
Andrey Golubev	13	8	46%	1
Di Wu	        14	9	45%	1
Steve Darcis	15	11	35%	1
...
Florian Mayer	7	8      -14%	1
Gilles Muller	9	10     -15%	1
Alejandro Falla	8	9      -17%	1
John Isner	9	11     -19%	1
Jurgen Melzer	20	25     -22%	1

The tables seem to overlap with some conventional wisdom floating through the tennis sphere. Namely, that Steve Darcis, despite his recent losses at the Davis Cup final, plays above expectations. Also, Jurgen Melzer is known for regularly disappointing Austrian Davis Cup fans. (In his defense, he created several moments of joy, too).

If we were to pick a Davis Cup hero for the active and inactive group of players, Slobodan Zivojinovic and Andrey Golubev seem to be good choices. Golubev has a record of 13-6 (68%) and outperforms expected wins by 46%. He provides a good combination of consistently beating players he should beat and scoring more than his share of exceptional upsets (Wawrinka 2014, Goffin 2014, Melzer 2013 and Berdych 2011).

Zivojinovic provides a similar pattern with a record of 20-8 (71%), 72% better than expected. He tallied six wins out of ten matches in which Elo assigned him a win probability of less than 25%. Further, he only lost one match in when his pre-match odds of winning were greater than 35%.

This post provides insight into how Elo ratings help in quantifying a player’s performance. We identified players who have (not) shown great improvement on what the algorithm expected based on results from the regular tour. For future research it would be interesting to delve into Davis Cup doubles heroes: Where there are no dead rubbers, stakes are always high.

—

Peter Wetz is a computer scientist interested in racket sports and data analytics based in Vienna, Austria.

Forecasting the Laver Cup

Italian translation at settesei.it

This weekend brings us the first edition of the Laver Cup, a star-studded three-day affair that pits Europe against the rest of the world. The European team features Roger Federer and Rafael Nadal, and even though several other elites from the continent are missing due to injury, the European team is still much stronger on paper.

Here are the current rosters, along with each competitor’s weighted hard court Elo rating and rank among active players:

EUROPE                  Elo Rating  Elo Rank  
Roger Federer                 2350         2  
Rafael Nadal                  2225         4  
Alexander Zverev              2127         7  
Tomas Berdych                 2038        14  
Marin Cilic                   2029        15  
Dominic Thiem                 1995        17  
                                              
WORLD                   Elo Rating  Elo Rank  
Nick Kyrgios                  2122         8  
John Isner                    1968        22  
Jack Sock                     1951        23  
Sam Querrey                   1939        25  
Denis Shapovalov              1875        36  
Frances Tiafoe                1574       153  
Juan Martin del Potro*        2154         5

*del Potro has withdrawn. I’ve included his singles Elo rating and rank to emphasize how damaging his absence is to the World squad.

“Weighted” surface Elo is the average of overall (all-surface) Elo and surface-specific Elo. The 50/50 split is a much better predictor of match outcomes than either number on its own.

Nick Kyrgios can hang with anybody on a hard court. But despite some surface-specific skills represented by the American contingent, every other member of the World team rates lower than every member of team Europe. This isn’t a good start for the rest of the world.

What about doubles? Here are the D-Lo (Elo for doubles) ratings and rankings for all twelve participants, plus Delpo:

EUROPE                  D-Lo rating  D-Lo rank  
Rafael Nadal                   1895          4  
Tomas Berdych                  1760         28  
Marin Cilic                    1676         76  
Roger Federer**                1650         90  
Alexander Zverev               1642         99  
Dominic Thiem                  1521        185  
                                                
WORLD                   D-Lo rating  D-Lo rank  
Jack Sock                      1866          8  
John Isner                     1755         29  
Nick Kyrgios                   1723         45  
Sam Querrey                    1715         49  
Denis Shapovalov**             1600        130  
Frances Tiafoe                 1546        166  
Juan Martin del Potro*         1711         55

** Federer hasn’t played tour-level doubles since 2015, and Shapovalov hasn’t done so at all. These numbers are my best guesses, nothing more.

Here, the World team has something of an edge. While both sides feature an elite doubles player–Rafa and Jack Sock–the non-European side is a bit deeper, especially if they keep Denis Shapovalov and last-minute Delpo replacement Frances Tiafoe on the sidelines. Only one-quarter of Laver Cup matches are doubles (plus a tie-breaking 13th match, if necessary), so it still looks like team Europe are the heavy favorite.

The format

The Laver Cup will take place in Prague over three days (starting Friday, September 22nd), and consist of four matches each day: three singles and one doubles. Every match is best-of-three sets with ad scoring and a 10-point super-tiebreak in place of the third set.

On the first day, the winner of each match gets one point; on the second day, two points, and on the third day, three points. That’s a total of 24 points up for grabs, and if the twelve matches end in a 12-12 deadlock, the Cup will be decided with a single doubles set.

All twelve participants must play at least one singles match, and no one can play more than two. At least four members of each squad must play doubles, and no doubles pairing can be repeated, except in the case of a tie-breaking doubles set.

Got it? Good.

Optimal strategy

The rules require that three players on each side will contest only one singles match while the other three will enter two each. A smart captain would, health permitting, use his three best players twice. Since matches on days two and three count for more than matches on day one, it also makes sense that captains would use their best players on the final two days.

(There are some game-theoretic considerations I won’t delve into here. Team World could use better players on day one in hopes of racking up each points against the lesser members of team Europe, or could drop hints that they will do so, hoping that the European squad would move its better players to day one. As far as I can tell, neither team can change their lineup in response to the other side’s selections, so the opportunities for this sort of strategizing are limited.)

In doubles, the ideal roster deployment strategy would be to use the team’s best player in all three matches. He would be paired with the next-best player on day three, the third-best on day two, and the fourth-best on day one. Again, this is health permitting, and since all of these guys are playing singles, fatigue is a factor as well. My algorithm thus far would use Nadal five times–twice in singles and three times in doubles–and I strongly suspect that isn’t going to happen.

The forecast

Let’s start by predicting the outcome of the Cup if both captains use their roster optimally, even if that’s a longshot. I set up the simulation so that each day’s singles competitors would come out in random order–if, say, Querrey, Shapovalov, and Tiafoe play for team World on day one, we don’t know which of them will play first, or which European opponent each will face. So each run of the simulation is a little different.

As usual, I used Elo (and D-Lo) to predict the outcome of specific matchups. Because of the third-set super-tiebreak, and because it’s an exhibition, I added a bit of extra randomness to every forecast, so if the algorithm says a player has a 60% chance of winning, we knock it down to around 57.5%. When I dug into IPTL results last winter, I discovered that exhibition results play surprisingly true to expectations, and I suspect players will take Laver Cup a bit more seriously than they do IPTL.

Our forecast–again, assuming optimal player usage–says that Europe has an 84.3% chance of winning, and the median point score is 16-8. There’s an approximately 6.5% chance that we’ll see a 12-12 tie, and when we do, Europe has a slender 52.4% edge.

If Delpo were participating, he would increase the World team’s chances by quite a bit, reducing Europe’s likelihood of victory to 75.5% and narrowing the most probable point score to 15-9.

What if we relax the “optimal usage” restriction? I have no idea how to predict what captains John McEnroe and Bjorn Borg will do, but we can randomize which players suit up for which matches to get a sense of how much influence they have. If we randomize everything–literally, just pick a competitor out of a hat for each match–Europe comes out on top 79.7% of the time, usually winning 15-9. There’s a 7.6% chance of a tie-breaking 13th match, and because the World team’s doubles options are a bit deeper, they win a slim majority of those final sets. (When we randomize everything, there’s a slight risk that we violate the rules, perhaps using the same doubles pairing twice or leaving a player on the bench for all nine singles matches. Those chances are very low, however, so I didn’t tackle the extra work required to avoid them entirely.)

We can also tweak roster usage by team, in case it turns out that one captain is much savvier than the other. (Or if a star like Nadal is unable to play as much as his team would like.) The best-case scenario for our World team underdogs is that McEnroe chooses the best players for each match and Borg does not. Assuming that only European players are chosen from a hat, the probability that the favorites win falls all the way to 63.1%, and the typical gap between point totals narrows all the way to 13-11. The chance of a tie rises to 10%.

On the other hand, it’s possible that Borg is better at utilizing his squad. After all, it doesn’t take an 11-time grand slam winner to realize that Federer and Nadal ought to be on court when the stakes are the highest. This final forecast, with random roster usage from team World and ideal choices from Borg, gives Europe a whopping 92.3% chance of victory, and median point totals of 17 to 7. The World team would have only a 4% shot at reaching a deadlock, and even then, the Europeans win two-thirds of the tiebreakers.

There we have it. The numbers bear out our expectation that Europe is the heavy favorite, and they give us a sense of the likely margin of victory. Tiafoe and Shapovalov might someday be part of a winning Laver Cup side, but it looks like they’ll have to wait a few years before that happens.

–

Update: One more thing… What about doubles specialists? Both captains have two discretionary picks to use on players regardless of ranking. Most great doubles players are much worse at singles, but as we’ve seen, a player can be relegated to a lone one-point singles match on day one, and as a doubles player, he can have an effect on three different matches, totaling six points.

Sure enough, swapping out Dominic Thiem (a very weak doubles player for whom indoor hard courts are less than ideal) for Nicolas Mahut would have increased Europe’s chances of winning from 84.3% to 88.5%. On the slight chance that the Cup stayed tight through the final doubles match and into a tiebreaker, the doubles team of Mahut-Nadal (however unorthodox that sounds) would be among the best that any captain could put on the court.

There’s even more room for improvement on the World side, especially with del Potro out. At the moment, the third-highest rated hard court player by D-Lo is Marcelo Melo, who would be a major step down in singles but a huge improvement on most of the potential partners for Sock in doubles. If we give him a singles Elo of 1450 and put him on the roster in place of Tiafoe and pit the resulting squad against the original Europe team (with Thiem, not Mahut), it almost makes up for the loss of Delpo–World’s chances of winning increase from 15.7% to 19.3%.

Unfortunately, Borg and McEnroe may have missed their chance to eke out extra value from their six-man rosters–this is a trick that will only work once. If both teams made this trade, Mahut-for-Thiem and Melo-for-Tiafoe, each side’s win probability goes back to near where it started: 85.8% for Europe. That’s a boost over where we started (84.3%), just because Mahut is better suited for the competition than Melo is, as an elite doubles specialist who is also credible on the singles court. No one available to the World team (except for Sock, who is already on the roster) fits the same profile on a hard court. Vasek Pospisil comes to mind, though he has taken a step back from his peaks in both singles and doubles. And on clay, Pablo Cuevas would do nicely, but on a faster surface, he would represent only a marginal improvement over the doubles players already playing for team World.

Maybe next year.

 

Denis Shapovalov and Fast ATP Starts

Italian translation at settesei.it

18-year-old Canadian lefty Denis Shapovalov has had one heck of a summer. In Montreal, he defeated Juan Martin del Potro and Rafael Nadal in back-to-back matches, and at the US Open, he qualified for the main draw, upset Jo Wilfried Tsonga, and reached the fourth round in only his second appearance at a major.

Thanks to those wins and the big stages on which he achieved them, he has cracked the ATP top 60, despite playing fewer than 20 tour-level matches. The Elo rating system, which awards points based on opponent quality, is even more optimistic. By that measure, with his win over Tsonga, Shapovalov improved to 1950–good for 34th on tour–before losing about 25 Elo points in his loss to Pablo Carreno Busta.

While an Elo score of 1950 is an arbitrary number–there’s nothing magical about any particular Elo threshold; it’s just a mechanism to compare players to each other–it gives us a way to compare Shapovalov’s hot start with other players who made quick impacts at tour level. Since the early 1980s, only 13 players have reached a 1950 Elo score in fewer matches than the Canadian needed. As usual with early-career accomplishments, there are a few unexpected names in the mix, but overall, it’s very promising company for an 18-year-old:

Player               Matches   Age  
Lleyton Hewitt             7  16.9  
Jarkko Nieminen            7  20.2  
Juan Carlos Ferrero       10  19.4  
David Ferrer              12  20.4  
Kenneth Carlsen           12  19.4  
Tommy Haas                13  19.1  
Peter Lundgren            13  20.7  
John Van Lottum           14  21.8  
Sergi Bruguera            14  18.4  
Julian Alonso             15  20.0

Player               Matches   Age   
Xavier Malisse            16  18.6  
Jan Siemerink             16  20.9  
Ivo Minar                 16  21.2  
Florian Mayer             17  20.7  
Cristiano Caratti         17  20.7  
Nick Kyrgios              17  19.3  
Denis Shapovalov          17  18.4  
Martin Strelba            17  22.1  
Jay Berger                17  20.2  
Andy Roddick              18  18.6

I identified just over 350 players who, at some point in their careers, peaked with an Elo score of at least 1950. On average, these players needed 75 matches to reach that level (the median is 59), and two active tour-regulars, Gilles Muller and Albert Ramos, needed almost 300 matches to achieve the threshold.

Shapovalov’s record so far is equally impressive when we consider it in terms of age. Again, he’s among the top 20 players in modern tennis history: Only 11 players got to 1950 before their 18th birthday. The Canadian is only a few months beyond his. And many of the other ATPers who reached that score at an early age needed much more tour experience. I’ve included the top 30 on this list to show how Shapovalov compares to so many of the game’s greats:

Player                  Matches   Age  
Aaron Krickstein             25  16.4  
Michael Chang                32  16.5  
Lleyton Hewitt                7  16.9  
Boris Becker                 27  17.5  
Mats Wilander                27  17.5  
Guillermo Perez Roldan       26  17.6  
Andre Agassi                 46  17.6  
Pat Cash                     66  17.6  
Goran Ivanisevic             35  17.7  
Andrei Medvedev              22  17.8  

Player                  Matches   Age
Rafael Nadal                 44  17.9  
Sammy Giammalva              21  18.0  
Horst Skoff                  19  18.1  
Jimmy Arias                  61  18.2  
Kent Carlsson                56  18.3  
Sergi Bruguera               14  18.4  
Denis Shapovalov             17  18.4  
Andy Murray                  22  18.4  
Juan Martin del Potro        31  18.4  
Fabrice Santoro              59  18.5  

Player                  Matches   Age
John McEnroe                 28  18.5  
Roger Federer                40  18.5  
Stefan Edberg                40  18.5  
Andy Roddick                 18  18.6  
Pete Sampras                 56  18.6  
Thomas Enqvist               28  18.6  
Xavier Malisse               16  18.6  
Novak Djokovic               33  18.8  
Jim Courier                  51  18.8  
Yannick Noah                 41  18.8

There are no guarantees when it comes to tennis prospects, but this is very good company. On average, the 23 other players to reach the 1950 Elo threshold at age 18 improved their Elo ratings to 2100 before age 20, and rose to 2250 at some point in their careers. The first number would be good for 12th on today’s list, and the second would merit 5th place, just behind the Big Four. Nadal and del Potro were the first of Shapovalov’s high-profile victims, and judging from this sharp career trajectory, they won’t be the last.

Quantifying Cakewalks, or The Time Rafa Finally Got Lucky

Italian translation at settesei.it

During this year’s US Open, much has been made of some rather patchy sections of the draw. Many great players are sitting out the tournament with injury, and plenty of others crashed out early. Pablo Carreno Busta reached the quarterfinals by defeating four straight qualifiers, and Rafael Nadal could conceivably win the title without beating a single top-20 player.

None of this is a reflection on the players themselves: They can play only the draw they’re dealt, and we’ll never know how they would’ve handled a more challenging array of opponents. The weakness of the draw, however, could affect how we remember this tournament.  If we are going to let the quality of the field color our memories, we should at least try to put this year’s players in context to see how they compare with majors in the past.

How to measure draw paths

There are lots of ways to quantify draw quality. (There’s an entire category on this blog devoted to it.) Since we’re interested in the specific sets of opponents faced by our remaining contenders, we need a metric that focuses on those. It doesn’t really matter that, say, Nick Kyrgios was in the draw, since none of the semifinalists had to play him.

Instead of draw difficulty, what we’re after is what I’ll call path ease. It’s a straightforward enough concept: How hard is it to beat the specific set of guys that Rafa (for instance) had to play?

To get a number, we’ll need a few things: The surface-weighted Elo ratings of each one of a player’s opponents, along with a sort of “reference Elo” for an average major semifinalist. (Or finalist, or title winner.) To determine the ease of Nadal’s path so far, we don’t want to use Nadal’s Elo. If we did that, the exact same path would look easier or harder depending on the quality of the player who faced it.

(The exact value of the “reference Elo” isn’t that important, but for those of you interested in the numbers: I found the average Elo rating of every slam semifinalist, finalist, and winner back to 1988 on each of the three major surfaces. On hard courts, those numbers are 2145, 2198, and 2233, respectively. When measuring the difficulty of a path to the semifinal round, I used the first of those numbers; for the difficulty of a path to the title, I used the last.)

To measure path ease, then, we answer the question: What are the odds that an average slam semifinalist (for instance) would beat this particular set of players? In Rafa’s case, he has yet to face a player with a weighted-hard-court Elo rating above 1900, and the typical 2145-rated semifinalist would beat those five players 71.5% of the time. That’s a bit easier than Kevin Anderson‘s path the semis, but a bit harder than Carreno Busta’s. Juan Martin del Potro, on the other hand, is in a different world altogether. Here are the path ease numbers for all four semifinalists, showing the likelihood that average contenders in each round would advance, giving the difficulty of the draws each player has faced:

Semifinalist   Semi Path  Final Path  Title Path  
Nadal              71.5%       49.7%       51.4%  
del Potro           9.1%        7.5%       10.0%  
Anderson           69.1%       68.9%       47.1%  
Carreno Busta      74.3%       71.2%       48.4%

(We don’t yet know each player’s path to the title, so I averaged the Elos of possible opponents. Anderson and Carreno Busta are very close, so for Rafa and Delpo, their potential final opponent doesn’t make much difference.)

There’s one quirk with this metric that you might have noticed: For Nadal and del Potro, their difficulty of reaching the final is greater than that of winning the title altogether! Obviously that doesn’t make logical sense–the numbers work out that way because of the “reference Elos” I’m using. The average slam winner is better than the average slam finalist, so the table is really saying that it’s easier for the average slam winner to beat Rafa’s seven opponents than it would be for the average slam finalist to get past his first six opponents. This metric works best when comparing title paths to title paths, or semifinal paths to semifinal paths, which is what we’ll do for the rest of this post.

Caveats and quirks aside, it’s striking just how easy three of the semifinal paths have been compared to del Potro’s much more arduous route. Even if we discount the difficulty of beating Roger Federer–Elo thinks he’s the best active player on hard courts but doesn’t know about his health issues–Delpo’s path is wildly different from those of his semifinal and possible final opponents.

Cakewalks in context

Semifinalist path eases of 69% or higher–that is, easier–are extremely rare. In fact, the paths of Anderson, Carreno Busta, and Nadal are all among the ten easiest in the last thirty years! Here are the previous top ten:

Year  Slam             Semifinalist               Path Ease  
1989  Australian Open  Thomas Muster                  84.1%  
1989  Australian Open  Miloslav Mecir                 74.2%  
1990  Australian Open  Ivan Lendl                     73.8%  
2006  Roland Garros    Ivan Ljubicic                  73.7%  
1988  Australian Open  Ivan Lendl                     72.2%  
1988  Australian Open  Pat Cash                       70.1%  
2004  Australian Open  Juan Carlos Ferrero            69.2%  
1996  US Open          Michael Chang                  68.8%  
1990  Roland Garros    Andres Gomez                   68.4%  
1996  Australian Open  Michael Chang                  66.2%

In the last decade, the easiest path to the semifinal was Stan Wawrinka‘s route to the 2016 French Open final four, which rated 59.8%. As we’ll see further on, Wawrinka’s draw got a lot more difficult after that.

Del Potro’s draw so far isn’t quite as extreme, but it is quite difficult in the historical context. Of the nearly 500 major semifinalists since 1988, all but 15 are easier than his 9.1% path difficulty. Here are the top ten, all of whom faced draws that would have given the average slam semifinalist less than an 8% chance of getting that far:

Year  Slam             Semifinalist              Path Ease  
2009  Roland Garros    Robin Soderling                1.6%  
1988  Roland Garros    Jonas Svensson                 1.9%  
2017  Wimbledon        Tomas Berdych                  3.7%  
1996  Wimbledon        Richard Krajicek               6.4%  
2011  Wimbledon        Jo Wilfried Tsonga             6.6%  
2012  US Open          Tomas Berdych                  6.8%  
2017  Roland Garros    Dominic Thiem                  6.9%  
2014  Australian Open  Stan Wawrinka                  7.0%  
1989  Roland Garros    Michael Chang                  7.1%  
2017  Wimbledon        Sam Querrey                    7.5%

Previewing the history books

In the long term, we’ll care a lot more about how the 2017 US Open champion won the title than how he made it through the first five rounds. As we saw above, three of the four semifinalists have a path ease of around 50% to win the title–again, meaning that a typical slam winner would have a roughly 50/50 chance of getting past this particular set of seven opponents.

No major winner in recent memory has had it so easy. Nadal’s path would rate first in the last thirty years, while Carreno Busta’s or Anderson’s would rate in the top five. (If it comes to that, their exact numbers will depend on who they face in the final.) Here is the list that those three men have the chance to disrupt:

Year  Slam             Winner                  Path Ease  
2002  Australian Open  Thomas Johansson            48.1%  
2001  Australian Open  Andre Agassi                47.6%  
1999  Roland Garros    Andre Agassi                45.6%  
2000  Wimbledon        Pete Sampras                45.3%  
2006  Australian Open  Roger Federer               44.5%  
1997  Australian Open  Pete Sampras                44.4%  
2003  Australian Open  Andre Agassi                43.9%  
1999  US Open          Andre Agassi                41.5%  
2002  Wimbledon        Lleyton Hewitt              39.9%  
1998  Wimbledon        Pete Sampras                39.1%

At the 2006 Australian Open, Federer lucked into a path that was nearly as easy as Rafa’s this year. His 2003 Wimbledon title just missed the top ten as well. By comparison, Novak Djokovic has never won a major with a path ease greater than 18.7%–harder than that faced by more than half of major winners.

Nadal has hardly had it easy as he has racked up his 15 grand slams, either. Here are the top ten most difficult title paths:

Year  Slam             Winner                Path Ease  
2014  Australian Open  Stan Wawrinka              2.2%  
2015  Roland Garros    Stan Wawrinka              3.1%  
2016  Us Open          Stan Wawrinka              3.2%  
2013  Roland Garros    Rafael Nadal               4.4%  
2014  Roland Garros    Rafael Nadal               4.7%  
1989  Roland Garros    Michael Chang              5.0%  
2012  Roland Garros    Rafael Nadal               5.2%  
2016  Australian Open  Novak Djokovic             5.4%  
2009  US Open          J.M. Del Potro             5.9%  
1990  Wimbledon        Stefan Edberg              6.2%

As I hinted in the title of this post, while Nadal got lucky in New York this year, it hasn’t always been that way. He appears three times on this list, facing greater challenges than any major winner other than Wawrinka the giant-killer.

On average, Rafa’s grand slam title paths haven’t been quite as harrowing as Djokovic’s, but compared to most other greats of the last few decades, he has worked hard for his titles. Here are the average path eases of players with at least three majors since 1988:

Player           Majors        Avg Path Ease  
Stan Wawrinka         3                 2.8%  
Novak Djokovic       12                11.3%  
Rafael Nadal         15                13.6%  
Stefan Edberg         4                14.6%  
Andy Murray           3                18.8%  
Boris Becker          4                18.8%  
Mats Wilander         3                19.8%  
Gustavo Kuerten       3                22.0%  
Roger Federer        19                23.5%  
Jim Courier           4                26.4%  
Pete Sampras         14                28.9%  
Andre Agassi          8                32.3%

If Rafa adds to his grand slam haul this weekend, his average path ease will take a bit of a hit. Still, he’ll only move one place down the list, behind Stefan Edberg. After more than a decade of battling all-time greats in the late rounds of majors, it’s fair to say that Nadal deserved this cakewalk.

Update: This post reads a bit differently than when I first wrote it: I’ve changed the references to “path difficulty” to “path ease” to make it clearer what the metric is showing.

Nadal and Anderson advanced to the final, so we can now determine the exact path ease number for whichever one of them wins the title. Rafa’s exact number remains 51.4%, and should he win, his career average across 16 slams will increase to about 15%. Anderson’s path ease to the title is “only” 41.3%, which would be good for ninth on the list shown above, and just barely second easiest of the last 30 US Opens.

Measuring the Impact of Wimbledon’s Seeding Formula

Italian translation at settesei.it

Unlike every other tournament on the tennis calendar, Wimbledon uses its own formula to determine seedings. The grass court Grand Slam grants seeds to the top 32 players in each tour’s rankings, and then re-orders them based on its own algorithm, which rewards players for their performance on grass over the last two seasons.

This year, the Wimbledon seeding formula has more impact on the men’s draw than usual. Seven-time champion Roger Federer is one of the best grass court players of all time, and though he dominated hard courts in the first half of 2017, he still sits outside the top four in the ATP rankings after missing the second half of 2016. Thanks to Wimbledon’s re-ordering of the seeds, Federer will switch places with ATP No. 3 Stan Wawrinka and take his place in the draw as the third seed.

Even with Wawrinka’s futility on grass and the shakiness of Andy Murray and Novak Djokovic, getting inside the top four has its benefits. If everyone lives up to their seed in the first four rounds (they won’t, but bear with me), the No. 5 seed will face a path to the title that requires beating three top-four players. Whichever top-four guy has No. 5 in his quarter would confront the same challenge, but the other three would have an easier time of it. Before players are placed in the draw, top-four seeds have a 75% chance of that easier path.

Let’s attach some numbers to these speculations. I’m interested in the draw implications of three different seeding methods: ATP rankings (as every other tournament uses), the Wimbledon method, and weighted grass-court Elo. As I described last week, weighted surface-specific Elo–averaging surface-specific Elo with overall Elo–is more predictive than ATP rankings, pure surface Elo, or overall Elo. What’s more, weighted grass-court Elo–let’s call it gElo–is about as predictive as its peers for hard and clay courts, even though we have less grass-court data to go on. In a tennis world populated only by analysts, seedings would be determined by something a lot more like gElo and a lot less like the ATP computer.

Since gElo ratings provide the best forecasts, we’ll use them to determine the effects of the different seeding formulas. Here is the current gElo top sixteen, through Halle and Queen’s Club:

1   Novak Djokovic         2296.5  
2   Andy Murray            2247.6  
3   Roger Federer          2246.8  
4   Rafael Nadal           2101.4  
5   Juan Martin Del Potro  2037.5  
6   Kei Nishikori          2035.9  
7   Milos Raonic           2029.4  
8   Jo Wilfried Tsonga     2020.2  
9   Alexander Zverev       2010.2  
10  Marin Cilic            1997.7  
11  Nick Kyrgios           1967.7  
12  Tomas Berdych          1967.0  
13  Gilles Muller          1958.2  
14  Richard Gasquet        1953.4  
15  Stanislas Wawrinka     1952.8  
16  Feliciano Lopez        1945.3

We might quibble with some these positions–the algorithm knows nothing about whatever is plaguing Djokovic, for one thing–but in general, gElo does a better job of reflecting surface-specific ability level than other systems.

The forecasts

Next, we build a hypothetical 128-player draw and run a whole bunch of simulations. I’ve used the top 128 in the ATP rankings, except for known withdrawals such as David Goffin and Pablo Carreno Busta, which doesn’t differ much from the list of guys who will ultimately make up the field. Then, for each seeding method, we randomly generate a hundred thousand draws, simulate those brackets, and tally up the winners.

Here are the ATP top ten, along with their chances of winning Wimbledon using the three different seeding methods:

Player              ATP     W%  Wimb     W%  gElo     W%  
Andy Murray           1  23.6%     1  24.3%     2  24.1%  
Rafael Nadal          2   6.1%     4   5.7%     4   5.5%  
Stanislas Wawrinka    3   0.8%     5   0.5%    15   0.4%  
Novak Djokovic        4  34.1%     2  35.4%     1  34.8%  
Roger Federer         5  21.1%     3  22.4%     3  22.4%  
Marin Cilic           6   1.3%     7   1.0%    10   1.0%  
Milos Raonic          7   2.0%     6   1.6%     7   1.7%  
Dominic Thiem         8   0.4%     8   0.3%    17   0.2%  
Kei Nishikori         9   1.9%     9   1.7%     6   1.9%  
Jo Wilfried Tsonga   10   1.6%    12   1.4%     8   1.5%

Again, gElo is probably too optimistic on Djokovic–at least the betting market thinks so–but the point here is the differences between systems. Federer gets a slight bump for entering the top four, and Wawrinka–who gElo really doesn’t like–loses a big chunk of his modest title hopes by falling out of the top four.

The seeding effect is a lot more dramatic if we look at semifinal odds instead of championship odds:

Player              ATP    SF%  Wimb    SF%  gElo    SF%  
Andy Murray           1  58.6%     1  64.1%     2  63.0%  
Rafael Nadal          2  34.4%     4  39.2%     4  38.1%  
Stanislas Wawrinka    3  13.2%     5   7.7%    15   6.1%  
Novak Djokovic        4  66.1%     2  71.1%     1  70.0%  
Roger Federer         5  49.6%     3  64.0%     3  63.2%  
Marin Cilic           6  13.6%     7  11.1%    10  10.3%  
Milos Raonic          7  17.3%     6  14.0%     7  15.2%  
Dominic Thiem         8   7.1%     8   5.4%    17   3.8%  
Kei Nishikori         9  15.5%     9  14.5%     6  15.7%  
Jo Wilfried Tsonga   10  14.0%    12  13.1%     8  14.0%

There’s a lot more movement here for the top players among the different seeding methods. Not only do Federer’s semifinal chances leap from 50% to 64% when he moves inside the top four, even Djokovic and Murray see a benefit because Federer is no longer a possible quarterfinal opponent. Once again, we see the biggest negative effect to Wawrinka: A top-four seed would’ve protected a player who just isn’t likely to get that far on grass.

Surprisingly, the traditional big four are almost the only players out of all 32 seeds to benefit from the Wimbledon algorithm. By removing the chance that Federer would be in, say, Murray’s quarter, the Wimbledon seedings make it a lot less likely that there will be a surprise semifinalist. Tomas Berdych’s semifinal chances improve modestly, from 8.0% to 8.4%, with his Wimbledon seed of No. 11 instead of his ATP ranking of No. 13, but the other 27 seeds have lower chances of reaching the semis than they would have if Wimbledon stopped meddling and used the official rankings.

That’s the unexpected side effect of getting rankings and seedings right: It reduces the chances of deep runs from unexpected sources. It’s similar to the impact of Grand Slams using 32 seeds instead of 16: By protecting the best (and next best, in the case of seeds 17 through 32) from each other, tournaments require that unseeded players work that much harder. Wimbledon’s algorithm took away some serious upset potential when it removed Wawrinka from the top four, but it made it more likely that we’ll see some blockbuster semifinals between the world’s best grass court players.

Men’s Doubles On the Dirt

Angelique Kerber wasn’t the only top seed to crash out early at this year’s French Open. In the men’s doubles draw, the top section opened up when Henri Kontinen and John Peers, the world’s top-ranked team, lost to the Spanish pair of David Marrero and Tommy Robredo. It’s plausible to attribute the upset to the clay, as Kontinen-Peers have tallied a pedestrian five wins against four losses on the dirt this season and one could guess that the Spaniards are at their strongest on clay.

Fortunately we don’t have to guess. Using a doubles variant of sElo–surface-specific Elo, which I began writing about a few days ago in the context of women’s singles–we can make rough estimates of how Kontinen/Peers would fare against Marrero/Robredo on each surface. The top seeds are solid on all surfaces–less than a year ago, they won a clay title in Hamburg–but stronger on hard courts. sElo ranks them 4th and 8th on hard, but 10th and 13th on clay among tour regulars.  Marrero is the surface-specialist of the bunch, ranking 37th on clay and 78th on hard. Robredo throws a wrench into the exercise, as he has played very little doubles recently, only eight events since the beginning of 2016.

Using these numbers–including those derived from Robredo’s limited sample–we find that sElo would have given Kontinen/Peers a 73.6% chance of winning yesterday, compared to a 78.3% advantage on a hard court. Even if we adjust Robredo’s clay-court sElo to something closer to his all-surface rating, the top seeds still look like 69% favorites.

A more striking example comes from yesterday’s other big upset, in which Julio Peralta and Horacio Zeballos took out Feliciano Lopez and Marc Lopez. On any surface, the Lopezes are the superior team, but Peralta and Zeballos have a much larger surface differential:

Player    Hard sElo  Clay sElo  
M Lopez        1720       1804  
F Lopez        1713       1772  
Zeballos       1651       1756  
Peralta        1517       1770

On a hard court, sElo gives the Lopezes a 68.1% chance of winning this matchup. But on clay, the gap narrows all the way to 53.6%. It’s still a bit of an upset for the South Americans, but not one that should come as much of a surprise.

Mismatches

I’ve speculated in the past that surface preferences aren’t as pronounced in doubles as they are in singles. Regardless of surface, points are shorter, and many teams position one player at the net even on the dirt. While some hard-courters are probably uncomfortable on clay (and vice versa), I wouldn’t expect the effects to be as substantial as they are in singles.

The numbers tell a different story. Here are the top ten, ranked by hard court sElo:

Rank  Player          Hard sElo  
1     Jack Sock            1947  
2     Nicolas Mahut        1893  
3     Marcelo Melo         1883  
4     Henri Kontinen       1879  
5     P-H Herbert          1862  
6     Bob Bryan            1851  
7     Mike Bryan           1846  
8     John Peers           1842  
9     Bruno Soares         1829  
10    Jamie Murray         1828

By clay court sElo:

Rank  Player                Clay sElo  
1     Mike Bryan                 1950  
2     Bob Bryan                  1950  
3     P-H Herbert                1894  
4     Nicolas Mahut              1889  
5     Jack Sock                  1887  
6     Robert Farah               1850  
7     Juan Sebastian Cabal       1849  
8     Pablo Cuevas               1824  
9     Rohan Bopanna              1812  
10    John Peers                 1810

Jamie Murray and Bruno Soares, who appear in the hard court top ten, sit outside the top 25 in clay court sElo. Robert Farah and Juan Sebastian Cabal are 41st and 42nd in hard court sElo, despite ranking in the clay court top seven. Pablo Cuevas, another clay court top-tenner, is 87th on the hard court list.

To go beyond these anecdotes–noteworthy as they are–we need to compare the level of surface preference in men’s doubles to other tours. To do that, I calculated the correlation coefficent between hard court and clay court sElo for the top 50 players (ranked by overall Elo) in men’s doubles, men’s singles, and women’s singles. (I don’t yet have an adequate database to generate ratings for women’s doubles.)

In other words, we’re testing how much a player’s results on one surface predict his or her results on the other major surface. The higher the correlation coefficient, the more likely it is that a player will have similar results on hard and clay. Here’s how the tours compare:

Tour             Correl  
Men's Singles     0.708  
Women's Singles   0.417  
Men's Doubles     0.323

In contrast to my hypothesis above, surface preferences in men’s doubles appear to be much stronger than in either men’s or women’s singles. (And there’s a huge difference between men’s and women’s singles, but that’s a subject for another day.)

Randomness

I suspect that the low correlation of surface-specific Elos in men’s doubles is partly due to the more random nature of doubles results. Because the event is more serve-dominated, there are more close sets ending in tiebreaks, and because of the no-ad, super-tiebreak format used outside of Slams, tight matches are decided by a smaller number of points. Thus, every doubles player’s results–and their various Elo ratings–reflect the influence of chance more than the singles results are.

Another consideration–one that I haven’t yet made sense of–is that surface-specific ratings don’t improve doubles forecasts they way that they do men’s and women’s singles predictions. As I wrote on Sunday, sElo represents a big improvement over surface-neutral Elo for women’s forecasts, and in an upcoming post, I’ll be able to make some similar observations for the men’s game. Using Brier score, a measure of the calibration of predictions, we can see the effect of using surface-specific Elo ratings in 2016 tour-level matches:

Tour             Elo Brier  sElo Brier  
Men's Singles        0.202       0.169  
Women's Singles      0.220       0.179  
Men's Doubles        0.171       0.181

The lower the Brier score, the more accurate the forecasts. This isn’t a fluke of 2016: The differences in men’s doubles Brier scores are around 0.01 for each of the last 15 seasons. By this measure, Elo does a very good job predicting the outcome of men’s doubles matches, but the surface-specific sElo represents a small step back. It could be that the smaller sample–using only one surface’s worth of results–is more damaging to forecasts in doubles than it is in singles.

Doubles analytics is particularly uncharted territory, and there’s plenty of work remaining for researchers even in this narrow subtopic. There’s lots of work to do for the world’s top doubles players as well, now that we can point to a noticeably weaker surface for so many of them.

The Steadily Less Predictable WTA

Italian translation at settesei.it

Update: The numbers in this post summarizing the effectiveness of sElo are much too high–a bug in my code led to calculating effectiveness with post-match ratings instead of pre-match ratings. The parts of the post that don’t have to do with sElo are unaffected and–I hope–remain of interest.

One of the talking points throughout the 2017 WTA season has been the unpredictability of the field. With the absence of Serena Williams, Victoria Azarenka, and until recently, Petra Kvitova and Maria Sharapova, there is a dearth of consistently dominant players. Many of the top remaining players have been unsteady as well, due to some combination of injury (Simona Halep), extreme surface preferences (Johanna Konta), and good old-fashioned regression to the mean (Angelique Kerber).

No top seed has yet won a title at the Premier level or above so far this year. Last week, Stephanie Kovalchik went into more detail, quantifying how seeds have failed to meet expectations and suggesting that the official WTA ranking system–the algorithm that determines which players get those seeds–has failed.

There are plenty of problems with the WTA ranking system, especially if you expect it to have predictive value–that is, if you want it to properly reflect the performance level of players right now. Kovalchik is correct that the rankings have done a particularly poor job this year identifying the best players. However, there’s something else going on: According to much more accurate algorithms, the WTA is more chaotic than it has been for decades.

Picking winners

Let’s start with a really basic measurement: picking winners. Through Rome, there had been more than 1100 completed WTA matches. The higher-ranked player won 62.4% of those. Since 1990, the ranking system has picked the winner of 67.9% of matches, and topped 70% during several years in the 1990s. It never fell below 66% until 2014, and this year’s 62.4% is the worst in the 28-year time frame under consideration.

Elo does a little better. It rates players by the quality of their opponents, meaning that draw luck is taken out of the equation, and does a better job of estimating the ability level of players like Serena and Sharapova, who for various reasons have missed long stretches of time. Since 1990, Elo has picked the winner of 68.6% of matches, falling to an all-time low of 63.1% so far in 2017.

For a big improvement, we need surface-specific Elo (sElo). An effective surface-based system isn’t as complicated as I expected it to be. By generating separate rankings for each surface (using only matches on that surface), sElo has correctly predicted the winner of 76.2% of matches since 1990, almost cracking 80% back in 1992. Even sElo is baffled by 2017, falling to it’s lowest point of 71.0% in 2017.

(sElo for all three major surfaces is now shown on the Tennis Abstract Elo ratings report.)

This graph shows how effectively the three algorithms picked winners. It’s clear that sElo is far better, and the graph also shows that some external factor is driving the predictability of results, affecting the accuracy of all three systems to a similar degree:

Brier scores

We see a similar effect if we use a more sophisticated method to rate the WTA ranking system against Elo and sElo. The Brier score of a collection of predictions measures not only how accurate they are, but also how well calibrated they are–that is, a player forecast to win a matchup 90% of the time really does win nine out of ten, not six out of ten, and vice versa. Brier scores average the square of the difference between each prediction and its corresponding result. Because it uses the square, very bad predictions (for instance, that a player has a 95% chance of winning a match she ended up losing) far outweigh more pedestrian ones (like a player with a 95% chance going on to win).

In 2017 so far, the official WTA ranking system has a Brier score of .237, compared to Elo of .226 and sElo of .187. Lower is better, since we want a system that minimizes the difference between predictions and actual outcomes. All three numbers are the highest of any season since 1990. The corresponding averages over that time span are .207 (WTA), .202 (Elo), and .164 (sElo).

As with the simpler method of counting correct predictions, we see that Elo is a bit better than the official ranking, and both of the surface-agnostic methods are crushed by sElo, even though the surface-specific method uses considerably less data. (For instance, the clay-specific Elo ignores hard and grass court results entirely.) And just like the results of picking winners, we see that the differences in Brier scores of the three methods are fairly consistent, meaning that some other factor is causing the year-to-year differences:

The takeaway

The WTA ranking system has plenty of issues, but its unusually bad performance this year isn’t due to any quirk in the algorithm. Elo and sElo are structured completely differently–the only thing they have in common with the official system is that they use WTA match results–and they show the same trends in both of the above metrics.

One factor affecting the last two years of forecasting accuracy is the absence of players like Serena, Sharapova, and Azarenka. If those three played full schedules and won at their usual clip, there would be quite a few more correct predictions for all three systems, and perhaps there would be fewer big upsets from the players who have tried to replace them at the top of the game.

But that isn’t the whole story. A bunch of no-brainer predictions don’t affect Brier score very much, and the presence of heavily-favored players also make it more likely that massively surprising results occur, such as Serena’s loss to Madison Brengle, or Sharapova’s ouster at the hands of Eugenie Bouchard. Many unexpected results are completely independent of the top ten, like Marketa Vondrousova’s recent title in Biel.

While some of the year-to-year differences in the graphs above are simply noise, the last several years looks much more like a meaningful trend. It could be that we are seeing a large-scale changing of a guard, with young players (and their low rankings) regularly upsetting established stars, while the biggest names in the sport are spending more time on the sidelines. Upsets may also be somewhat contagious: When one 19-year-old aspirant sees a peer beating top-tenners, she may be more confident that she can do the same.

Whatever influences have given us the WTA’s current state of unpredictability, we can see that it’s not just a mirage created by a flawed ranking system. Upsets are more common now than at any other point in recent memory, whichever algorithm you use to pick your favorites.