Rescoring past competitions w/ new SOV+rules

There’s been a lot of speculation on how competition results might change with the flurry of rule changes that were passed at the recent ISU congress, as well as the update scale of values that’s supposed to go into effect next season. In order to arrive at a more grounded idea of what might happen, I decided to rescore some old competitions according to the new rules. This chiefly affects the men’s competition, but I figured some of you might be interested in ladies as well, so I also did a little rescoring for ladies (I found it mostly fairly boring, the scores didn’t really change very much, though placements sometimes did).

Here are some of the more interesting results:

2018 Worlds
1. Nathan Chen 321.4 (old score) -> 306.84 (new score)
2. Mikhail Kolyada 272.32 -> 264.9 (moves from 3rd to 2nd)
3. Shoma Uno 273.77 -> 263.89 (moves from 2nd to 3rd)

2018 Olys
1. Yuzuru Hanyu 317.85 -> 314.06
2. Javier Fernandez 305.24 -> 301.15 (moves from 3rd to 2nd)
3. Shoma Uno 306.9 -> 296.04 (moves from 2nd to 3rd)

2017 Worlds
1. Yuzuru Hanyu 321.59 -> 317.47
2. Shoma Uno 319.31 -> 312.18
3. Javier Fernandez 301.19 -> 298.51 (moves from 4th to 3rd)
4. Boyang Jin 303.58 -> 296.04 (moves from 3rd to 4th)

2017 4 Continents
1. Yuzuru Hanyu 303.71 -> 299.39 (moves from 2nd to 1st)
2. Nathan Chen 307.46 -> 295.59 (moves from 1st to 2nd)

Notable for being the one case in mens I checked where the total score actually rose instead of dropping (I doubt there are other cases):
2015 GPF
1. Yuzuru Hanyu 330.43 -> 331.29

(Yes, somehow despite the removal of a triple jump AND the quad BV nerf, GPF2015 Yuzuru Hanyu manages to have a higher score under the new system than the old one. Go figure.)

Ladies:

2018 Worlds
1. Kaetlyn Osmond 223.23 -> 224.75
2. Carolina Kostner 208.88 -> 211.51 (moves from 4th to 2nd)
3. Wakaba Higuchi 210.9 -> 211.17 (moves from 2nd to 3rd)
4. Satoko Miyahara 210.08 -> 209.77 (moves from 3rd to 4th)

There are no changes to Olympics 2018 rankings for the top 6, though the margin of Alina’s victory does narrow to a few tenths of a point.

The full results may be found here for men, and here for ladies.

Methodology
The changes that I took into account were:
1. The updated scale of values
2. Changes to backloading bonus
3. Removal of one jumping pass in the men’s competition

I did NOT take into account the new quad repetition rule (it doesn’t affect many skates—only Nathan Chen 2018 Worlds + Olys, Yuzuru Hanyu 2018 Olys out of the skates that I rescored—but I would have to redo the program for those skates and it would just mess up the rescore too much. If you want to have an idea of how those programs might score if they obeyed the new quad repetition rule, subtract a few points.)

All elements were rescored using their new BVs and the new GOE scale running from -5 to 5. I did not personally rescore the elements, but converted them by finding the average of the original judges’ scores from -3 to 3 and multiplying by 5/3. If you look at the score break down, most of the junk in the various columns labeled some variation of “GOE” are steps in this calculation.

In order to determine which jumps to give the backloading bonus to, I largely chose to go with the 3 intended jumps of highest BV out of the jumps that the skater received the backloading bonus for under the original rules. There were a couple of cases where I thought it was unlikely that the skater would backload the jumps selected by this methodology (for instance, I didn’t think they would backload a lutz if they could only choose 3 jumping passes because the skater often gets edge calls on the lutz), so I did use a little discretion. The jumps that I gave the backloading bonus to are indicated by an x, like they usually are in protocols, so if you disagree with me, feel free to rerun the calculations using your own judgment.

For the removal of one jumping pass, I removed the lowest BV intended jump (ie. if someone popped a 4S into a 2S, I did not remove the 2S, because it was unintentional). Sometimes, the lowest BV intended jump was the first jump in a combination. In that case, I removed the first jump, but kept the base value for the rest of the combination, so that the skater would not wind up with less BV owing to having fewer combinations than they should.

Limitations
Obviously, the methodological assumptions I made come with some limitations. For instance, we do not know how the judges will actually use the -5 to 5 scale in practice. Their behavior may differ significantly from the way the -5 to 5 scale is modeled here. (Edit: in particular, I think there's a good chance they'll be stingier about scores in the 4-5 range than past judges were about handing out 3s. If so, you can account for this in a ballpark manner by reducing the scores of high GOE skaters like Yuzuru, Javier, Carolina, Alina, and Evgenia by a few points.)

Changes to the backloading bonus may significantly affect program compositions, and the quality of jumps landed under those new program compositions (less backloading=less tired=higher GOEs, possibly). I cannot account for these types of effects using my methodology, which is essentially just a mechanical conversion of scores using some simple rules. Nor do I know which jumps a skater will choose to backload now that they are more limited in the number of jumps they get the bonus for—I can only make educated guesses.

I have the sense that combinations tend to have lower GOEs than solo jumps, so its possible that in the cases where a deleted jump was part of a combo, the fact that the skater still gets the BV for the rest of the combo without “having to attach it” to a different jump potentially means that the skater has slightly higher GOEs than they would get in an actual performance of the altered program.

Finally, while I have done *some* error checking, I haven’t gone through with a fine tooth comb or anything, so it’s possible that there are mistakes. If you find a mistake, please alert me (unless it’s about negative GOEs on quads—negative GOEs on quads are factored weirdly under the old rules, so when I input the original GOEs I adjusted them slightly to make them easier to convert).

Taking all these factors into account, it’s important to remember that the numbers here are only an estimate of how a skate might have scored under these new rules. Therefore, it’s probably best not to get hung up on minor differences of scores. Even though competition placement changes might be more dramatic, I think looking at the actual numbers and thinking of them as a ballpark figure is more useful.

Conclusions
Overall, as might be expected, the new rules favor skaters who can score high in GOE. If we compare the changes to different skaters’ scores, we’ll see that of the men, the skaters who see the least decrease in score are skaters like Yuzuru Hanyu and Javier Fernandez. On the other hand skaters who depend more on BV for their scores, like Nathan Chen, suffer a lot more. Over the competitions I looked at, Nathan Chen typically scores about 10 points lower in the free skate under the new system versus the old one, even when he is relatively clean, whereas Yuzuru Hanyu’s points go down a far more modest 2 to 6 points, typically around 4. Yuzuru actually gains points in the short provided he’s clean (as does Javier Fernandez), whereas even on Nathan’s cleaner skates he’s losing points relative to his scores under the old system. Boyang Jin and Shoma Uno, who are to varying extents also more BV skaters than GOE ones, also lose a fairly significant amount of points relative to the old system, but not as much as Nathan.

Note that contrary to some speculation I’ve seen, the advantage that a high GOE skater has does not go away when the skater does not skate clean. Just to illustrate, when rescored, Nathan’s personal best doesn’t even beat Yuzuru Hanyu’s Olympics score, despite the fact that Yuzuru made two jumping errors in that competition, one of which was fairly serious. (Hell, even if we add up Nathan Chen’s personal bests from different competitions, it doesn’t match Yuzuru’s Olympic score.) This is because even if a high GOE skater makes a mistake on one or two elements, the extra GOE they get on the other elements in comparison with the GOE from the old system makes up for the loss. On the ladies side, when Carolina Kostner completes her elements successfully, she gets high GOE, and though this effect is less pronounced in ladies, it does allow her to move from 4th to 2nd in the 2018 Worlds re-score despite having an error-riddled free skate (well, unless someone can find a mistake in my calculations, since the scores are really close together in that competition).

However, the new rules do a good job of punishing skaters who are messy. While scores don’t go down that much for a single error, once there are multiple errors the effect starts becoming very apparent. For instance, in the 2018 Worlds Ladies competition, Alina Zagitova (207.72->203.74) scores closer to Bradie Tennell (199.89->200.27) than to Carolina Kostner (208.88->211.51) or Satoko Miyahara (210.08->209.77). Alina, as most of us probably unfortunately remember, fell down several times in her free skate, and is punished significantly more harshly for it under the new system. On the men’s side, Nathan Chen’s Olympic short (I’m starting to feel bad for picking on Nathan lol) drops another 8 points under the new system. Ouch.

TES drops significantly in the men’s and increases slightly in ladies’, at least for the highest levels of competition. The drop in TES in men's is largely attributable to the loss of a jumping pass. GOE increases more than make up for BV nerfs, whether those nerfs come in the form of reductions in jump values or restrictions on the backloading bonus. And apparently, GOEs can even make up for the loss of a jumping pass if your name is Yuzuru Hanyu (although notably if we add up individual segment personal bests, Javier Fernandez also manages to outscore himself on the new system versus the old one. Maybe it’s a little early to retire?)

Overall, my impression is that if you wanted to design a scoring system to encourage skaters to be more like Yuzuru Hanyu, this new system would be a pretty good attempt. Of course, the skater who is most like Yuzuru Hanyu…is Yuzuru Hanyu. When rescored, Yuzuru Hanyu’s personal best is nearly 20 points better than the next highest scoring man’s, none of whom manage to beat even his 4th highest scoring skate (2018 Olys) with their personal bests. Javier Fernandez comes the closest (and he could if he could put two clean skates together), but he’s still a couple of points off and is unlikely to try in the future. Shoma Uno is 3rd (3.5ish points off of 2018 Olys Yuzu, 20+ points off 2015 GPF Yuzu), and poor Nathan Chen, who the new rules seem to target with a vengeance, is 4th on the personal bests list (~7 points short of 2018 Olys Yuzu, ~24 points short of 2015 GPF Yuzu).

Again, we don’t know how this new system is going to look like in practice, so the calculations here are only a model of how past competitions might have looked. Still, what we see so far is highly suggestive.

Additional Conclusions

First additional conclusion is that I have too much time on my hands. The second is that this new system does succeed in making it easier for skaters with high quality triples to outscore skaters with meh quality elements who can land a quad. We can illustrate this by looking at Deniss Vasiljevs' result in 2018 Worlds. Originally, Deniss, who does not jump quads, placed below both Alexei Bychenko and Kazuki Tomono, both of whom jumped quads in the SP and the FS, but on rescore, he places 4th, ahead of both of them.

4. Deniss Vasiljevs 254.86 -> 252.72 (moves from 6th to 4th)
5. Alexei Bychenko 258.28 -> 252.41 (moves from 4th to 5th)
6. Kazuki Tomono 256.11 -> 249.52 (moves from 5th to 6th)

However, despite Shoma Uno and Mikhail Kolyada's many mistakes in the free skate, Deniss is still well (specifically about 11 points) away from outscoring them due to their BV and PCS advantages--though note that Shoma actually received slightly lower TES than Deniss, without even including deductions. The gap between Deniss and Shoma does drop significantly, however. Under the old system there's 19 points separating the two.

Also, the alterations to the backloading bonus rules ultimately did not have a very significant effect on competition results, at least in the competitions I examined. Alina is still Olympic Champion off the back of her high GOEs and BV and perfectly competitive PCS. The gap between Alina+Evgenia and the others did not change very much and in some cases actually widened under application of the new rules, largely because the two of them also receive very high GOE scores. Though the gap between Alina and Evgenia narrowed, I believe this has as much to do with Zhenya's higher GOEs in the FS as it does with Alina's backloading (Alina lost a couple points on her first 3Lz because she received relatively low GOE for that element.) Therefore, all this drama about backloading may, at the end of the day, be much ado about nothing.

Finally, popped jumps are now in general significantly less bad errors than falls, at least if you only go down one rotation and don't get GOE that's too negative (though it depends on a lot of factors, so this is only a rule of thumb). We can illustrate this by looking at Carolina Kostner's FS in 2018 Worlds. In the FS, Caro popped two jumps: her 3Lz and her 2A. She received a score of 2.6 (positive GOE) and 0.72 (negative GOE) respectively on those elements in the re-score. If she had fallen while fully rotating both of those jumps instead, she would have received

5.9/2=2.95 (-50% from -5 GOE)
2.95-1=1.95 (deduction)

instead for the 3Lz, 0.65 less than her score of 2.6 for her popped element, which had positive GOE

and 3.3/2=1.65 (-50%)
1.65+0.33 points=1.98 (backloading bonus)
1.98-2 = -0.02 (deduction)

For the 2A (she gets an extra point deducted because with the 3S, that would have been 3 falls. Note that if she only fell on the 2A, then I believe 2A-fall would have been better than 1A with -4.5 GOE because of both the GOE and the backloading bonus.)

In total, by popping her jumps instead of falling, Carolina received 3.32-1.93=1.39 extra points--more than her margin of victory over Wakaba in the re-score.

Under old rules, she would have received
6.1-2.1-1=3 points for falling on a 3Lz (more than what she actually got, 2.53 points)
and
3.3*1.1-1.5-2=0.13 for falling on 2A

For a total of 3.13 points, whereas she actually received 3.2 points--about the same.

For falling versus popping on quad jumps, we see that (taking 4S as an example):
4S-fall: 9.7/2=4.85
4.85-1=3.85

whereas

3S, 0GOE: 4.3
3S, -1GOE: 3.87

so again, as long as the GOE isn't too negative, popping is better than falling (especially if there are multiple falls in the program). However, quads tend to be popped into doubles, which are strictly worse than falling on a quad (unless you've already fallen many, many times--but at that point it hardly matters).

Thanks. Very interesting.
So most of ladies' scores rise while men's drecrease i see. So who have triples is more rewarded than who have quads.

Shanshani said:

2018 Worlds
1. Nathan Chen 321.4 (old score) -> 306.84 (new score)
2. Mikhail Kolyada 272.32 -> 264.9 (moves from 3rd to 2nd)
3. Shoma Uno 273.77 -> 263.89 (moves from 2nd to 3rd)

Click to expand...

A good show of why this fixes nothing -- Kolyada and Uno should both have finished off the podium. Yet, there they both are, because of the huge PCS handed to them.

2018 Olys
1. Yuzuru Hanyu 317.85 -> 314.06
2. Javier Fernandez 305.24 -> 301.15 (moves from 3rd to 2nd)
3. Shoma Uno 306.9 -> 296.04 (moves from 2nd to 3rd)

Click to expand...

This works out! Of course, Uno's 4Lo< remains uncalled.

2017 Worlds
1. Yuzuru Hanyu 321.59 -> 317.47
2. Shoma Uno 319.31 -> 312.18
3. Javier Fernandez 301.19 -> 298.51 (moves from 4th to 3rd)
4. Boyang Jin 303.58 -> 296.04 (moves from 3rd to 4th)

Click to expand...

This doesn't. What swapped Fernandez and Jin? The results were bang on.

Ah, Fernandez's undeserved GOE make the SP margin between Jin and him larger. Of course. Yet again, this fixes nothing.

2017 4 Continents
1. Yuzuru Hanyu 303.71 -> 299.39 (moves from 2nd to 1st)
2. Nathan Chen 307.46 -> 295.59 (moves from 1st to 2nd)

Notable for being the one case in mens I checked where the total score actually rose instead of dropping (I doubt there are other cases):
2015 GPF
1. Yuzuru Hanyu 330.43 -> 331.29

Click to expand...

Meh. 20174CC is just a travesty in general, and scores getting increased doesn't really mean anything in FS, where differences are all that matter.

2018 Worlds
1. Kaetlyn Osmond 223.23 -> 224.75
2. Carolina Kostner 208.88 -> 211.51 (moves from 4th to 2nd)
3. Wakaba Higuchi 210.9 -> 211.17 (moves from 2nd to 3rd)
4. Satoko Miyahara 210.08 -> 209.77 (moves from 3rd to 4th)

Click to expand...

Oh it will be fun this year!

chopinskate said:

This doesn't. What swapped Fernandez and Jin? The results were bang on.

Click to expand...

To be honest, this was a little bit of a quirk of methodology. For the most part, when skaters had a jump dropped, they lost a positive GOE jump (since it's their easiest jump that's dropped). However, Javi actually screwed up his easiest jump in 2017 Worlds, getting negative GOE, so dropping that element actually wound up benefitting him relative to others.

Shanshani said:

To be honest, this was a little bit of a quirk of methodology. For the most part, when skaters had a jump dropped, they lost a positive GOE jump (since it's their easiest jump that's dropped). However, Javi actually screwed up his easiest jump in 2017 Worlds, getting negative GOE, so dropping that element actually wound up benefitting him.

Click to expand...

Fernandez also had 2 more points in the SP than he did originally -- the problem of judging remains, as the GOE and PCS in that program were undeserved. This is exactly what is going to happen again, because the judges haven't changed, so why will the judging?

I didn't evaluate the program itself, so I don't have anything objective to say about whether the GOE and PCS is undeserved (based off my impressions, I also would have preferred Boyang to come in 3rd and I do think Javi is a little overscored in general). But regardless, it's true that if a skater gets generous GOE, the effect of that GOE will be magnified by the new system.

Boyang should be a BV and GOE skater. Again, it isn't the skater the problem but the judging.

This is really interesting. It looks like Hanyu's world record is here to stay!

Congratulation! Exellent work and big thank you!

I find especially interesting that skaters who get high GOE but have error ridden programms don`t get punished by the new system (see Kostner scores at world). In other words it means skaters with low BV who are loved by judges are the big profiteur of the rule changes. They will skate easier programms, get high GOE and still can afford a step-out, hand down here and there.

Wow, this will create a lot of controversy.

Thank you. Really interesting !
Quite surprised to see than some scores for the men are going up though, which means breaking records is still possible. But well, I'm not complaining

Fascinating. Thanks so much for putting the time into this.

If I'm understanding correctly, you converted the given GOE to it's numerical equivalent under the new system rather than rescoring. I think the one flaw with this type of analysis is that I think the judges' mindset will change with the new system. It's natural human tendency to play it safe and the greater the options, the greater the liklihood to fall into this trap - my guess is that judges will be pretty stingy with the extremes of the scale no matter how deserving the element may be except in cases where it is explicit like a fall. In the first season, I'll be surprised to see many GOE's above 4 and I'm guessing even above 3.5 will be rare. With this in mind, at least next year, I doubt Nathan and Shoma will be hit as hard as we are predicting either

Otoh, of the handful of competitions I studied under the old system, except for the occasional jump here or there, I find the composite GOE scoring to be pretty fair - only very stingy at >2.5 and deductions for SP required steps that no longer exist, so I'm very cautiously optimistic about it.

Of course, 95% of the judging problems come from either the technical caller or the PCS so even if the GOE scoring is perfect, it doesn't really fix anything.

Wow, impressive work! Huge thanks for compiling all of this

So, numerically and ideally speaking, the new system does value quality over quantity (TES over BV) as it says. But Worlds 2017 rescored did show that reputation GOE judging cannot be fixed by changing the system - it might even get amplified. In the end, it all comes down to judging :confused2:

Still, a fun mental exercise :agree:

Old Cat Lady said:

Fascinating. Thanks so much for putting the time into this.

If I'm understanding correctly, you converted the given GOE to it's numerical equivalent under the new system rather than rescoring. I think the one flaw with this type of analysis is that I think the judges' mindset will change with the new system. It's natural human tendency to play it safe and the greater the options, the greater the liklihood to fall into this trap - my guess is that judges will be pretty stingy with the extremes of the scale no matter how deserving the element may be except in cases where it is explicit like a fall. In the first season, I'll be surprised to see many GOE's above 4 and I'm guessing even above 3.5 will be rare. With this in mind, at least next year, I doubt Nathan and Shoma will be hit as hard as we are predicting either

Otoh, of the handful of competitions I studied under the old system, except for the occasional jump here or there, I find the composite GOE scoring to be pretty fair - only very stingy at >2.5 and deductions for SP required steps that no longer exist, so I'm very cautiously optimistic about it.

Of course, 95% of the judging problems come from either the technical caller or the PCS so even if the GOE scoring is perfect, it doesn't really fix anything.

Click to expand...

Oh yeah, definitely. That's a huge assumption, and I doubt we'll be seeing very many +4-5 elements under this new system in actuality (although, who knows what's going on in the judges' head). So the advantages of people like Yuzu, Javi, and Carolina will probably be fewer than my model makes them out to be. But converting GOEs directly is the most objective way of modeling how the new system might work, in my opinion, since we can't know what's going on in the judges' head other than what past judges have already committed to paper.

halulupu said:

Congratulation! Exellent work and big thank you!

I find especially interesting that skaters who get high GOE but have error ridden programms don`t get punished by the new system (see Kostner scores at world). In oder words it means skaters with low BV who are loved by judges are the big profiteur of the rule changes. They will skate easier programms, get high GOE and still can afford a step-out, hand down here and there.

Wow, this will create a lot of controversy.

Click to expand...

Yes Kostner will enjoy this new system.

This system is similar to the old system of 2006-2010 quad i think. Yuna Kim could afford to have two pops and still get high TES, sometimes even the highest. Her GOEs were unmatchables (deservly, unlike Kostner high GOEs). Mao was the Nathan Chen of that system. Or Shoma Uno would be more accurate (but she got dinged much much more).

dobrevaria said:

Thank you. Really interesting !
Quite surprised to see than some scores for the men are going up though, which means breaking records is still possible. But well, I'm not complaining

Click to expand...

Breaking records in the short should actually be easier, as in general SP scores actually go up under the new system versus the old, at least for people who execute well like Yuzuru and Javier. Breaking records in the FS will be signficantly more difficult, but should in theory be doable, though you would need squeaky, squeaky clean and beautiful elements (even the cleanest FS scores go down by a couple of points in the rescore). Ladies should have an easier time breaking records period: in general, ladies scores go up slighty for top tier skaters.

Shanshani said:

Breaking records in the short should actually be easier, as in general SP scores actually go up under the new system versus the old, at least for people who execute well like Yuzuru and Javier. Breaking records in the FS will be signficantly more difficult, but should in theory be doable, though you would need squeaky, squeaky clean and beautiful elements (even the cleanest FS scores go down by a couple of points in the rescore). Ladies should have an easier time breaking records period: in general, ladies scores go up slighty for top tier skaters.

Click to expand...

Yuzu's 223 score at Worlds 2017 could have been higher back then. His PCS in Barcelona was nearly two points better than in Helsinki and he didn't score max. GOE on any element, so he could have finished somewhere around 228-230, if he had skated last or the judges had been in a better mood that day.

That's why I think, the 223 is still in sight for Yuzu, especially with a potential 4A and 4Lz in the bag. I'm not that sure about the rest of the field at the moment.

Question: Am I right in the assumtion that the new system "incentives" popping over falling? Let´s say a not too ugly popped double triple which still gets +1 GOE is worth more than a rotated triple with a fall (didn`t calculate)

halulupu said:

Question: Am I right in the assumtion that the new system "incentives" popping over falling? Let´s say a not too ugly popped double triple which still gets +1 GOE is worth more than a rotated triple with a fall (didn`t calculate)

Click to expand...

Probably? If we look at Carolina Kostner's 2018 Worlds Free Skate, we see that she gets 2.6 points for a popped 2Lz, which received quite good GOE (1.43 unfactored on the -3 to 3 scale). 3Lz-fall would be 5.9/2=2.95 BV, but taking into account the deduction, she would net 1.95 off a 3Lz-fall, less than off her 2Lz.