Judging math details | Golden Skate

Judging math details

QuadThrow

Medalist
Joined
Oct 1, 2014
Rounding errors (IceDance worlds 2014)

I had a deeper look at S/M´s FS score from SA:

http://www.isuresults.com/results/season1718/gpusa2017/gpusa2017_Pairs_FS_Scores.pdf

I tried to calculate the PCS of 75.27 points and checked some algorithms.:gclap:

I calculated the trimmed mean of every single component, multiplied by 1.6, rounded to two and added them up and I got 75.25
I calculated the trimmed mean of every single component, multiplied by 1.6, added them up and rounded to two and I got 75.26
I calculated the trimmed mean of every single component, added them up, multiplied by 1.6 and rounded by two and I got 75.26
I calculated the trimmed mean of every single component, added them up, rounded to 2 and multiplied by 1.6 and rounded again and I got 75.26
I calculated the trimmed mean of every single component, rounded to two, added them up, multiplied by 1.6 and I got 75.26
I calculated the trimmed mean of every single component, rounded to two, multiplied by 1.6, rounded to two again and summed them up and I got 75.27!


The last algorithm should be the way the ISU is useing to sum up the components, because they want to receive a value for every single component from 0 to 10.00 which is rounded by two. But sorry. This is the worst algorithm, because you round twice! You learn this during the last years of high school.:angry1:

The ISU should find a way to create scores without rounding
 
Joined
Jun 21, 2003
The ISU should find a way to create scores without rounding

The weirdest case was when Johnny Weir and Evan Lysacek exactly tied at U.S. nationals. Then it was discovered that the algorithm programmed into the computer was different from what the ISU rules actually specified.

The best way, as you say, is not to round at all, either before or after summing and averaging. This would actually be easy. The components are graduated in quarter units, and there are 7 scores to average for each component. So each score (no rounding) is a fraction with 28 in the denominator. Count up the number of 28ths (a whole number), multiply by 1.6, and you have an answer expressed in tenths and of infinite accuracy!
 

bobbob

Medalist
Joined
Feb 7, 2014
Good point...I looked back at worlds 2014 ice dance...turns out Weaver Poje would have been 3rd under exact decimals and Pechalat Bourzat second....is this rounding scheme listed anywhere by ISU...if not, do you think these results could be rectified?
 

Baron Vladimir

Record Breaker
Joined
Dec 18, 2014
Good point...I looked back at worlds 2014 ice dance...turns out Weaver Poje would have been 3rd under exact decimals and Pechalat Bourzat second....is this rounding scheme listed anywhere by ISU...if not, do you think these results could be rectified?

Thats interesting... But when you are working with the numbers in that way, there are rooms for small errors, especially cause those numbers are given by humans observation which is by default prone to mistakes. I think in that case you mentioned or any similar case when differenies in scores are that little they should award the same placements for the skaters at the end (so in case of world 2014 ice dance competition 3 gold medals for example). Because those smalls differences are more product of working with numbers than the judges desicions.
 

Miller

Final Flight
Joined
Dec 29, 2016
Isn't there also some sort of problem with mandatory deductions, can't remember exactly which, but -3 and judges can still put in positive GOEs. Sounds like computer program needs a bit of an overhaul, extra couple of decimal points, to be on the safe side, while doing the calculations then round at the end, that's what we used to do on the bank computer system I worked on when it came to things like interest calculations.
 

moriel

Record Breaker
Joined
Mar 18, 2015
Thats interesting... But when you are working with the numbers in that way, there are rooms for small errors, especially cause those numbers are given by humans observation which is by default prone to mistakes. I think in that case you mentioned or any similar case when differenies in scores are that little they should award the same placements for the skaters at the end (so in case of world 2014 ice dance competition 3 gold medals for example). Because those smalls differences are more product of working with numbers than the judges desicions.

Actually, the little difference is not so little if you look closely at it.
I recall someone analysed the score variance once, and it was quite high, making quite a few current scores impossible to distinguish.

As for rounding up twice, there is an explanation:
ISU displays the mean of each individual component. Currently, scores have two digits, so this one has 2 digits also.
Which means that if they dont average twice, the sum of individual components scores could be different from the PCs.
Now, see people freaking out because of that. Trust me, this would be WAY worse.

The difference is very small, and considering the imprecision of the scores, it is neglectable.
 

QuadThrow

Medalist
Joined
Oct 1, 2014
Good point...I looked back at worlds 2014 ice dance...turns out Weaver Poje would have been 3rd under exact decimals and Pechalat Bourzat second....is this rounding scheme listed anywhere by ISU...if not, do you think these results could be rectified?

I analyzed the score from Ice Dance at worlds 2014. took me hours

ISU final score:

C/L 175.43
W/P 175.41
P/B 175.37

my score without rounding:

C/L 175.4554...
W/P 175.3946...
P/B 175.4054...

The bronze and the silver has to be switched!

Maybe I should write a letter to the ISU.


Concerning human errors: Of course judges are people with errors which affect the score by far more than rounding. But the solution is not to accept the wrong decision. The solution must be a scoring system with less selectivity which accepts three Gold medals.
 
Joined
Jun 21, 2003
....is this rounding scheme listed anywhere by ISU?

Yes, it is clearly specified in the ISU rules. I think that they use the "round, then sum" algorithm for the reason that Moriel mentions. You want the displayed numbers to add up to the displayed total, otherwise the Internet would explode when our favorite guy lost by a tenth of a point. :laugh:

By the way, I think that you can download the actual code from the ISU. You might have to be a skating club to get it, though.

I think in that case you mentioned or any similar case when differenies in scores are that little they should award the same placements for the skaters at the end (so in case of world 2014 ice dance competition 3 gold medals for example). Because those smalls differences are more product of working with numbers than the judges decisions.

I think that the ISU feels that a sports contest should always have a winner, come hell or high water. In the 6.0 era the judges were forbidden from giving the exact same score to two skaters. I don't think that the ISU would ever admit that their numbers are fuzzy.
 

gkelly

Record Breaker
Joined
Jul 26, 2003
I think that the ISU feels that a sports contest should always have a winner, come hell or high water. In the 6.0 era the judges were forbidden from giving the exact same score to two skaters.

Which didn't stop it from happening occasionally, usually in large events where a judge lost track of a score s/he had given to a different skater hours earlier in the messy middle of the standings. I remember finding two or three examples in the ladies' event at 1994 Olympics.

And of course when individual judges did their jobs but disagreed with each other, producing mixed ordinals across the panel, sometimes even after exhausting all the tiebreakers the skaters still ended up tied for that phase of the event. Usually the results of the other phases would sort out the final results.

I remember seeing results posted for one of the compulsory dances in a novice dance event at US Pacific Coast sectionals in the early 2000s (IIRC) in which there were 5 judges who completely disagreed with each other, so 5 teams ended up tied for 1st place in that dance.

With 9 judges, you could end up with 3 skaters/teams perfectly tied.

Also, as I recall, at the 1996 Centennial on Ice competition, Maria Butyrskaya tied for 1st in the short program and tied for 2nd in the free skate, but with different skaters each time, so she ended up sole 2nd overall.
 
Joined
Jun 21, 2003
I remember seeing results posted for one of the compulsory dances in a novice dance event at US Pacific Coast sectionals in the early 2000s (IIRC) in which there were 5 judges who completely disagreed with each other, so 5 teams ended up tied for 1st place in that dance.

This is called the "rock-paper-scissors-lizard-Spock" paradox. :)
 

GGFan

Record Breaker
Joined
Nov 9, 2013
Concerning human errors: Of course judges are people with errors which affect the score by far more than rounding. But the solution is not to accept the wrong decision. The solution must be a scoring system with less selectivity which accepts three Gold medals.

It's crazy to me that we decide medals on these margins. Has anyone ever done a margin of error analysis on the entire system? That has always been my issue with this system. Figure skating is of course not the only sport that has the problem of deciding medals on margins that don't hold up. Track and field for example is a hot mess given the differences in wind, etc. But human beings like rankings so 3-gold medal event would be unsatisfying. I mean the IOC forces sports to limit ties. If you don't they kick you out.

Despite the many problems with 6.0 I did like that at the end of the day it was very clear that this was an opinion and sometimes a heavily biased one. Numbers can give a sense of scientific precision that doesn't exist in figure skating. Argue, but have fun with it :dev2:
 
Joined
Jun 21, 2003
Numbers can give a sense of scientific precision that doesn't exist in figure skating...

That is my main gripe about the IJS. Look everybody, I got 204.22 points. Pity the fool who only got 204.21.

Argue, but have fun with it :dev2:

And that is my second gripe. Ordinal judging is inherently fun to play with mathematically, as well as fun to argue about. IJS -- add up a row a numbers. All we have to argue about is, should this jump have received a < or not.

Oh well. points is points. I am getting used to it.
 

QuadThrow

Medalist
Joined
Oct 1, 2014
It's crazy to me that we decide medals on these margins. Has anyone ever done a margin of error analysis on the entire system? That has always been my issue with this system. Figure skating is of course not the only sport that has the problem of deciding medals on margins that don't hold up. Track and field for example is a hot mess given the differences in wind, etc. But human beings like rankings so 3-gold medal event would be unsatisfying. I mean the IOC forces sports to limit ties. If you don't they kick you out.

Despite the many problems with 6.0 I did like that at the end of the day it was very clear that this was an opinion and sometimes a heavily biased one. Numbers can give a sense of scientific precision that doesn't exist in figure skating. Argue, but have fun with it :dev2:

It is quite fascinating that there is almost no discussion about the adequacy of the current scores. Numbers always have the aura of precision and truth around them.:bow:

Why don´t we messure the scores by intervals of half a point on each element and each component? The big majority of results would be the same.
 

Anyasnake

Record Breaker
Joined
Mar 28, 2016

moriel

Record Breaker
Joined
Mar 18, 2015
That is my main gripe about the IJS. Look everybody, I got 204.22 points. Pity the fool who only got 204.21.



And that is my second gripe. Ordinal judging is inherently fun to play with mathematically, as well as fun to argue about. IJS -- add up a row a numbers. All we have to argue about is, should this jump have received a < or not.

Oh well. points is points. I am getting used to it.

And the worst is, they dont really have the 2 digits precision there at all, considering the judging method.
Honestly, I even suspect that the interval would be larger than +-1, considering GOE and PCs


Mildly tempted to do some bootstrapping with real scores to see what happens
 
Joined
Jun 21, 2003
Why don´t we messure the scores by intervals of half a point on each element and each component?

It would be even cooler if we measured by full points. Then the decimal part of the average for each component would always be 0 or the digits

142857

always in the same order, but just starting at a different place (like 285714). :)
 
Joined
Jun 21, 2003
And the worst is, they dont really have the 2 digits precision there at all, considering the judging method.
Honestly, I even suspect that the interval would be larger than +-1, considering GOE and PCs.

It would be weird, though, in the context of a sporing event. Suppose one guy got a total score of 254.14 +/- 1.31 and another guy has 253.88 +/ 1.24. We could say "too close to call" or "We cannot be 95% confident that the first skater really skated the best." Or we could say something like "The probability is 56% that the first guy is the true winner and 44% that the second guy is. Now what?
 
Last edited:

QuadThrow

Medalist
Joined
Oct 1, 2014
It would be weird, though in the context of a sporing event. Suppose one guy got a total score of 254.14 +/- 1.31 and another guy has 253.88 +/ 1.24. We could say "too close to call" or "We cannot be 95% confident that the first skater really skated the best." Or we could say something like "The probability is 56% that the first guy is the true winner and 44% that the second guy is. Now what?

The 95% confidence interval would be a really progessive method and a good shot form the mathematics point of view. All science are using this 95% interval so why not sports?

Honestly I think this will never happen and it would make a whole world of difference. But only because we all do not grow up with those statistics and we all have not get used to it so far.
 

moriel

Record Breaker
Joined
Mar 18, 2015
It would be weird, though in the context of a sporing event. Suppose one guy got a total score of 254.14 +/- 1.31 and another guy has 253.88 +/ 1.24. We could say "too close to call" or "We cannot be 95% confident that the first skater really skated the best." Or we could say something like "The probability is 56% that the first guy is the true winner and 44% that the second guy is. Now what?

Oh i know they wouldnt do it, but in practice those scores probably have some precision like that.
But, in this case, they could say those 2 skaters are tied and go to tiebreakers.

I personally think that it would be better if the scores were rounded to closest integer instead, and then, if two final scores are equal, tiebreakers (such as TES, BV, etc) could be used.
 
Top