Could judges be replaced by computers in giving TES score?

[CanadianSkaterGuy]

But I think it's hard to get a good sense of what is "average" vs. "good" when you're mainly focusing on the elite. Even the 24th best skater in the world is typically well above average compared to the senior field as a whole. And the average senior is typically much stronger than the average skaters at lower levels, who are also scored by the same standards.

This is an important point. The 24th best skater in the world might not have the best 2A. But if they perform a double axel with what is perceived as very good height/distance, then a GOE bullet should be awarded for it. These are elite skaters, so several of them are performing elements in a fashion that meet GOE criteria.

GOE bullets are reserved for any skater who sufficiently achieves them -- not reserved only for the skaters who are executing the best version of that GOE bullet.

 

[Mathman]

Again, the GOE bullet is very good height/length - not "superior" height/length ... or anything that implies how that element was executed relative to other skaters.

But what could "very good height/length" possibly mean besides, compared to the standards set by the great body of figure skaters actually performing them?

(For that matter, what could "superior" height and distance mean except the same thing?)

 

[Sam-Skwantch]

You guys are wrinkling my brain :shock2:

 

[zounger]

I think the problem would be, how would the computer know when it is getting better instead of getting worse? In successful artificial intelligence projects there is a well-defined goal. When the machine does something that gets it closer to the goal, that sort of behavior is internally rewarded and repeated.

The canonical example would be something like a program that plays chess. The machine "knows" whether it won or lost. What would constitue "winning" a figure skating judging exercise?

Humans will give to the machines/computers GOEs if they did their job correctly And here you have your "learning" function.

 

[Mathman]

Humans will give to the machines/computers GOEs if they did their job correctly And here you have your "learning" function.

Exactly. And the result will be a computer who judges just like humans do.

But we already have judges that judge like humans do. :yes:

 

[gkelly]

But what could "very good height/length" possibly mean besides, compared to the standards set by the great body of figure skaters actually performing them?

(For that matter, what could "superior" height and distance mean except the same thing?)

Suppose that 1,000 skaters around the world in each discipline executed triple toe loops in competition sometime this season.

How many of them do you think might achieve "very good height/length"? Choose numbers if you want the definition to be measurable. Maybe different numbers for men and women. Maybe different numbers for different competition levels or based on the age or the height of the skater, though I don't think that's a profitable can of worms to open.

Now suppose you set the number high enough that only 50 (5%) of all skaters achieved it. Or maybe there's a more natural cutoff point at 10% of the triple toe population.

Let's say half of those skaters with the biggest jumps competed at Worlds. And almost all who qualified for the freeskate were among them.

So if you watch senior B or JGP competition, you might only see one or two skaters with triple toes big enough to qualify. But if you watch the Worlds freeskate, you might only see one or two skaters whose triple toes don't qualify.

By the standards of all triple toes in the world this year, almost all the examples you saw had "very good height and distance." But if you're defining "very good" as "significantly bigger than the majority of triple toes at Worlds," then you might be setting standards that would result in less than 1% of all examples qualifying.

And of course there will be skaters who have big jumps but for other reasons typically lose more GOE points elsewhere than they gain for height and distance.

 

[Mathman]

Now suppose you set the number high enough that only 50 (5%) of all skaters achieved it. Or maybe there's a more natural cutoff point at 10% of the triple toe population.

Let's say half of those skaters with the biggest jumps competed at Worlds. And almost all who qualified for the freeskate were among them.

So if you watch senior B or JGP competition, you might only see one or two skaters with triple toes big enough to qualify. But if you watch the Worlds freeskate, you might only see one or two skaters whose triple toes don't qualify.

That is what I expect would happen. I don't see anything wrong with that.

Still, I think we could incorporate that expectation into the 0 GOE category. A certain amount of height and distance are required to qualify for "OK." 0 GOE = OK in all aspects, or else good in some, bad in others.

I don't know. I don't really have a strong position on this. The only thing I don't like is that someone can get a +5 just by getting pretty good height, pretty good air position, pretty good landing edge, does not offend the music. etc. It seems like there should be something special for such a large reward.

 

[CanadianSkaterGuy]

But what could "very good height/length" possibly mean besides, compared to the standards set by the great body of figure skaters actually performing them?

(For that matter, what could "superior" height and distance mean except the same thing?)

It's still not really easy to say what determines "very good" height and distance. And it shouldn't be a relative thing -- to me "superior" implies having some of the best amplitude - but since skaters aren't marked on elements relative to each other, or rather shouldn't be, then that adjective is not appropriate.

Hypothetically, which skaters in each of these scenarios should be awarded very good height/distance?

1) skater A jumps 0.2m/1m; skater B jumps 0.3m/1.1m; skater C 0.25m/1.05m -- all jump around the same height which isn't very good but relative to each other they're about the same (to me, none of them get the height/distance bullet)

2) skater A B and C all jump 0.8 m/3.5m -- all jump exactly the same with none of them jumping better relative to the others (all of them get the height/distance bullet)

3) skater A and B jump 1.0 m/4 m and skater C jumps 1.05m/4.1m -- all jump huge, well above average, but C is the biggest (all of them get the height/distance bullet; no bonus for C for having the highest)

The thing about comparing skaters using relative numbers ... how do judges then assess the first skater in a field, when there is no IceScope or "relative" height/distance from other skaters to compare them to? The only way you can assess height/distance with technology and accordingly give out a bonus is to set an actual threshold number for each, and then any skater who surpasses that number gets the GOE bullet. If the threshold is 0.5m height and 2.5m length, then a skater should get that GOE bullet whether they jump 0.52m/2.52m or 0.95m/3.50m. It's a checkmark, essentially.

 

[Mathman]

It's still not really easy to say what determines "very good" height and distance.

True. Like many things in figure skating judging, it is not easy to specify when you have it and when you don't.

And it shouldn't be a relative thing -- to me "superior" implies having some of the best amplitude - but since skaters aren't marked on elements relative to each other, or rather shouldn't be, then that adjective is not appropriate.

OK, but to me this is splitting semantic hairs without much actual content. I suppose technically the comparative form "superior" sneaks in a little bit of, "I am superior to you," while "very good" is a little less one-on-one confrontational (I am superior to an objective (but undefined) cut-off point). But we could just as well call it "wow!" instead of "very good," with the same problem of, when is a jump wow and when isn't it.

I don't know why this is an argument anyway -- the word "superior" is not used in the current rules, so ...?

 

[tral]

I don't know why this is an argument anyway -- the word "superior" is not used in the current rules, so ...?

That's because it is not. This poster's main talent appears to be building a wall of words with little meaning. Hey, a metaphor for CoP programs!

I think the main issue is that the judges don't use their eyes, and appear less than educated. Only those people who have watched enough skating to know what is good, difficult, etc. should be allowed to judge, and a large degree of objectivity can be brought to the sport like that, through shared understanding. But, this is unlikely to happen.

Being able to use IScope data can be useful to determine "very good height" and "very good distance" (both of which are SEPARATE aspects of jump quality, and yet are grouped into a single GOE bullet...). Unlike the ridiculous strawman about the judges remembering jump data and using it to score the skaters no matter how they jump in the current competition, past competition data CAN be used for real time judging: keep a running database of all 3As jumped in elite competitions, for instance, and that way the highest jumped 3A, the longest jumped 3A can be determined. Then agree upon a visual standard/percentage-based standard for "very good height" and "very good distance". Such a database will take a season or two to collect in a good enough quantity to be able to start implementing it over the rules, and it will keep track of jump data per competition.

This is just an idea, BTW. I am sure the oh-so-accomplished members of the ISU can come up with something if they put their heads together.

 

[CanadianSkaterGuy]

OK, but to me this is splitting semantic hairs without much actual content. I suppose technically the comparative form "superior" sneaks in a little bit of, "I am superior to you," while "very good" is a little less one-on-one confrontational (I am superior to an objective (but undefined) cut-off point). But we could just as well call it "wow!" instead of "very good," with the same problem of, when is a jump wow and when isn't it.

I don't know why this is an argument anyway -- the word "superior" is not used in the current rules, so ...?

Yeah, semantics can muddy the waters a bit. I think "superior" was used in previous IJS iterations, but I can't remember exactly. I think "very good" implies above average, but I don't take that to me a specific average benchmark height/distance. It's more like, you look at the jump and see if it had good height and distance.

It's kinda like a "well-centred spin"... the judges aren't taking a microscopic look to ensure that the tracing of the spin makes a perfect circle (a la figures) and ensure that none of the rotations deviate outside of that (99.999% of spin aren't perfectly centred if you look at the tracing). There is some leeway and the idea is, the spin *looks* well centred for the most part (a subjective call), the same way if a jump *looks* huge it is thus more likely to be awarded a GOE bullet.

 

[Mathman]

I think the main issue is that the judges don't use their eyes, and appear less than educated. Only those people who have watched enough skating to know what is good, difficult, etc. should be allowed to judge, and a large degree of objectivity can be brought to the sport like that, through shared understanding. But, this is unlikely to happen.

I have to confess that I am not nearly so harsh on the judges as all that.

I think that the judges do use their eyes, that they are well-educated on the sport, and that they have watched enough skating to know what is good, difficult, etc. I also think (happy fellow that i am ) that by and large they try their best to judge with fairness and objectivity.

And they absolutely exhibit "shared understanding" -- how else can we account for the fact that almost all of the judges' marks are clustered so closely together?

The judges don't always agree 100% (that would certainly be suspicious if it ever happened). I don't always agree with the consensus of the judges and I often agree with one judge but disagree with another. It is what it is.

 

[Shanshani]

I don’t see why the height/distance GOE bullet has to be binary at all, especially if height/distance judging can be done by machine. What makes most sense is if a certain number of possible GOE points are available for height and distance, and you get a % of them depending on how far/high you jumped. The standards could be set using historical data.

For the sake of demonstration, it could work like this. Let’s say we allot 1 GOE point available to be earned for height and distance, maybe split 0.5 for height and 0.5 for distance. Suppose we take a bunch of 3A data and we find that the biggest height is 0.7m and the biggest distance is 3.62m (which indeed was the case in the World’s SP). Let’s also say that, once we take into account lower tiers of competition, we find that the average 3A height is 0.56m and the average length is 2.62m. Let’s stipulate that your 3A has to at least be slightly above average to start earning height/distance GOE points (but of course we can argue about this—maybe the standard should be higher, maybe lower). From this, we could construct a scale where your height/distance GOE is proportional to how far your 3A is above the average, and the standard for full marks is set by the maximum height/distance among the historical data.

So, applying this to the Worlds men’s SP 3As, assuming those measurements are accurate, it would look like this:

Yuzuru Hanyu would get maximum marks for his 3A on both height and distance, as their height and distance matches the maximum height and distance in the historical data (obviously this is cheating a bit since I’m using his 3A in the historical data but this makes the math easier to demonstrate and the numbers somewhat grounded in reality, so deal with it haha). Therefore, he gets the full 1 point for height and distance.

Shoma Uno would get 0 points for height on his 3A, since his height of 0.51 is below the average height of 0.56. However, he would also receive (3.44-2.62)/(3.62-2.62)=82% of full marks for distance, as his distance of 3.44 is 82% of the way between the average (2.62) and the maximum (3.62) in the historical data. Therefore, he would earn 82%*0.5=0.41 GOE for distance.

Mikhail Kolyada, on the other hand, would receive (0.65-0.56)/(0.7-0.56)=64% of full marks on height, as his height is 64% of the way between the average and the maximum, so 0.32 points. But he would receive nothing for distance, as the distance on his 3A (2.5m) is below average. (I personally kind of disagree with this and would argue for weighing height more, but we can argue over the details, this is just demonstration.)

Nathan Chen would receive (0.58-0.56)/(0.7-0.56)=14% of the full marks on height, for 0.07 GOE, and (2.66-2.62)/(3.62-2.62)=4% of the full marks on distance, for 0.02 GOE. Therefore, he would earn 0.09 GOE in total for jumping slightly, but only slightly, higher and longer than average. (Of course, let me note that I completely made up the average numbers—the actual average for clean 3As in the Worlds SP was 0.59m height and 2.87m distance, but I put lower numbers on the theory that the men at Worlds would have bigger 3As than all of the men who can jump a 3A do on average).

Keegan Messing, on the other hand, would receive (0.64-0.56)/(0.7-0.56)=57% of full marks for height, giving him 0.29, and (3.33-2.62)/(3.62-2.62)=71% of full marks for distance, giving him 0.36, for a total of 0.65 GOE for a strongly above average but still somewhat short of the maximum jump in terms of size.

Anyway, you get the idea. Again, the details can be tweaked, but I find the general idea to be much more sensical than a binary choice of whether the jump had “very good” height and distance. Of course, all the math can be done by computer, so all this is fully automated. It also has interesting strategic consequences—in addition to doing a better job of incentivizing jumping big, which I think the current judging is very bad at, it also has the interesting side effect that if you can figure out how to jump much bigger than your opponents, you can suppress their GOE scores (presuming the historical data is continually updated, which I think it should be). For instance, Yuzuru’s 3A was a whole 5 centimeters higher than the next highest 3A, from Mikhail Kolyada. Consequently (under this example scoring regime), no other competitor was able to score more than 64% of the points available for height! I think that would add an extra dimension to the competition and really encourage bigger jumps.

 

[Shanshani]

Admittedly, it would be difficult to set standards for jumps that are rarely jumped, like 4Lo and 4F. In those cases, perhaps standards could be set using a data set pooled from all of the quads. Unfortunately, I think that might short change 4Lo jumpers a bit, since I don’t think loops tend to get as high and far as other jumps because of the mechanics of the jump, but it would be a fair compromise until a bigger data set is built, and certainly better than whatever passes for height/distance judging now.

 

[tral]

I think that the judges do use their eyes, that they are well-educated on the sport, and that they have watched enough skating to know what is good, difficult, etc.

They certainly have a strange way of showing it.

I do not believe that trained, well-educated judges, who have good understanding of the sport, and who have been selected via some proper standard would come up with some of the judging that they have.

And they absolutely exhibit "shared understanding" -- how else can we account for the fact that almost all of the judges' marks are clustered so closely together?

Ah, but see, as "Mathman" I'm sure you won't be confused when I cite "corridors" here, based off reputation of the skater in question.

 

[Mathman]

Ah, but see, as "Mathman" I'm sure you won't be confused when I cite "corridors" here, based off reputation of the skater in question.

Corridors are very interesting, but more because of psychology than mathematics, I believe.

Judges try to stay in the corridor -- meaning that they try to guess what the other judges are going to do so that they can go along -- because, well, that's human nature. I think that "reputation" is only a secondary factor, in the sense that when I ask myself, "What scores are my peers going to pressure me into giving out this time?" the best guess revolves around, "I wonder what my peers will give this famous and dominant skater compared to that nobody?"

However, I think that fans give "the corridor" a bad rap. The corridor is so wide that It is practically impossible for a judge to be so far off as to land sufficiently outside the the corridor to get in trouble with the ISU.

I would love to see the following experiment attempted: Let nine competing artificial intelligence teams design nine computers to serve as figure skating judges. I bet we would see scores all over the place, compared to their human counterparts.

 

[Sam-Skwantch]

I would love to see the following experiment attempted: Let nine competing artificial intelligence teams design nine computers to serve as figure skating judges. I bet we would see scores all over the place, compared to their human counterparts.

Lol...so true

 

[CanadianSkaterGuy]

I don’t see why the height/distance GOE bullet has to be binary at all, especially if height/distance judging can be done by machine. What makes most sense is if a certain number of possible GOE points are available for height and distance, and you get a % of them depending on how far/high you jumped. The standards could be set using historical data.

For the sake of demonstration, it could work like this. Let’s say we allot 1 GOE point available to be earned for height and distance, maybe split 0.5 for height and 0.5 for distance. Suppose we take a bunch of 3A data and we find that the biggest height is 0.7m and the biggest distance is 3.62m (which indeed was the case in the World’s SP). Let’s also say that, once we take into account lower tiers of competition, we find that the average 3A height is 0.56m and the average length is 2.62m. Let’s stipulate that your 3A has to at least be slightly above average to start earning height/distance GOE points (but of course we can argue about this—maybe the standard should be higher, maybe lower). From this, we could construct a scale where your height/distance GOE is proportional to how far your 3A is above the average, and the standard for full marks is set by the maximum height/distance among the historical data.

So, applying this to the Worlds men’s SP 3As, assuming those measurements are accurate, it would look like this:

Yuzuru Hanyu would get maximum marks for his 3A on both height and distance, as their height and distance matches the maximum height and distance in the historical data (obviously this is cheating a bit since I’m using his 3A in the historical data but this makes the math easier to demonstrate and the numbers somewhat grounded in reality, so deal with it haha). Therefore, he gets the full 1 point for height and distance.

Shoma Uno would get 0 points for height on his 3A, since his height of 0.51 is below the average height of 0.56. However, he would also receive (3.44-2.62)/(3.62-2.62)=82% of full marks for distance, as his distance of 3.44 is 82% of the way between the average (2.62) and the maximum (3.62) in the historical data. Therefore, he would earn 82%*0.5=0.41 GOE for distance.

Mikhail Kolyada, on the other hand, would receive (0.65-0.56)/(0.7-0.56)=64% of full marks on height, as his height is 64% of the way between the average and the maximum, so 0.32 points. But he would receive nothing for distance, as the distance on his 3A (2.5m) is below average. (I personally kind of disagree with this and would argue for weighing height more, but we can argue over the details, this is just demonstration.)

Nathan Chen would receive (0.58-0.56)/(0.7-0.56)=14% of the full marks on height, for 0.07 GOE, and (2.66-2.62)/(3.62-2.62)=4% of the full marks on distance, for 0.02 GOE. Therefore, he would earn 0.09 GOE in total for jumping slightly, but only slightly, higher and longer than average. (Of course, let me note that I completely made up the average numbers—the actual average for clean 3As in the Worlds SP was 0.59m height and 2.87m distance, but I put lower numbers on the theory that the men at Worlds would have bigger 3As than all of the men who can jump a 3A do on average).

Keegan Messing, on the other hand, would receive (0.64-0.56)/(0.7-0.56)=57% of full marks for height, giving him 0.29, and (3.33-2.62)/(3.62-2.62)=71% of full marks for distance, giving him 0.36, for a total of 0.65 GOE for a strongly above average but still somewhat short of the maximum jump in terms of size.

Anyway, you get the idea. Again, the details can be tweaked, but I find the general idea to be much more sensical than a binary choice of whether the jump had “very good” height and distance. Of course, all the math can be done by computer, so all this is fully automated. It also has interesting strategic consequences—in addition to doing a better job of incentivizing jumping big, which I think the current judging is very bad at, it also has the interesting side effect that if you can figure out how to jump much bigger than your opponents, you can suppress their GOE scores (presuming the historical data is continually updated, which I think it should be). For instance, Yuzuru’s 3A was a whole 5 centimeters higher than the next highest 3A, from Mikhail Kolyada. Consequently (under this example scoring regime), no other competitor was able to score more than 64% of the points available for height! I think that would add an extra dimension to the competition and really encourage bigger jumps.

Some questions:

1) Wouldn't these scores have to be tabulated AFTER every skater skates? So, you wouldn't actually know your final score until everyone skated and THEN the relative height/distances were tabulated for every skater to come up with an average -- and then bonuses were allotted? This is why it makes more sense to have a standard minimum threshold height/distance to earn the bonus, (e.g. 0.5 m and 2.5 m, and bonus is given to anyone who exceeds both - as assessed by accurate technology, which IceScope has yet to prove itself to be), rather than having to wait until everyone has skated and the computer tabulates the average height/distance of the field and determine results.


2) About the 1.0 point bonus (a rather arbitrary number). So is that 1.0 points on the 3A (11.7% of the BV of 8.5) allotted to height/distance? What happens for a 2A? Is the amplitude bonus also 11.7% of the BV -- which is 3.3 (i.e. up to 0.38 points amplitude bonus can be given on a 2A - 0.19 for height, 0.19 for distance)... or does a double axel get up to 1.0 point on amplitude bonus as well? I know certain people love to count every hundredth of a point, but are we really implementing all this technology and effort over less than a point of difference? And, speaking of 3A/2A... this brings me to my next point:

3) what do you do when assessing relative scores for those who do different layouts/elements? Chen/Hanyu was the only skater in the Worlds FS to successfully do a 4F/4L... does that mean he gets full bonus points on height/distance for that element no matter how big or small he executed it? Does Samohin get the maximum amplitude bonus on his 2S since his pop was "higher/farther" than Kolyada's/Tanaka's 2S? If in the ladies, (hypothetically) everyone popped a certain jump and Miyahara was the only one to land it, does she earn the maximum amplitude bonus?

4) Sure this could encourage people to have bigger jumps, but that could get dangerous -- what if skaters throw themselves into jumping passes in an attempt to get that bonus? And say they fell (or erred, like a touchdown or stepout) but still had the greatest height/distance - do they still earn the bonus?

5) Do < and << jumps get any amplitude bonus? ... e.g. Uno was the only skater to have a 4F<, so does he earn 11.7% of the BV of 8.25 - or 0.9625 points because he was the only one to do that?

6) If we're talking about record scores and personal bests -- if the skaters with the most amplitude are absent from your competition, wouldn't that skew your scores higher than if those skaters were present? Say Skater A with decent jumps just skated cleanly... but Skater B who typically has the biggest jumps in the field shows up. If skater B nails every jump, then they lower Skater A's amplitude bonus and skater A's overall score is lower because of how well skater B skated -- if skater B has a bad day and pops every jump then Skater A has a higher amplitude bonus and skater A's overall score is higher thanks to how poorly skater B skated.

The IJS was fundamentally designed to start treating skaters more like absolutes (rather than relatives), especially on the technical side. So your score is what you earned and not contingent on whether everyone else performed well or poorly. Your technical score should be formulated by what YOU put out there.. not based on what OTHERS put out there. (The same is supposed to go for PCS -- if everyone else performs terribly, and a skater that typically gets 7's goes clean, they shouldn't all of a sudden get 9's just because everyone else did poorly)..

 

[tral]

I would love to see the following experiment attempted: Let nine competing artificial intelligence teams design nine computers to serve as figure skating judges. I bet we would see scores all over the place, compared to their human counterparts.

If they let the computers train on the data that is already available to them, then the 9 would still be judging as the judges currently do.

 

[flanker]

Exactly. And the result will be a computer who judges just like humans do.

But we already have judges that judge like humans do. :yes:

I think this is the golden line of the discussion.

For me, the idea is interresting about as much as if there should be computers deciding about being "guilty or not guilty". No judges, no jury, no advocates, just computers working on the base of what other people put into them.

Axelcam is nice for the TV spectators, but not sophisticated or accurate enough yet.

I am not opposed to computer assisted evaluation and scoring, but the system has to be 100% accurate and reliable. We don't want technical delays or the inevitable breakdown when judges have to do it all themselves again and objectivity suffers.
Because, inevitably, human judges will get lazy and less observant with machine help.

Before I had a cell phone I perfectly remembered about 40 phone numbers I needed. Within a few months I put that into the phone memory, I completely forgot all of them. Completely. And that's just one example. In my view, human mind should not be "assisted" too much.

Should technology replace human decisions, because humans are deeply subjective, irrational systems who think rationally only from time to time? But how would we know the creators of the assisting software we rational during the process of crerating it? We have driving assistants, autopilots and uber taxi colliding with pedestrians because human mind relying on technology just couldn't react properly within seconds. And I don't think that's something different. No.

But there is also one more thing. With such technology I'm deeply afraid that skaters would start to learn not how to be characters, having individual styles, programs etc. They would be forced to learn how to comply the system, how to fit into the prescribed algorythms. There are sports when e.g. height is the norm, in basketball you have to be high ehnough to get the advantage over the defence and score. BUt in figure skating you can be tall european man or small japanese woman and both your jumps can be evaluated as "good" or "unexpected".

I think I prefer some human judging errors before computerized accuracy.