Hypothetical discussion : How would you reduce national bias in judging figure skating

[snowed]

Maybe the easiest way to do that would be to have buttons that the jump judges could click to indicate whenever they take a GOE reduction for what they saw as <<, <, or e errors.

Then the "voting" would be through the computer: if two or all three of the judges press the relevant buttons, the computer would either alert the tech panel to review the other video angle, or if you really think the judges should override the tech panel it could just apply the lower base value based on the judges' pressing the buttons.

Yes this would work! if 2 judges alert the tech panel of under or edge, they should add it to their voting.

 

[snowed]

Oh, and there would be other questions about how this system could be used for lesser events. For domestic events, national bias is not relevant. For small internationals, nine judges per event to be able to divide the panels like this, and an additional camera, could create a financial burden for the host clubs/federations.
So should skating get more expensive at the grass roots level (as it already has with the introduction if IJS vs. 6.0)? Or should lower events use a different system than the elite events?

Domestic events and even small international events judging should stay just as it is now. I saw domestic events with 4 judges for sure, I don't remember if I saw 3 judges, the point is with less than 9 judges there cannot be 3 panels of 3 judges. Maybe for competition with less than 9 judges they can add the judges vote for under and edges, that's it.

 

[Andrea82]

Domestic events and even small international events judging should stay just as it is now. I saw domestic events with 4 judges for sure, I don't remember if I saw 3 judges,

I've seen domestic competitions with only 3 judges in ice dance. Nationals in smaller countries don't have sometimes have enough Dance officials.
For instance, this year Estonian Nationals in Ice Dance had only 3 judges. One of the judges was also the Referee. There wasn't a technical controller, only the 2 specialists.
Austrian nationals had only only 1 technical specialist + the controller. So another tech panel of 2.
Danish nationals had only 2 judges and one of the judges was also the referee.

 

[Mathman]

At the Belgian national championships there were quite a few years, under both 6.0 and IJS, when Kevin van der Perren was the only entry in senior men's singles.

This year in ladies there were two. Nina Pinzarrone and Jade Hovine (Loenna Henrickx not participating). Pinzarrone got pretty good marks but I think that they were about the same as she was getting internationally on the Grand Prix. I always wondered how many judges and tech specialists are required when the field of competitors is so small.

f memory serves, back in the late 1990s when the ISU was undertaking to seize control of pro-ams and cheesefests a rule was was put in place that there had to be 5 judges representing at least three different countries. Typically there would be three local judges with the panel fleshed out with two invited guest judges who were often living/coaching in the host country anyway so travel wasn't a problem.

How is it done in the Japan Open?

 

[Mathman]

@snowed, Ok, I think I get the main features of your proposal now. Thank you for an interesting contribution. My opinion is that for edge calls and undrrotations, it would be just as good to leave the tech panel as it is, but perhaps seat the two tachnical specialsists on opposite sides of the rink, to get two perspectives instead of 1. They would have to be able to communicate with each other in real time, but that's what modern technology like i-phones are for.

The idea of having three judges dedicated to "voting" yes or no on each of the bullet points for jumps, etc. and then let the computer take it from there, that's actually pretty cool IMHO. Even without the judges talking to each other the computer could just record that it got 2 yesses and 1 no, or whatever, and add up the number of bullet points on which the yesses have the majority to determine the GOE for each element. (The responsibilities of the technical controller might have to be redefined somewhat.)

This might cut down on the tendency of judges to give out too high GOEs for quads just becasue of their difficulty, and it would also decrease the temptation of just saying, "Wow, what a great jump! That's a +5", without avtually following the rules about bullet pojnts.

 

[snowed]

My opinion is that for edge calls and undrrotations, it would be just as good to leave the tech panel as it is, but perhaps seat the two tachnical specialsists on opposite sides of the rink, to get two perspectives instead of 1. They would have to be able to communicate with each other in real time, but that's what modern technology like i-phones are for.

Yes, a split tech panel would work too. I think they have to use something else than phones so they can be recorder, plus they could get a random phone call. They should have wired headset in my opinion. Or they should just add a second video from a different angle, whatever is easier.
But I really thing ISU should do something ASAP about tech panel being to see more angles so they can call correctly edges and underrotations ...

 

[skatingguy]

How is it done in the Japan Open?

Japan Open had a panel of 5 judges from 5 different countries, and used the same panel and technical panel for both men & women.

Japan Open 2023

 

[CanadianSkaterGuy]

I think that there is a tension between instances of national bias and the perception of widespread skulduggery that may be harbored by fans, either casual or intense. It would be natural for the ISU to want to keep control of this in their own hands. While accepting their responsibility to identifiy instances of judging bias and to take whatever actions they deem appropriate, nevertheless they want to avoid taking a public relations hit that adversely affects the reputation of the whole sport.

I can certainly see why the ISU would not want a commentator to point out to the TV audience, "Look at those ridiculously high marks that the judge from Slovenia gave to his own skaters. What a crooked sport. Why are you watching this travesty anyway?"

For sure, but most viewers aren’t looking at the protocols in real time.

Indeed commentators still shade the judges/tech panel “that jump looked clean but it seems the technical panel decided to give it an underrotation call”; “that PCS I can’t understand/is a bit of a gift/etc”.

It’s important to not diminish the credibility of the sport for the audience but you can still hold officials to task.

In a protocol a judge knows they will have everyone see the scores they give but they won’t get called out in real time by a commentator because the score is an aggregate.

 

[CanadianSkaterGuy]

Sure but with skating scores available one click away, we get that easily, and as you say, when looking at protocols, sometimes I see something fishy and I am able to guess the nationality of the judge, and verify it quickly... So I don't think it would be enough.

True but to attach a name or nationality to a score might make a judge think twice about biased scoring rather than just “judge 4”. It’s the age old issue of the ISU protecting judges emboldening them to dole out shady scores without accountability or ownership of their marks.

 

[gkelly]

True but to attach a name or nationality to a score might make a judge think twice about biased scoring rather than just “judge 4”. It’s the age old issue of the ISU protecting judges emboldening them to dole out shady scores without accountability or ownership of their marks.

Under 6.0 every judge gave a total of 2 scores per skater. It was easy to show all the scores judge by judge on the arena scoreboard, and on the TV screen, during the kiss-and-cry, with associated the two rows of scores with another row of judges' flags, and often the total of the two scores and the interim ordinals.

But those ordinals were very likely to change as subsequent skaters skated. Only real diehard spectators would bother to keep track after each skater so they could predict whether the segment standings were likely to change as the event went on. Let alone the factored placements for short program and/or figures combined with free skate. The casual fans didn't follow that closely.

All that information was available in the printed programs available (for purchase) after the end of the event.

Domestically, there were only judge numbers above the columns for each skater, since they were all from the same country.

If you wanted to know the names of the judges, or the names of the referee, assistant referee, and substitute judge (whose scores were not published), you needed to look at the cover page that listed the officials. If you wanted to see which individual judge was judge 4 who gave that out-of-line mark to skater Q, you needed to cross reference to the cover sheet.

With IJS, once scrambling the judges columns in an attempt at anonymity was no longer an issue, it would be possible to include the three-letter country abbreviation next or instead of the judge number in the protocols. There isn't room for more than that, especially now that there are up to 15 scores for each skater from each judge (more a few years ago before the number of elements and number of components were reduced).

Anonymity aside, it's just not practical to show individual judges' IJS scores in real time during the kiss-and-cry. They don't even show separate scores for each component.

Anyone who wants to study the protocols to see exactly how each element and component was scored and which judge did what will h ave to wait a few minutes and then find the detailed protocols, which contain a lot more information than what can be shown during the brief K&C time.

Casual fans don't follow that closely. Most probably don't even know where to find the protocols.

Fans who care enough to go look up the detailed scores know where to find them, and they also know where to find the names of the judges on a separate page, same place they were under 6.0.

Yes, it would be possible to identify the countries of the judges for each column now, for those who are dedicated enough to peruse the detailed protocols but not willing to cross reference with the cover sheet.

But there really isn't a way to share the judge nationalities with the casual fans who only pay attention to the scores show during the K&C (and to the scoring box during the program for broadcasts/streams that include that).

 

[SmileHappy34]

Have the tech panel not jUdge their country's skaters nor the judges. Have the backup/reserve.

For ex if a tech is us have the assist tech who is bel . A judge is USA have one of the backup or reserve.

 

[NaVi]

I came across this at the MIT Tech Review How AI is changing gymnastics judging

 

[gkelly]

I came across this at the MIT Tech Review How AI is changing gymnastics judging

Thanks for posting!

So how could something like this work for figure skating?

Today’s JSS system no longer relies on lasers but uses four to eight high-definition cameras positioned at each apparatus to capture a three-dimensional view of a gymnast’s performance, analyzing positions of the joints and then comparing those positions with the standards for each element in the Code of Points—almost in real time.

How many cameras would be needed to capture all areas of the rink from enough angles? Each gymnastic apparatus is more fixed in location -- but there are multiple apparatuses, as opposed to just one, much larger, ice surface.

So it shouldn't need more cameras than a whole gymnastics competition.

Are the cameras fixed, or are there camera operators who follow the movement of the athletes? Especially in floor exercise, which covers the most land area, so to speak (assuming that the running into the vault is not a judge part of the element)?

As always, my concern about adding extra cameras in figure skating is that not all rinks are built to accommodate cameras on all sides. So a system designed for large arenas might not be practical in local rinks -- even if the competitions held in those smaller venues are used to determine who qualifies to compete in the big events.

For elements like spins and lifts, AI might be able to determine whether a skater achieved necessary positions and how soon/how long. It might have trouble with more creative positions.

Cameras would also need to focus on the blades, which are relevant to some aspects of spinning and to the man's skating during a lift.
The most important aspects of skating jumps are all about the blades. Variations in body position (positive or negative) are either not relevant or could be evaluated by humans awarding GOE if some subjectivity would still be part of the scoring.

And then there's the actual skating, in step sequences (and choreo sequences, and other ice dance elements, not to mention synchro elements) and also between elements.

The system had to be taught the difference between an element and an interval between elements, as well as how much, or how little, movement constitutes “stopping.”

Unlike gymnastics, diving, etc., what's being evaluated in a figure skating program is not just elements. The skating between elements or within the extended skating elements listed above is not just intervals between the parts that are scored. It's arguably the most important, certainly the most fundamental, aspect of the performance being judged.

So if AI were to be used for judging skating (as opposed to just jumps, spins, etc.), it would need different equipment that could keep track of what the blades are doing at all times, and very different programming to be able to evaluate all that skating that takes place between the tricks.

Some of what is important about the actual skating should be easily measurable, but other aspects are more complex and perhaps inherently more subjective than the kinds of questions relevant to execution of the tricks.

And then the performance aspect his still relevant to both sports in ways that this article acknowledges gymnastics AI does not capture.

 

[Skating91]

Let's just start with one extra camera for the judges to look at (cameras are extremely inexpensive nowadays), and maybe down the track get into elaborate, expensive AI setups. If one more camera does not fix the obvious errors, then add one more camera.

I pick up stuff the experts miss watching Youtube, it's really not that complicated to make correct decisions.

 

[skatingguy]

Thanks for posting!

So how could something like this work for figure skating?



How many cameras would be needed to capture all areas of the rink from enough angles? Each gymnastic apparatus is more fixed in location -- but there are multiple apparatuses, as opposed to just one, much larger, ice surface.

So it shouldn't need more cameras than a whole gymnastics competition.

Are the cameras fixed, or are there camera operators who follow the movement of the athletes? Especially in floor exercise, which covers the most land area, so to speak (assuming that the running into the vault is not a judge part of the element)?

As always, my concern about adding extra cameras in figure skating is that not all rinks are built to accommodate cameras on all sides. So a system designed for large arenas might not be practical in local rinks -- even if the competitions held in those smaller venues are used to determine who qualifies to compete in the big events.

For elements like spins and lifts, AI might be able to determine whether a skater achieved necessary positions and how soon/how long. It might have trouble with more creative positions.

Cameras would also need to focus on the blades, which are relevant to some aspects of spinning and to the man's skating during a lift.
The most important aspects of skating jumps are all about the blades. Variations in body position (positive or negative) are either not relevant or could be evaluated by humans awarding GOE if some subjectivity would still be part of the scoring.

And then there's the actual skating, in step sequences (and choreo sequences, and other ice dance elements, not to mention synchro elements) and also between elements.



Unlike gymnastics, diving, etc., what's being evaluated in a figure skating program is not just elements. The skating between elements or within the extended skating elements listed above is not just intervals between the parts that are scored. It's arguably the most important, certainly the most fundamental, aspect of the performance being judged.

So if AI were to be used for judging skating (as opposed to just jumps, spins, etc.), it would need different equipment that could keep track of what the blades are doing at all times, and very different programming to be able to evaluate all that skating that takes place between the tricks.

Some of what is important about the actual skating should be easily measurable, but other aspects are more complex and perhaps inherently more subjective than the kinds of questions relevant to execution of the tricks.

And then the performance aspect his still relevant to both sports in ways that this article acknowledges gymnastics AI does not capture.

Some information on how tennis is doing it.

“In total here, we have 204 cameras ... tracking the ball and the player across all 17 courts at the U.S. Open, ” said Ben Figueiredo, the director of tennis at Hawk-Eye Innovations. “We have 12 tracking cameras around all the courts. And then we also have six foot-fault cameras.”

Hawk-Eye says the equipment for one court costs nearly $100,000 and takes about three days to set up. The cameras track the ball at 340 frames per second and transfer images immediately to the Hawk-Nest, where an “in” or “out” call can be made.

How Sony's Hawk-Eye electronic line-calling system transformed the U.S. Open

CNBC got a behind-the-scenes look at Sony's Hawk-Eye line-calling system to understand how the tech works in tennis and other major sports.

www.cnbc.com

 

[4everchan]

I can give away my webcam if anyone needs it I used it a lot during lockdown but I am I really sick of Zoom meetings

 

[Skating91]

I will preface this by saying I really enjoy watching Isabeau and want her to do well, but I've never seen such GOE on jumps like this (jumps are a huge weakness for her obviously) as in her short program. If you read how they are supposed to award GOE on jumps she does not tick much of the criteria. Then the PCS for Isabeau and Amber (I also like to see her do well and hope she can stabilise her 3A) it just hurts the credibility of the officials and the sport in general.

In jumps the bullets for positive GOEs are:
1. very good height and distance
2. good take-off and landing
3. effortless throughout (including rhythm in a combination or sequence)
4. steps into a jump, unexpected or creative entry
5. very good body position from take-off to landing
6. element matches the music
1, 2 and 3 are mandatory ones for +4 and +5 GOE.

The reductions for errors are:
Fall: -5
Landing on two feet, stepping out: -3 to -4
Two three-turns between jumps in a combination: -2 to -3
Wrong edge (“e”): -3 to -4
Unclear edge (“!”): -1 to -3
Downgrade (“<<”): -3 to -4
Underrotation (“<”): -2 to -3
Poor speed, height or air position: -1 to -3
Touch-down with both hands: -2 to -3
Touch-down with one hand or free foot: -1 to -2
Loss of flow or rhythm in combination or sequence: -1 to -2
Weak landing: -1 to -3
Poor take-off: -2 to -3
Long preparation (telegraphing) -2 to -3

 

[4everchan]

Just a pointer... this thread is about National Bias in international competition.
If we start looking at National bias at national competitions... there are only a few countries who seem to be relatively fair... some countries like the USA and Russia have very generous panels... some countries like Japan (at times) and Canada (for singles especially) have very strict panels...

 

[Icey]

Perhaps the judging would be more balanced if the panel was a 50/50 mixture of foreign and American judges.

 

[4everchan]

Perhaps the judging would be more balanced if the panel was a 50/50 mixture of foreign and American judges.

For US nationals, that could work. For international competitions, that would just really help even more the American skaters don't you think ?