Scoring bias at the national level

[snowed]

I do think that it is a plus not a minus that figure skating judges show a range of variation in interpretting the scoring rules.

Me too... I want for judges to judge honestly, how they see it, in their own experience and culture and interpretation of the publish rules, and not be afraid that they'll be accused of bias or worse. If they are afraid they will go to corridor judging...

 

[4everchan]

I watched that performance (on TV) and the twist looked perfectly fine, but then the slow motion review was from a different angle and you could see the problem with the catch. So I think is very possible that from a certain angle (like the one we saw on tv), a judge will not see the catch well and the other judges will see it better from different angles

definitely... and as their twist is usually their best element...would a judge necessarily review it if they thought it was good in real time ?

How does this work ? Do judges look at everything or just a few elements ? I am assuming it's the latter.

 

[gkelly]

definitely... and as their twist is usually their best element...would a judge necessarily review it if they thought it was good in real time ?

How does this work ? Do judges look at everything or just a few elements ? I am assuming it's the latter.

I know with the tech panel, if any one of the three members has a question about an element during the program they'll say "Review" as soon as the TS1 calls the element, and then the three of them will review it together after the program. If no one calls "Review," they won't review it.

For the judges, I think they have the option to rewatch any of the elements they want to check on, if they had a question when they saw it during the program. They probably wouldn't have time to check all the elements, because they also need to enter their component scores during that time.

 

[gkelly]

I think that the dictionary of the ISU is too restrictive and does not reflect the richness of the history and traditions of the discipline.


All is grist that comes to our mill. Give me a Russian split jump and a delayed Axel.

But instead we have triple flip, 4.90. Triple flip, 5.30. Delayed Axel, 1.10. Split jump 0.00. Delayed Axels and Split jumps have been written out of the dictionary. I think that this impoverishes the sport and prevents it from being all that it could be.

The split jump is written out of the scale of values. The delayed axel (or full-revolution split-flip) are included in the scale at a low base value so even though the air position variation should earn them high GOE, that's still not enough points to be worth giving up a rotational jump.

However, these elements are permitted and rewarded in the Choreographic Sequence, which "Consists of any kind of movements like steps, turns, spirals, arabesques, spread eagles, Ina Bauers, hydroblading, any jumps with a maximum of two revolutions, spins, etc."

And the half-revolution split jump can be used as a transitional move anywhere in the program. We still see it approximately as often as any other eye-catching transitional move.

In that sense they're still part of the vocabulary of available moves to include in the program and be rewarded the choreo sequence GOE and in the program components -- especially Composition now that there's no more Transitions score.

How could such moves be "written in" to the scoring better, so skaters would have more incentive to include them?

One approach would be to take every individual move that has ever been done and named in the past hundred years and assign it a base value, with guidelines for positive and negative GOE from the judges, and perhaps with level features to be assigned by the tech panel.

That approach would leave out any new and original move that a skater might come up with that doesn't quite fit into the definitions of any of the specified individual moves.

Or there could be categories of moves (e.g., "low-revolution jumps with extended air positions" or "spiral" or "hydroblading") with a general base value for the category and then GOE guidelines and maybe levels for any element in that category, whether it's a familiar one with a familiar name or a new variation that didn't have a name yet.

Or there could be a new kind of leveled sequence element that takes various kinds of moves that are currently relegated to the unleveled choreo sequence. E.g., a "low-revolution jump sequence" or a "field moves sequence" -- with features for the sequence that these known or novel moves and the ways they're combined within the sequence could fill to earn a high level and high base value for that sequence element.

Or the Transitions component could be reinstated.

And/or there could be more latitude given for judges to score the current unleveled choreo sequence based on all the current bullet points and also on variety and difficulty of the moves that are included.

The question is whether it's better to standardize and quantify these kinds of moves, so that the Scale of Values and the Tech Panel handbook would require dozens or hundreds of pages to specify every way skaters can earn points by including them and exactly how many points each variation or combination would be worth.

Or, as now, to reward them in PCS and leave it up to the skaters to decide how athletically adventurous and creative they want to get with their transitions and choreo sequence, and up to the judges how to reward them within the current handful of relevant criteria.

 

[4everchan]

I know with the tech panel, if any one of the three members has a question about an element during the program they'll say "Review" as soon as the TS1 calls the element, and then the three of them will review it together after the program. If no one calls "Review," they won't review it.

For the judges, I think they have the option to rewatch any of the elements they want to check on, if they had a question when they saw it during the program. They probably wouldn't have time to check all the elements, because they also need to enter their component scores during that time.

So... in that sense, let's say I am judging the event... I didn't see the crash on the twist, just that it wasn't as high as usual. I am Canadian... so I have seen them do stellar twists forever... so I opt to review the jumps instead which truly were problematic and perhaps the death spiral because well, how low did she get in that awkward position and was it really effortless etc.

So, I leave the twist at + 2 and call it a day.

Does that make me biased ?

 

[snowed]

For the judges, I think they have the option to rewatch any of the elements they want to check on, if they had a question when they saw it during the program. They probably wouldn't have time to check all the elements, because they also need to enter their component scores during that time.

Wasn't there a thread "ask Ted questions to cover in his competition commentary"? Maybe he can find out how judging process goes...

 

[snowed]

All of these judges have been trained on the same rules, but come to sometimes very vastly different conclusions. Should this really be the case?

I would think it goes like in judges judging the law. They have the law and it says this crime gets 5 to 10 years prison. The judge decide how many years depending on the circumstances. And different judges would give different punishment based on their own understanding of the law and the circumstances

 

[snowed]

That can be seen in the Men's FS - Take a look at Boyang Jin's protocol, his scores had a spread of 147.63 (USA) to 180.27 (NED), so that both the judging of the Mexican judge (158.99) and the Italian judge (177.56) were counted for most elements & in the PCS, despite still having a big deviation from the actual TSS. In this case, it did not make much difference as there were the same number of outliers in either direction, but that's more down to luck than anything.

My thinking is that if a judge has his/her own system/ standard and it applies it uniformly to all competitors, it will even out in the end. IJS is not about judges "placing" the skaters in order. But through their marks they determine the placement. So if judge 1 gives 7 to skater A and 8 to skater B, and judge 2 give 8 to skater A and 10 to skater B, the placement is the same. And I see in this forum many time being said that "I don't agree with the marks but I agree with the placement".

 

[Magill]

I think that the dictionary of the ISU is too restrictive and does not reflect the richness of the history and traditions of the discipline.


All is grist that comes to our mill. Give me a Russian split jump and a delayed Axel.

But instead we have triple flip, 4.90. Triple flip, 5.30. Delayed Axel, 1.10. Split jump 0.00. Delayed Axels and Split jumps have been written out of the dictionary. I think that this impoverishes the sport and prevents it from being all that it could be.

Here I agree completely and you have my vote on bringing such elements back to the table and giving them a proper point value
What I was opposing was singling out and rewarding one as opposed to all the others.

 

[Mathman]

How could such moves be "written in" to the scoring better, so skaters would have more incentive to include them?

I think that such a task would be impossible. I don't really know what I am grousing about. I think the scoring system and judging procedures are satisfactory as they are.

And yet... something lost, something gained.

I just watched Mone Chiba’s Four Continents performance on network TV. I was enthralled and delighted from beginning to end. I don’t know what jumps she did or how many points she got for them. Each element seemed effortless inevitable, yet at the same time beyond expectation.

Here is the trouble with “analysis,” by which we mean breaking something down into its fundamental atoms. This is such a powerful weapon that it destroys the thing being analyzed.

I like Ming vases. A physical chemist puts one under an electron microscope and tells me what the bonding strength is between molecules of silicon dioxide and kaolinite particles. Very interesting. But where’s the vase?

 

[Jumping_Bean]

I would think it goes like in judges judging the law. They have the law and it says this crime gets 5 to 10 years prison. The judge decide how many years depending on the circumstances. And different judges would give different punishment based on their own understanding of the law and the circumstances

I don't know if comparing a system that's known for being heavily biased and corrupt to another system that's known to be heavily biased and corrupt and basically saying "Hey, our system is all good because this is also what this other system does." is really what we want to be doing. I am not trying to get overly political here, but no, the justice system is not one you want to model yourself on when it comes to fairness. Even if in figure skating the lives of people aren't on the line.
I am not saying a bit of variation isn't normal in either system, of course it is, everyone is human, but big variations in either system are alarming. If one judge sentences someone to 5 years in prison and another judge absolves them of all guilt, somebody messed up. Either one of the judges (or maybe even both) doesn't understand the rules (incompetence) or is biased (subconsciously or on purpose), there's no way around it.

My thinking is that if a judge has his/her own system/ standard and it applies it uniformly to all competitors, it will even out in the end. IJS is not about judges "placing" the skaters in order. But through their marks they determine the placement. So if judge 1 gives 7 to skater A and 8 to skater B, and judge 2 give 8 to skater A and 10 to skater B, the placement is the same. And I see in this forum many time being said that "I don't agree with the marks but I agree with the placement".

Except for that requiring about an equal amount of judges with an equal amount of bias in either direction (in favour and against). Because otherwise, suddenly the rankings are actually not the "right" ones anymore.

And even though people like to act as if scores don't matter at all in this sport, only placements do, that's not really true either, is it? Both the JGP and GP series use the scores as tie-breakers when the placement points are the same, so even if the placements were right, if you skated two more toughly scored events, too bad for you.
How generous the technical elements score is (both tech panel calls and judges' GOEs) counts if you want to go to Worlds or Euros.
Your Season's best score counts if you want a chance at getting a GP spot next season, particularly if you are from a country with a lot of skaters already high in the SB and WS lists, and you are unlikely to go to Worlds. In women's, for example, the current score difference between 24th (guaranteed GP spot) and 25th is just 0.11, and between 75th (consideration for GP spots) and 76th (no chance at all of GPs) is 0.28! When scores are clustered very closely together like this (also in competitions), each outlier in judging counts.

 

[Mathman]

^ Still… judges make judgments. That’s their job.

And there is also the question of how do we get there from here. Suppose I issue a solemn proclamation that from now on all judges must be competent and fair-minded. All judges must be knowledgeable in the law, respectful of the precedents of case law, and incorruptible. Have I solved the problem?

What we can do is try to make tomorrow a little better than today. But it's a long, hard struggle, in my opinion.

 

[Jumping_Bean]

^ Still… judges make judgments. That’s their job.

And there is also the question of how do we get there from here. Suppose I issue a solemn proclamation that from now on all judges must be competent and fair-minded. All judges must be knowledgeable in the law, respectful of the precedents of case law, and incorruptible. Have I solved the problem?

What we can do is try to make tomorrow a little better than today. But it's a long, hard struggle, in my opinion.

Not talking about instances of weird scoring, ignoring biases because they align with our own and putting everything down to "Well that's just the way it is and has always been, so it's fine" doesn't lead to a better tomorrow though, conversations like these are incredibly important to get to a better point in the future. Think of how the IJS started out, a full-blown scandal was necessary for any changes to be made away from the 6.0 system!

Just to refer back to the case that started this thread, the technical committee could have chosen to ignore Mr Williams' clear scoring bias, but they didn't. While he didn't suffer from any actual consequences, future transgressions will be seen much more critically because there's a paper trail now.

 

[Jumping_Bean]

I think that such a task would be impossible. I don't really know what I am grousing about. I think the scoring system and judging procedures are satisfactory as they are.

And yet... something lost, something gained.

I just watched Mone Chiba’s Four Continents performance on network TV. I was enthralled and delighted from beginning to end. I don’t know what jumps she did or how many points she got for them. Each element seemed effortless inevitable, yet at the same time beyond expectation.

Here is the trouble with “analysis,” by which we mean breaking something down into its fundamental atoms. This is such a powerful weapon that it destroys the thing being analyzed.

I like Ming vases. A physical chemist puts one under an electron microscope and tells me what the bonding strength is between molecules of silicon dioxide and kaolinite particles. Very interesting. But where’s the vase?

I want to offer a different perspective - There's beauty in the details (and their analysis) too. This does not mean that you are wrong in your opinion, it's just a different worldview.

Maybe it's my personal experience as a biologist and as someone who has studied ballet for many years, where each detail is broken down and perfected until a complete picture emerges, but I cannot just separate the bigger picture from many, many small details it is made up of. A plant is of course beautiful, but so are its cells under the microscope. And of course, the Nutcracker pas de deux is beautiful, but so are a ballet dancer's hands, posture and arches.
Beauty can be found in so many aspects of life, and I want to allow myself to enjoy as many of them as I can - Be it a textbook jump, a complete performance, beautiful posture and hands, or extraordinary expressiveness. And sometimes, if you're lucky, you even get the privilege of seeing all aspects that can make skating so beautiful and unique come together.

 

[el henry]

Judges are human beings who are judging an event. They will will vary, and sometimes vary greatly, in assigning numbers to different elements of an event. Saying that numbers from a single comp and for a single skater is not evidence of discrimination does NOT mean that I am

1. saying it's the way it's always been

2. saying that I like judges who judge the way I would

3. accepting corruption or discrimination

Others can feel differently, and that's the beauty of skating. But attribution of motivation based on a single instance doesn't work when one doesn't really know the motivation. Either for judges or for posters.

 

[Mathman]

I want to offer a different perspective - There's beauty in the details (and their analysis) too.

Beauty can be found in so many aspects of life, and I want to allow myself to enjoy as many of them as I can - Be it a textbook jump, a complete performance, beautiful posture and hands, or extraordinary expressiveness. And sometimes, if you're lucky, you even get the privilege of seeing all aspects that can make skating so beautiful and unique come together.

I wish I'd said that.

 

[Jumping_Bean]

Judges are human beings who are judging an event. They will will vary, and sometimes vary greatly, in assigning numbers to different elements of an event. Saying that numbers from a single comp and for a single skater is not evidence of discrimination does NOT mean that I am

1. saying it's the way it's always been

2. saying that I like judges who judge the way I would

3. accepting corruption or discrimination

Others can feel differently, and that's the beauty of skating. But attribution of motivation based on a single instance doesn't work when one doesn't really know the motivation. Either for judges or for posters.

Bias, even unconscious and limited to a small number of people and incidents, is still bias. Bias is also not the same as discrimination, though discrimination is a form of bias.

Doug Williams did not have a history of significant overscoring US skaters either, but there were definitely some iffy moments before Worlds. They just were never deemed significant enough to call out, it seems. Maybe, if his scoring had been discussed earlier, it would not have gone as far as it did.

I must also say, that it is very confusing how talking about certain judges or certain countries seems to cause particularly strong negative and protective reactions in this forum, while other judges and referees can be talked about very differently. Everyone just accepts the opinion that Molina has a negative bias against American and Canadian teams as a referee, despite the bias of referees being even harder to determine than that of judges, as there's no direct comparison. If I had talked about for example the Georgian judge in Ice Dance at Euros (Yulia Levshunova, who didn't overscore D/S but instead horrendously (imo) underscored their direct competitors for 5th in the Free - D/LM, T/V, T/T, O/P), I am not sure I would have gotten the same pushback. Why?

 

[gkelly]

Bias, even unconscious and limited to a small number of people and incidents, is still bias. Bias is also not the same as discrimination, though discrimination is a form of bias.

Nevertheless, unconscious bias/preference is unavoidable when relying on human judgment.

Or, for that matter, on AI designed and trained by humans.

 

[Jumping_Bean]

Nevertheless, unconscious bias/preference is unavoidable when relying on human judgment.

Or, for that matter, on AI designed and trained by humans.

Of course, it's unavoidable, but that doesn't mean we shouldn't be striving to minimise the degree of bias, or else we can just give up on the scoring system as a whole and just drop the pretense of this being a sport.

There's a normal degree of variance, and an abnormal one. If we don't allow ourselves to discuss where those borders lie and if we shy away from becoming conscious of our own biases and those of others, there can be no progress in the direction of a fairer sport, as fair as it can be. I guess that's fine, but then I also don't want to see any more complaining about unfair judging or about why figure skating is so unpopular.

 

[snowed]

I guess that's fine, but then I also don't want to see any more complaining about unfair judging or about why figure skating is so unpopular.

I'm newish on the forum but in my opinion, a fair amount of complaining comes from biased fans or fans that don't know or ignore the rules on purpose....
There was a thread about improving judging and it went very quickly towards AI judging.
I'm not saying that skating judging is perfect but, personally, I don't really see how to practically improve it (except to add a second camera for tech reviews for edges and underestimation).
The next problem I see is that judging is difficult to understand, ISU should find ways to explain the rules to the fans.
Us (or judges) talking about differences in judges scores without analyzing a video of the program and following the rules, doesn't give much information.