Why can't judges just judge what they see?

Mathman said:

Saying something loud and saying it over and over doesn't make it true. Your assertions are quite clearly not plain to see. If they were "plain to see," there would not be different points of view on these threads.

Click to expand...

I think one point BoP mentioned that (I think) we all agree with is that scoring of elements, both rotation/edge calls and GOE scores, has almost as much to do with who is skating as it is how well the element is performed. Particularly for clean elements, I think skaters like Alina and Nathan tend to be a bit overscored compared to some of their rivals. I don't subscribe to the "Russia is evil" school of thought, but I think reputation scoring is alive and well, despite COP's efforts to break down the scoring to each specific element and PCS category.

drivingmissdaisy said:

I think one point BoP mentioned that (I think) we all agree with is ...

Click to expand...

I don't think so. I don't think we all agree on under-rotation, GOE or anything else. Certainly we do not all agree on what the GOE ought to be for a particular jump, whether by Alina Zagitova, Nathan Chen, or whoever.

The point I was trying to make, however clumsily, is this. To say that something is true is not the same as to assert that it is obviously and self-evidently true. I just get a bad vibe from claims like, "This is true and anyone who can't see it is a fool." It is quite possible (regardless of the truth or falsity of the assertion) that I can't see it, yet I am not a fool.

Politicians do this all the time: "You know as well as I do that the Fed ought to raise the prime lending rate." No, I do not know this as well as you do. Stop saying that I do.

Mathman said:

If they were "plain to see," there would not be different points of view on these threads.

Click to expand...

I should instead say "plain to see, if you analyze the protocols and judging trends". Much work has been put in to point it out over the years. Some people choose to keep wool over their eyes and simply don't respond when analysis is given.

Like for example, how it could be possible that Plushenko's program in 2006 vastly outscored Matt Savoie's on Transitions. Or how Ashley Wagner received the highest skating skills component at 2016 Worlds. Or how Patrick Chan could receive such high performance/choreography/interpretation marks for bland performances with many jarring mistakes. Or how Nathen Chen is getting a Patrick Chan-level skating skills component now. Or how Sotnikova could get Level 4 footwork calls when her footwork was only Level 3, and could get her flutz and underrotation ignored. Or any number of other various issues, either in the marks or the tech panel calls.

There have also been judges themselves who have talked about how their scores created a very different result than what they actually felt about the programs. They are being told to assign numbers that they themselves don't understand.

Mathman said:

I don't think so. I don't think we all agree on under-rotation, GOE or anything else. Certainly we do not all agree on what the GOE ought to be for a particular jump, whether by Alina Zagitova, Nathan Chen, or whoever.

Click to expand...

Of course. My opinion of Alina's and Nathan's scoring is just my own. My point, that I also made clumsily, was really that if you're a medal contender, your elements and PCS are viewed by the judges in a more favorable way. That's also an opinion, but I do think it's problematic because certain flaws are heavily penalized and, if overlooked, they have an outsized effect on final placements. At Worlds in the Ladies LP, there was one < given (to Evgenia's 2A) and no "e" or "!" amongst the top 10 skaters in that segment. When everyone gets a pass on technical elements, the events get decided by PCS and that ends up creating a reputation-based outcome.

Blades of Passion said:

There have also been judges themselves who have talked about how their scores created a very different result than what they actually felt about the programs.

Click to expand...

I actually don't find this to be a problem. I'd much rather a judge's score reflect as much objectivity as possible, rather than an "overall feel" about a program.

The score in that case isn't reflecting objectivity, though. It's reflecting the judge's inability to transfer their assessment of the programs into a score that accurately represents their assessment. If they think a certain performance is CLEARLY superior to another, but their PCS only go up a tiny amount in comparison - perhaps because of feeling the need to be in a "corridor", then the score is not reflecting the differential they actually felt. The same goes for GOE on elements. If a judge is being overly generous to a certain skater just because it's what feels "comfortable", then the superior quality they see in another skater will not be accurately reflected in the scores.

There is a problem in the sport if judges think a performance definitely deserves to beat another, yet their scores show the opposite. That means the numbers are not serving their purpose. The point of having a concrete set value for jumps/spins/footwork (and PCS) is supposed be safeguarding skaters so they receive an accurate amount of points for what they accomplish, rather than nebulous "5.6, 5.7, etc" marks that don't hold the judge accountable for each element. However, if judges aren't scoring the elements with a proper amount of differentiation, then their opinion is no longer being accurately rendered. It removes the reason for having the numbers, because numbers are supposed to represent an exact property. 6 apples means 6 apples. Not 5, not 7. That number has a meaning. If the numbers are just being thrown around willy-nilly, then they no longer serve a relevant function.

The points system should mainly serve to decide close competitions, wherein a judge could go either way, and in the end it's left up to the set numbers to decide a more exacting value and do the "checks and balances" for the judge. It's essentially supposed to be a calculator for the judge, having the numbers added up for them, rather than at the end of a performance needing to calculate the exact quality/quantity of everything the same time and weight everything against each other at the same time.

Although, judges will often need to look at technical elements again at the end of a program. Which is part of why there should be a separate panel for "technical element" and "PCS" in this system.

Blades of Passion said:

I should instead say "plain to see, if you analyze the protocols and judging trends". Much work has been put in to point it out over the years. Some people choose to keep wool over their eyes and simply don't respond when analysis is given.

Click to expand...

Still, it seems to me that different analysts come to different conclusions, even when they have access to the same data and are equally knowledgable.

As for wool over the eyes, not every skating fan has the expertise to do the kind of analysis you are talking about. (I don't, for instance. But that doesn't stop me from having an opinion. ) Plus, not everyone is interested in this sort of wrangle.

One of my biggest frustrations in these discussions is when someone posts a video in slow-motion of an under-rotation call or non-call. Half the posters say, "See? If you are not blind you have to agree that this was short." And the other half, looking at the exact same video, says, "See? It was fully rotated, as this video clearly proves."

There are lots of reasons why sports officials disagree with each other, disagree with the fans, or just plain blow a call. This doesn't mean that they are a gang of crooks.

Like for example, how it could be possible that Plushenko's program in 2006 vastly outscored Matt Savoie's on Transitions.

Click to expand...

They didn't get the Joe Inman memo?

Blades of Passion said:

Numbers are supposed to represent an exact property. 6 apples means 6 apples. Not 5, not 7. That number has a meaning. If the numbers are just being thrown around willy-nilly, then they no longer serve a relevant function.

Click to expand...

I think you are asking for the impossible. The judges will never be able, consistently from one competition to another and in unison one with another, to say that this performance has 6 apples worth of musical interpretation, not 5 or 7.

I think the present status -- halfway willy-nilly and halfway serving a relevant function -- is all we can reasonably hope for. The willy-nilly is mitigated to some extent by by averaging, leaving a kernel of relevancy.

Mathman said:

Still, it seems to me that different analysts come to different conclusions, even when they have access to the same data and are equally knowledgable.

Click to expand...

It would probably lessen the outrage over blown calls if they weren't penalized so harshly. One < or "e" could have dropped a skater from 2nd or 3rd to 5th at Worlds. Women have been struggling with rotation all season, and all of a sudden everyone has it fixed at Worlds? We can disagree about any particular call but, in the aggregate, you have to think the standards changed substantially over the course of the season.

drivingmissdaisy said:

Women have been struggling with rotation all season, and all of a sudden everyone has it fixed at Worlds? We can disagree about any particular call but, in the aggregate, you have to think the standards changed substantially over the course of the season.

Click to expand...

That is an interesting way to put it. Yes, I think that in many sports there is a tendency to make strict calls during the regular season, but then when the playoffs start the attitude is, "Let the boys (or girls) decide it on the court mano a mano, not by a referee's call."

I think analyzing the PCS is a great example of a way to question judges' marks. One thing I haven't come across is the tracking of someone who had one good season (say, Tuktamysheva in 2014-2015) versus a much poorer season (I think her 2016-2017 was rough... can't remember which was her worst season in the past four years). If judges place all of her PCS in a narrow corridor when the triple axels are landed (say, 9.00-9.50) but then changing her PCS to 7.50-8.25 when she's missing the jumps, what does that say about their assessment of her skating skills? That they got worse over time to the tune of perhaps over a point and a half? What does that say about the judges' "expertise" in scoring her transitions? That they "magically" got so much worse as her jumps faltered? What about her interpretation? Was it really a point or more better than Young Skater X in 2015 but then over a point worse than Skater X in 2017 all of a sudden? I'm not sure I've ever read a real defense of such decisions or, if I have, that I bought any stock whatsoever in it. Also: why even have different PCS categories if judges are so unwilling to differentiate between them?

Bonus comment: count me in as someone who thinks that the change in GOEs from -3/+3 to -5/+5 just lets the judges abuse scoring even more to prop certain skaters up.

I don't know that any judging system can satisfy all, but man, this current one induces plenty of head scratching, and it also minimizes mistakes. Judges have so much room to elevate scores that a fall in the short program, for example, no longer hits a skater as hard as it used, and I believe that is wrong.

drivingmissdaisy said:

I think one point BoP mentioned that (I think) we all agree with is that scoring of elements, both rotation/edge calls and GOE scores, has almost as much to do with who is skating as it is how well the element is performed. Particularly for clean elements, I think skaters like Alina and Nathan tend to be a bit overscored compared to some of their rivals. I don't subscribe to the "Russia is evil" school of thought, but I think reputation scoring is alive and well, despite COP's efforts to break down the scoring to each specific element and PCS category.

Click to expand...

Reputation scoring is alive and well but all the top skaters benefit from it. Even if some think only the rivals of their favourite skaters are benefitting from it and their fave top skaters aren't benefitting enough or even at all from their reputation.

People also look at different criteria and have varying levels of leeway. Like, judges aren't insanely obsessed with pre rotation as much as certain people are. So there will always be inherent disagreement between what some fans see and what the judges see.

Blades of Passion said:

Or how Sotnikova could get Level 4 footwork calls when her footwork was only Level 3, and could get her flutz and underrotation ignored.

Click to expand...

Mathman said:

Still, it seems to me that different analysts come to different conclusions, even when they have access to the same data and are equally knowledgable.

As for wool over the eyes, not every skating fan has the expertise to do the kind of analysis you are talking about. (I don't, for instance. But that doesn't stop me from having an opinion. ) Plus, not everyone is interested in this sort of wrangle.

One of my biggest frustrations in these discussions is when someone posts a video in slow-motion of an under-rotation call or non-call. Half the posters say, "See? If you are not blind you have to agree that this was short." And the other half, looking at the exact same video, says, "See? It was fully rotated, as this video clearly proves."

There are lots of reasons why sports officials disagree with each other, disagree with the fans, or just plain blow a call. This doesn't mean that they are a gang of crooks.

Click to expand...

Very true! I find it interesting that the people who always accuse the judges of getting it wrong never pause to think that perhaps they also have it wrong.

Especially when it comes to stuff like interpretation, a skater might not have the best jumps or skating skills or transitions, but they can interpret music like a dream!

They didn't get the Joe Inman memo?

Click to expand...

Oooo buuuuurn

Is Joe Inman still working for the ISU, because that memo was shady as heck, he also threw Brian Joubert under the bus!

anonymoose_au said:

Is Joe Inman still working for the ISU, because that memo was shady as heck, he also threw Brian Joubert under the bus!

Click to expand...

Inman should have waited until after the Olympics to go on his crusade against wonky judging. Bad optics.

But actually it was Plushenko who threw both himself and Joubert under the bus. The origin of the Inman memo was a complaint by Plushenko (in good humor, but with a grain of truth nonetheless), "Why does Joubert get such high marks in Transitions? He does the same transitions as I do. None."

Blades of Passion said:

Yes it is productive, because that's exactly what most judges are, and it has been hurting the sport for ages. The scores are a mess and great skating is being lost, as a result of competitors/coaches knowing that superficial programs and lower quality elements can pull huge marks. The judges are not paragons of objectivity and knowledge and scoring skill. They are people appointed by individual skating federations, needing no extensive skating knowledge or judging capability outside of VERY basic requirements, and they are persuaded to judge in such a way that best benefits their country's skating federation (or their own pocketbook, via "gifts" received).

Click to expand...

If you are actually going to accuse judges of actual bribery you’ll need some strong evidence to back it up. Let’s avoid straight up libel m’kay?

Blades of Passion said:

Which is part of why there should be a separate panel for "technical element" and "PCS" in this system.

Click to expand...

This is the only thing that I agree with so far. But I think there should be separate panels simply because of how quickly they have to score everything. I think you'd get better judging if judges were only looking at technical elements or only looking at PCS marks. It would just make the judges' work simpler for them so I think the scores might get more dialled in. Especially for PCS, I often look at detailed scores and it just seems like these are phoned in half the time. Basically giving a skater the same score in every category.

Mathman said:

Inman should have waited until after the Olympics to go on his crusade against wonky judging. Bad optics.

But actually it was Plushenko who threw both himself and Joubert under the bus. The origin of the Inman memo was a complaint by Plushenko (in good humor, but with a grain of truth nonetheless), "Why does Joubert get such high marks in Transitions? He does the same transitions as I do. None."

Click to expand...

Wow, really. I bet Joubert loved that!

Although it's a fair enough question in that case.

DSQ said:

Let’s avoid straight up libel m’kay?

Click to expand...

This would require the ISU to actually give any credence to what certain people write about them. They'll only consider the opinions of people with *actual* influence and not just random skating fans posting their musings/complaints on the Internet. I'd say the opinions of the actual skaters, coaches and other figures of prominence are what matter moreso to them.

And the ISU can't expend the time to go after every grumbling/libellous person. There are plenty of (usually quite biased) curmudgeons who, when they disagree with a result or scores, parrot the increasingly predictable go-to blanket statement that the ISU is all corrupt, blind, bought-off, uneducated, etc. Hey, whatever makes them sleep better. The ISU doesn't have the energy to address all the saltiness nor should they waste tim doing so. That's what us other random Internet skating fans/forums are for!

LeReveur said:

I think analyzing the PCS is a great example of a way to question judges' marks. One thing I haven't come across is the tracking of someone who had one good season (say, Tuktamysheva in 2014-2015) versus a much poorer season (I think her 2016-2017 was rough... can't remember which was her worst season in the past four years). If judges place all of her PCS in a narrow corridor when the triple axels are landed (say, 9.00-9.50) but then changing her PCS to 7.50-8.25 when she's missing the jumps, what does that say about their assessment of her skating skills? That they got worse over time to the tune of perhaps over a point and a half? What does that say about the judges' "expertise" in scoring her transitions? That they "magically" got so much worse as her jumps faltered?

Click to expand...

Here is a point of view that I think has some merit.

The way the IJS works is that there are two parts: Individual Elements and Program-as-a-whole. For individual Elements it is more or less straight forward. You get a base value for each element attempted, with some extra pluses for high quality and minuses for errors. So far so good.

But when it comes to the PCSs, the question that the judges are expected to keep uppermost in their minds (IMHO) is "How much did this or that feature of the skating contribute to the effectiveness of the program as a whole." For SS and TR, its not so much whether the skater has deep edges (check) and effortless acceleration (check) or whether she does chocktaws in both directions. It is, rather, how well did the skater utilize these aspects of her skating to present a complete program.

As for falling on your triple Axel, that lessens the impact of your wonderful deep edges and of the intricacy of your transitions. It detracts from the effectiveness of the choreography and musical interpretation. After all, if you plan a huge triple Axel right on the beat the the glorious musical crescendo, but then pop the jump -- well, the program-as-a-whole scores suffer appropriately, regardless of what the bullet points say.

Bonus comment: count me in as someone who thinks that the change in GOEs from -3/+3 to -5/+5 just lets the judges abuse scoring even more to prop certain skaters up.

Click to expand...

The overall effect of this change is not so clear to me. In the olden days you could only get a +3, but if you did you got 3 extra points. Now you can get a +5 on a judge's screen but in reality you only get a percentage of that.

At the 2010 Olympics LP, Yuna Kim got a total of 17.40 extra points on GOE, with mostly 1s and 2s. At 2019 Worlds, Alina Zagitova earned 16.22 in GOE with 3s and 4s and a smattering of 5s. It seems like both systems are equally vulnerable to being manipulated by unscrupulous judges, should they set their minds to doing so.