2017_18 ISU Judging Anomalies

[LKGwennire]

Ms. Ryan gave Kaetlyn's 2A with a fall -2 GOE... :noshake:

 

[dorispulaski]

We thought the "Shame" part of the title might not be a great idea. Tentatively, we retitled the thread:

"2017_18 ISU Judging Anomalies"

Does that work, do you think?

 

[Blades of Passion]

I think "shame" was exactly the correct title. If an individual in society does something that is universally considered wrong, then shame is exactly what they should feel. This emotion is supposed to prevent people from committing heinous acts. The judging was finally made non-anonymous again for this reason: so that the individuals responsible for the scores can be directly called out by anyone and taken to task for exactly what they are doing.

 

[FSGMT]

Ms Ryan must be a GoldenSkate member who has read this, since her PCS marks tonight looked completely different, up to the point of giving a 9 to Courtney... did Courtney train all night to improve her skating so much?

 

[Mathman]

I wonder if the referee had a discussion with the judge after the SP scores were in. In the LP, this judge gave much higher scores to all non-Osmond skaters except Pogoriliya. For Karen Chen and Courtney Hicks, the scores of judge #6 were distinctly higher than the average of the panel.

I remember in 6.0 skating, there would be a pause after the first skater performed. The referee would calculate the median score and communicate it to the judges. It was never completely clear to me what the judges were supposed to do with this information, but at least they could get a feel for what kind of scores the other judges were giving out that day.

 

[Sam-Skwantch]

I think "shame" was exactly the correct title. If an individual in society does something that is universally considered wrong, then shame is exactly what they should feel. This emotion is supposed to prevent people from committing heinous acts. The judging was finally made non-anonymous again for this reason: so that the individuals responsible for the scores can be directly called out by anyone and taken to task for exactly what they are doing.

Well..I’m not sure if we should give ourselves much credit here. It’s just as likely someone from the ISU noticed her scores were unusual. IMO they were strikingly unique and more importantly not reflective of consistent judging.

 

[gkelly]

It would be interesting to hear this judge's explanation of what criteria she is using when she gives out scores in the 5's and 6's for skating skills, etc.

Yes. It would be interesting to get explanations from all, or any, judges to have some idea of the kinds of things they're thinking, rather than just guessing.

Any skater who is invited to a senior grand prix event is going to have a lot going for her, even if she gives a flawed performance.

Sometimes what gets them invited is a combination of difficult elements and home country support, in which case average to above average senior-level program component skills might suffice to get an invitation.

But yeah, most skaters on the GP are going to be better than just "above average," at least in some of the components.

And how many are strong in some but just average in others? If you expect wide ranges of components, that's what would justify larger gaps.

Analysis, both of the judge's scores in relation to other judges, and critically examining the skating itself and discussing what the scores should have looked like.

So how do we know what the scores should have looked like?

Does each analyst decide for himself what the "right" scores are?

Does the panel as a whole determine what's right and the outliers are therefore wrong?

If judges could document their thought processes, would their scoring be good if the process they used to achieve those scores was good even if their scores don't match those of other judges who saw or focused on different details?

(Similarly, if a judge scores purely by protocol, knowing what kinds of scores each skater has earned from other panels in the past and adjusting slightly up and down for clean and messy performances, the process is all wrong but the numbers would usually match the majority. Should that be considered good or bad judging?)

 

[StitchMonkey]

We thought the "Shame" part of the title might not be a great idea. Tentatively, we retitled the thread:

"2017_18 ISU Judging Anomalies"

Does that work, do you think?

I like this as it can include what some are calling "good judging" i.e. the judges who go against the grain and judge accurately rather than say on reputation. That is still an anomaly of sorts even if a good one. I think too many threads discussing judging could be uncomfortable. But having one thread like this to look at shall we say statistical outliers, much like The Skating Protocol does, could be fun and useful.


But we need to remember that the judges are people to who are part of our great big family of those who love figure skating...aka no pitchforks. This should be a tongue in cheek with a beer in hand discussion.

 

[Blades of Passion]

I wonder if the referee had a discussion with the judge after the SP scores were in. In the LP, this judge gave much higher scores to all non-Osmond skaters except Pogoriliya.

It's an effective tactic: overscore a skater and underscore everyone else in the SP, so as to give that skater a cushion heading into the LP and make them "appear" more dominant (which then boosts their reputation score from the rest of the panel in the LP), and then after everything has been set in place by this manipulation, just score normally in the LP so as to not fall outside of the "judging corridor" again.

 

[noidont]

Ms. Ryan overcorrected herself today and scored everyone higher....
I think sometimes judges misjudge their peers' understanding of range. It's usually not a big deal.

 

[gkelly]

I remember in 6.0 skating, there would be a pause after the first skater performed. The referee would calculate the median score and communicate it to the judges. It was never completely clear to me what the judges were supposed to do with this information, but at least they could get a feel what what kind of scores the other judges were giving out that day.

If they were far off, they could change their scores for that first skater to be more in line with the median and then compare subsequent skaters to that adjusted score.

But a judge could still end up being out of line for the field as a whole if, say, she originally scored Skater A significantly lower than the median, adjusted her score to be only slightly lower than median, and then honestly according to the rules and standard believed Skater B deserved to place behind A while the majority of the panel thought B was better than A. This judge's scores for B would be significantly lower than the panel's, especially because in 6.0 judging she would have had to leave enough room, just in case it turned out she wanted to slot numerous skaters in between those first two.

With IJS, there's no need to leave room -- if the first two skaters are close in what they delivered, their scores can be very close.

 

[Miss Ice]

It's an effective tactic: overscore a skater and underscore everyone else in the SP, so as to give that skater a cushion heading into the LP and make them "appear" more dominant (which then boosts their reputation score from the rest of the panel in the LP), and then after everything has been set in place by this manipulation, just score normally in the LP so as to not fall outside of the "judging corridor" again.

This. Have seen this numerous time especially in the most pivotal competitions...

 

[drivingmissdaisy]

Ms. Ryan overcorrected herself today and scored everyone higher....
I think sometimes judges misjudge their peers' understanding of range. It's usually not a big deal.

The only problem, as BoP alluded to, is that if the judge doesn't score everyone higher (or lower), then only a select group of skaters will benefit. I have no issue with scoring "outside the corridor" as long as the judge is equally harsh (or lenient) with all skaters.

 

[Shayuki]

Unfortunately, judging is very fame-based and also all about creating interesting storylines rather than accurately gauging performance. It's ridiculous there can even possibly be a 2 point swing in the PCS scores the judges give out. They all should be relatively close to each other, there need to be some guidelines they have to follow. And the judges should be held at least somewhat accountable. There should be a governing body that actually punishes / demotes judges who judge poorly. 9.5 vs 7.5 in a PCS category just plain cannot be a "judgement call".

 

[Eclair]

I'm not sure whether this has been already mentioned, but the Australian judge (the reason for this thread) didn't only judge ladies but other disciplines, too and e.g. she gave Shoma even slightly higher PCS than Patrick. So much for being the secret Canadian judge ....

 

[Shayuki]

I'm not sure whether this has been already mentioned, but the Australian judge (the reason for this thread) didn't only judge ladies but other disciplines, too and e.g. she gave Shoma even slightly higher PCS than Patrick. So much for being the secret Canadian judge ....

Looking at her scoring, I just think that she's blatantly incompetent. So no malice meant.

But here we might get to some discussions like "should there be judges from so many countries, even ones that are rather obscure figure skating countries?" Maybe there are some fairness concerns, but actually I haven't found the judges from the same country to even be that massive homers. Would it really be a problem if judges were only from high calibre figure skating countries?

 

[Violet Bliss]

Unfortunately, judging is very fame-based and also all about creating interesting storylines rather than accurately gauging performance.

This is a personal opinion stated as an absolute fact.

It's ridiculous there can even possibly be a 2 point swing in the PCS scores the judges give out. They all should be relatively close to each other,

Another opinion/belief with no basis.

there need to be some guidelines they have to follow.

There are guidelines.

And the judges should be held at least somewhat accountable. There should be a governing body that actually punishes / demotes judges who judge poorly.

That governing body is ISU.

9.5 vs 7.5 in a PCS category just plain cannot be a "judgement call".

It is exactly a "judgement call".

If you mean the difference is between two different judges, that may be because they see the quality differently, with different appreciation biases and perspectives, or different standards so one may be judging everybody more harshly or leniently across the board. If judges can only score within tight limits, we wouldn't need so many judges. And who should decide the limits for all the judges to go by? Then who needs judges?

Judges are opinion givers, like you are, but they are pros trained for their jobs in a long strict process. Still, opinions vary, which is why we have a whole panel of them at each competition, and the lowest and the highest scores are discarded so the final score is as close to a consensus as possible. Someone like you and I have our opinions but we are not on the panel and we cannot swipe out the panel's consensus with a personal opinion.

 

[Blades of Passion]

That governing body is ISU.

But that's the problem.

And no it isn't a personal opinion that the scores are generally handed out based upon reputation and the storylines the federations want to create, rather than an objective assessment. MANY judges are on record about how they've been told (forced) to hold certain skaters up to create the most bankable result. Everyone who has any involvement inside the sport knows it, stop trying to play innocent.

 

[Sam-Skwantch]

Unfortunately this thread is no longer focused on discussing individual judge’s marks for events during the 2017 season which was the whole intention and has now lead to a discussion on judging anomalies in general. Good or bad I was interested in discussing specific scoring of individual judges. See the first post. Now it’s just gone off the rails about generalizations and hypothetical anomalies in judging.

 

[GF2445]

Wow.
Just Wow.
I love you guys so much for this discussion. Keep them coming