PCS/Reputation Judging

My understand is that PCs are currently not judged anywhere near how they are supposed to be.
This is why I prefer it without PCs - I´m not against the concept of FS as sport+art, but what is currently done is a travesty and we would be better without it.

moriel said:

My understand is that PCs are currently not judged anywhere near how they are supposed to be.
This is why I prefer it without PCs - I´m not against the concept of FS as sport+art, but what is currently done is a travesty and we would be better without it.

Click to expand...

OK, forget art for a moment. The Program Components honor the fact that a figure skating performance is more than just a catalogue of the jumps and spins that you tried. The skater is expected to weave the technical elements together and make a program out of them. Do you think that this is a bad thing?

People will always complain about the officiating of sports events. To me the greater danger is that figure skating will become just another Scandinavian/Alpine winter pastime, like snow-shoeing pr biathlon.

(Actually, biathlon is pretty cool if you think abut it. You ski across country for a while, then you stop and shoot a gun. )

moriel said:

My understand is that PCs are currently not judged anywhere near how they are supposed to be.

Click to expand...

How do you think they're supposed to be judged?

I get the impression that a lot of people have an idea in their minds about one particular way that they think scores should be awarded, which is not necessarily supported by the official rules, and then get upset that the reality doesn't seem to match their expectations and want to throw the whole thing away.

For me, I'm more interested in trying to figure out what the rules are asking for, what the typical thought process might be, and how the process could be made better within the current rules before throwing them away or revising significantly.

In the case of a big mistake such as fall, the judge who gave 10 points by PCS judged that the PCS was not affected by a big mistake.
Rules from this season take the idea that falls and the like affect PCS.
It is not clear whether mistakes will affect PCS.
Judges did not unify the way of thinking even before last season.
Even in this season, there is a judge who issued 10 PCS points when Medvedeva fall.
http://www.isuresults.com/results/season1718/gprus2017/gprus2017_Ladies_FS_Scores.pdf
I think that it is the cause of confusion that the idea of PCS is not unified.
Whether the PCS is affected by mistakes。

rabbit1234 said:

In the case of a big mistake such as fall, the judge who gave 10 points by PCS judged that the PCS was not affected by a big mistake.
Rules from this season take the idea that falls and the like affect PCS.
It is not clear whether mistakes will not affect PCS.

Click to expand...

Not every fall will affect the performance. In fact...Zhenya’s fall where she laughed and never really lost the mood or diminished the performance is probably a good example of this. Not saying I would score it a ten with the fall but then again...let’s say I was planning on scoring her 9.25 for Performance. I wouldn’t have adjusted that score due to the fall because it didn’t impact my view of the performance.

Kostner's case is the same. Those who criticize the judge think that "She had a fall."
On the other hand, does not the judge think that fall will affect PCS so much?

Sam-Skwantch said:

Not every fall will affect the performance. In fact...Zhenya’s fall where she laughed and never really lost the mood or diminished the performance is probably a good example of this. Not saying I would score it a ten with the fall but then again...let’s say I was planning on scoring her 9.25 for Performance. I wouldn’t have adjusted that score due to the fall because it didn’t impact my view of the performance.

Click to expand...

This is where it gets hazy though, because then a skater could fall 5 times and it won't affect a judge's view of the performance.

Whether a skater gets up or smiles after a fall, or if it is just a blip in a 4 minute program doesn't matter. It's an obvious error and blemish on the program, and thus it is not a 100% - i.e. perfect - performance. A 10 should only be reserved for 100% clean skates. THAT is what the ISU's "recommendation" should have been. Not "recommending less than a 10 for a fall".

They should be mandating judges can only give 10s for a clean skate. But the fact that they would have to do something like recommend that the judges only give perfect marks for perfect performances (imagine that!) just boggles the mind. The fact that a judge basically ignored that recommendation (and others essentially still ignore it with 9.5/9.75s) with no repercussions is even worse.

ask said:

And that should have never been the case. They should receive similar tier PCS, the difference should be small and should base on actual choreo, actual P/E of the day. Of course, Caro should always have a 1 point margin over Mao in SS.
She got the wrong end of the stick pre 2010 and right end post 2010. Mao, the reverse.

Click to expand...

Here is a heat map of the Sochi LP showing that Mao had the greatest rink coverage of any women skater, which is the best indication for the greatest speed and it also shows the greatest control because she placed her jumps with superior speed and flow so near the boards which is one of the things that gives her performance such an epic feel. The speed with which Mao covered so many corners of the rink with her final jumping passes especially shows her superior skating skills. Furthermore, Mao had a lot of transitions in the program with excellent posture and minimal pumping and hunching of the body to gain speed. To me, this in addition to the artistic and emotional power, choreographic subtlety and complexity should have given her the the highest PCS based on what she delivered on the ice, in addition to the highest tech score because her base was 5 points higher than any one else and she received bogus ur calls, strict edge calls and deflated GOE especially compared to the generous scoring in the final group. When, these are combined, I believe Mao should have set a record score for Rach II at Sochi that should stand to this day. Just as I believe her Nocturne SP at Worlds 2014 should still hold the record with a higher score especially in PCS.

Heat map of Sochi LP rink coverage:
Mao Asada with the most
http://4.bp.blogspot.com/-DOPsmfFI0...he-program-olympic-performance-2014-sochi.png
https://ibb.co/ctSu0w
https://ibb.co/ctSu0w
https://ibb.co/kFtbfw

Adelina Sotnikova
https://ibb.co/jvSK0w

Carolina Kostner:
https://ibb.co/ejPtSb

Yuna Kim's rink coverage:
https://ibb.co/noFNLw

Obviously, the rink coverage was similar in Worlds 2014 skating from the first group and Mao still did not receive the highest PCS even with a performance that was much cleaner than Carolina's. The political liability and disfavor that Mao faced in scoring and training conditions is evidenced by the fact that Mao was forced by the JSF to practice in a sandy and bitterly cold rink in Armenia which required major adjustment to her blades and because her team's request to train in Japan were rejected. Sounds like a plan to undermine not to support. Mao even returned early to Sochi because of the poor training conditions with little training and again had to readjust her blades. It has been said that this created a lot of stress and in my opinion, this contributed a lot to Mao's nerves in the SP.

As for Mao in 2010 before, there is widespread harsh criticism of Mao's underscoring in tech and PCS especially at Worlds 2010 and the Olympic SP. Furthermore, Mao certainly was not given big PCS advantages overall. Mao was underscored with a second place PCS in the Grand Prix Final LP 2007 with a score of only 59 for a stunning performance with two triple triples, a triple axel, excellent rink coverage, nice transitions and sophisticated choreography. Early in her career Mao had many of the same outstanding qualities in PCS she had in the Sochi LP, but had yet to be underscored to the same extent she would be later for the same quality of skating, though she was always hit with harsh tech calls comparatively which more than negated the minute advantage she definitely earned in PCS relatively few times anyway: no more than around 1 point in the SP or 2 points in the LP, and she worked with a PCS disadvantage beginning during the 2008-2009 season. For those criticizing Mao's Worlds 2009 LP, please remember that someone yelled for Mao to fall on the entrance to her second triple axel that can clearly be heard over the announcers and music during the performance. I didn't even know about it until I watched the video of the performance and heard it, so I believe this clearly interfered with her performance in a way that is not her fault and should have been compensated in scores for a competition where she was already underscored.

Here are a couple of pre-IJS free skates that haven't inspired a lot of fan analysis in the past, so even if you have seen them before you should be able to watch them with relatively fresh eyes and few preconceptions:

https://www.youtube.com/watch?v=72jCfd8UlRc
https://www.youtube.com/watch?v=pmuArdNNsVk

Choose one or more or all of the IJS program components as currently defined.

How would you rate these performances on your chosen component(s) on a scale of 0 to 10 based on your understanding of the guidelines.

Don't worry about what technical elements they have chosen to do or how the content would hold up in 2017 or fit 2017 well-balanced program rules. But do consider anything about the execution or placement of the elements that affects the relevant PCS.

I'm curious about the thought processes that go into evaluating these criteria, without room for much effect of reputation.

gkelly said:

How do you think they're supposed to be judged?

I get the impression that a lot of people have an idea in their minds about one particular way that they think scores should be awarded, which is not necessarily supported by the official rules, and then get upset that the reality doesn't seem to match their expectations and want to throw the whole thing away.

For me, I'm more interested in trying to figure out what the rules are asking for, what the typical thought process might be, and how the process could be made better within the current rules before throwing them away or revising significantly.

Click to expand...

Well, for instance, we can clearly see that judges give the same score to all the categories.
If a skated has a great performance and terrible skating skills, the skater will just get same 7-something in both, for example.
In many occasions, the PCs clearly correlate with skating order or reputation, which is once more not in the rules.
Honestly, I tried more than once to try correlating the numbers that we see, and the values that are given out. For example, I´m curious to see an explanation, from the rule's point, of the 2 point difference in PCs for the performances at Rostelecom and NHK of Caro. Ok, lets not compare between events (although why cant we, arent rules the same between events so technically things should be scored same way?), can someone explain the PCs ladies got at CoC?
And so on and so on.


While TES is also flawed, but it still makes some remote sense. PCs don't, or we wouldnt have so many discussions where people use "i liked it" / "i didnt liked it" as main arguments. Since this happens, something is being done wrong, very wrong, period.

As a PS, I strongly believe that a good scoring system should *make sense for a casual viewer*. So an untrained person can, without or with minor explanation, understand why one performance is better than the other. For example - the way jumps are scored is decent: one jump had 3 rotations, and the other had 4, 4 is more difficult so gets more points. The landing on one jump was scratchy, and the other was clean. And so on.
This does not happen with PCs really.

Overall, i believe that tracking more details about the judges decisions would improve it, because at least we would have some clue why a judge gave a 9 here, and a 7 here.
This could be done by, for example, forcing them to evaluate each bullet point individually, so we would have some idea where the points come from.
Same goes for GOE, imho.

The way it is currently done, imho, it is just frustrating for most people who watch it, because even after spending a few days studying the rules, it is usually not possible to figure out why one performance is better than another, and why each received a specific score. The FS scores are VERY debatable, and that is wrong in a judged sport.

I´m for whatever makes it more objective and less debatable, and since IS doesnt seem to be able to fix PCs, i prefer they remove it alltogether. Scores should make sense.

CanadianSkaterGuy said:

This is where it gets hazy though, because then a skater could fall 5 times and it won't affect a judge's view of the performance.

Click to expand...

If a skater can fall 5 times and keep the mood and quality of performance to the level of that of Zhenya’s smiley fall then I just might even go back and give them some extra GOE on my Scoresheet and maybe boost their PCS

I was addressing the point someone made and only using the current rules for the sake of this discussion. I’m sure you’ll remember I started a thread to ponder the idea of mandatory PCS reduction for falls.

https://www.goldenskate.com/forum/showthread.php?66901-Mandatory-PCS-Reduction-for-Falls&highlight=

Mikhail KOLYADA has fallen three times at ISU GP Rostelecom Cup 2017 and updated his PCS's highest score.
http://www.isuresults.com/results/season1718/gprus2017/gprus2017_Men_FS_Scores.pdf

rabbit1234 said:

Mikhail KOLYADA has fallen three times at ISU GP Rostelecom Cup 2017 and updated his PCS's highest score.
http://www.isuresults.com/results/season1718/gprus2017/gprus2017_Men_FS_Scores.pdf

Click to expand...

I would rather take his SP as example.
He was pretty meh (42.19 TES) at Rostelecom, and got 43.60 PCs.
At Coc, he had a clean awesome SP, almost hitting 60 TES btw, and got 43.50.

That is something I would really like to see explained from the rules standpoint.
And this is why I believe that, since IS seems incapable to make PCs work, they should just drop it out. PCs are not about art or performance. PCs are about politics.

ISU should clarify whether mistakes will affect PCS.
And each judge should share the way of thinking and make a judgment.
Because that point remains ambiguous, a judgment that can not reasonably be explained is made.

moriel said:

Well, for instance, we can clearly see that judges give the same score to all the categories.
If a skated has a great performance and terrible skating skills, the skater will just get same 7-something in both, for example.

Click to expand...

Can you give some examples?

This is the kind of thing I mean. The five components are indeed supposed to be evaluated independently, and it is true that most judges score most skaters' components within 1 full point of each other between highest and lowest.

The average scores for the whole panel are even closer, even if some judges use wider ranges. Even if all judges tended to score 2 or more points between highest and lowest scores, the averages might well flatten out to less than 1 point.

Does that mean that judges are not even trying to evaluate the scores independently?

Does the overall impression of the program affect judges' impressions of the individual components in the same direction?

Is there enough (too much?) overlap in the way the components are defined, so that if there's a weakness in one area it will often affect other components to some degree as well?

Even when scored independently, do most skaters' skills fall into a relatively narrow segment within the broader range of skills from beginners (0s) to creme de la creme (9s)?

My sense from all the skaters that I have seen on video and live (more often lower levels live lately) is that it's rare for there to be huge differences between one component and another. At higher levels there's more room for wider ranges, but skaters are often limited in how well they can achieve other components if their skating skills are limited, for example.

If I disagree with official judges' scores, does that mean that I'm right and they're wrong? That they're right and I'm wrong? That we both used the right process but came up with different results because of our different backgrounds and our different views of this performance?

No two human beings are ever going to come up with the exact same numbers for every component on every skater in an event. We each bring different knowledge levels and preferences/pet peeves and expectations and mental blind spots to the process. In real life we each have a slightly different view of the performance. That goes for all judges and all fans. In general, though, the judges have a better view and a better knowledge level about skating (but not necessarily about art) than most fans.

If we all watch the same video, we'll all have the same view, but it will be lacking immediacy compared to live viewings. Choice of camera angles, editing and commentary if present, etc., can all impose someone else's opinions between us and the actual skating.

So I think we need to expect differences of opinion, and respect that other judges and other fans might see things differently than we do, and judges will often differ from each other.

To me, discussions that share our different perspectives are more interesting and much more honest than in passing judgment against those we disagree with.

I like figuring out how I could use the PCS criteria better as currently written, how official judges could use the criteria better, what rule changes might improve the overall process for everyone, etc.

But before we can make good arguments for improvements, I think we need to do our best to understand the current process as well as possible and to understand

I do not think that expecting large gaps between highest and lowest components for most performances is an appropriate measure of whether the process is working mostly as it should. Nor do I think comparing the scores of different panels a different events is a good measure.

We don't get to talk to the judges, so we don't know their thought processes. My inclination is to give them benefit of the doubt and to focus on my own thought processes and those of other fans who are interested in going through the process themselves and discussing, including what might get in the way of us giving the best possible scores we could.

I think we learn more that way than by expecting numbers from the official panels based on superficial expectations before immersing ourselves in the process ourselves.

Honestly, I tried more than once to try correlating the numbers that we see, and the values that are given out. For example, I´m curious to see an explanation, from the rule's point, of the 2 point difference in PCs for the performances at Rostelecom and NHK of Caro. Ok, lets not compare between events (although why cant we, arent rules the same between events so technically things should be scored same way?), can someone explain the PCs ladies got at CoC?
And so on and so on.

While TES is also flawed, but it still makes some remote sense. PCs don't, or we wouldnt have so many discussions where people use "i liked it" / "i didnt liked it" as main arguments. Since this happens, something is being done wrong, very wrong, period.

Click to expand...

But it's fans who are using those arguments not judges. If fans are not doing a good job of applying the criteria, let's focus on doing a better job ourselves -- and respecting our differences while learning from each other -- before we pass judgment on what the official judges are doing.

As a PS, I strongly believe that a good scoring system should *make sense for a casual viewer*. So an untrained person can, without or with minor explanation, understand why one performance is better than the other.

Click to expand...

I think it is impossible to judge skating technique accurately without having spent many years in ice rinks observing analyzing, and if possible doing it oneself.

The Skating Skills component is all about technique. And it is, to my mind, the single most important thing that is being evaluated in skating competitions, even more than the elements. (Others may disagree on relative importance.)

The other components contain some criteria that are highly dependent on technique and others that are more independent, including some that draw on standards for evaluating performing arts. (And some fans are more knowledgeable about some artforms or artistic theories than many judges.)

But a casual viewer who knows nothing about skating technique and little about art beyond what they enjoy or don't enjoy watching is not in a position to have opinions beyond "I liked this," "I didn't like that," "Ooh, that was an obvious mistake."

Good commentary could help focus viewers on the kinds of criteria judges are applying when scoring the programs. But historically (at least in the US, which is the commentary I'm most familiar with), TV networks have focused more on human interest and jumps, geared toward casual viewers with no real knowledge or desire to learn much, than on really explaining skating technique or what the Presentation or all the PCS criteria are.

Overall, i believe that tracking more details about the judges decisions would improve it, because at least we would have some clue why a judge gave a 9 here, and a 7 here.
This could be done by, for example, forcing them to evaluate each bullet point individually, so we would have some idea where the points come from.
Same goes for GOE, imho.

Click to expand...

That would be useful for educating viewers -- and explaining decisions to already-educated viewers and skaters themselves.

The trick would be getting it done efficiently, affordably, and without the record-keeping/data entry tasks distracting from the actual evaluation.

Meanwhile, because practical limitations prevent something from being done perfectly, does that mean it shouldn't be done at all?

I´m for whatever makes it more objective and less debatable, and since IS doesnt seem to be able to fix PCs, i prefer they remove it alltogether. Scores should make sense.

Click to expand...

If they make sense and have value for the competitors, but not for someone who just tuned in knowing nothing about skating, does that mean that the needs of the casual viewers should take precedence over the needs of the athletes who have dedicated their lives to the sport?

I generally watch gymnastics or diving or freestyle skiing or similar sports during the Olympics. I don't know anything about the technique or the rules or scoring processes beyond what tv compentators tell me. I know which performances I enjoy more than others or look better to me, but I would never presume to question that the judges must be doing something wrong if the athletes I enjoyed best didn't win.

I'm sure judging in those sports has its limitations as well. Skating is much more complex than most, so I expect even more differences of opinion.

It's the complexity that I love. Even the fact that it is not conducive to black-and-white answers.

Click to expand...

rabbit1234 said:

Basically, because we judge with the same rule, it is wrong that scores can not be compared by competition.
The GPF selection criterion is also based on the total score.

Click to expand...

That will not be reasonable unless we have the same judges at each event. Until then you have to accept that judges use descretion. Plain and simple that is the nature and intent of judging. The judges in essence have replaced the "score board"

gkelly said:

We don't get to talk to the judges, so we don't know their thought processes.

Click to expand...

It is possible to get occasional insights into a judge's thinking though: Japanese TV regularly has Yukiko Okabe commentating on repeat showings of major international competitions. She is an experienced and active ISU judge and technical panellist, and as such her comments are often veeery interesting.

She doesn't give complete evaluations of each skater's performance, and what she says is surely nowhere near a full representation of an ISU judge's thought processes. But still - the comments she gives are very educational. She points out stuff that is good, and stuff that is "not so good" in performances... and some of it can be quite surprising, and/or go completely unnoticed by lay-people. It gives an idea of just how differently judges might see things, vs the rest of us.

If only such commentaries were easily available and in more languages...

rabbit1234 said:

And each judge should share the way of thinking and make a judgment

Click to expand...

At major events, there are round-table discussion meetings for the officials after each competition.

I don't exactly know what they discuss at those meetings, but if it's about how the judging, tech panel calls etc went for the competition, then I'd like to see videos of those meetings made public. Or even just transcripts of what was said if they think it has to be anonymous. I want them to justify their scoring. So that judging decisions are more transparent and accountable.

YesWay said:

It is possible to get occasional insights into a judge's thinking though: Japanese TV regularly has Yukiko Okabe commentating on repeat showings of major international competitions. She is an experienced and active ISU judge and technical panellist, and as such her comments are often veeery interesting.

She doesn't give complete evaluations of each skater's performance, and what she says is surely nowhere near a full representation of an ISU judge's thought processes. But still - the comments she gives are very educational. She points out stuff that is good, and stuff that is "not so good" in performances... and some of it can be quite surprising, and/or go completely unnoticed by lay-people. It gives an idea of just how differently judges might see things, vs the rest of us.

If only such commentaries were easily available and in more languages...

Click to expand...

Are there any translations of her commentaries? I would love to see them if possible

Can you give some examples?

Click to expand...

See Kolyada's SP at Rostelecom and at CoC. See Kostner's FS at Rostelecom and at NHK.

The average scores for the whole panel are even closer, even if some judges use wider ranges. Even if all judges tended to score 2 or more points between highest and lowest scores, the averages might well flatten out to less than 1 point.

Click to expand...

That is not how it works (and i speak here as a statistician, not an FS fan).
Suppose that we have a way to measure exactly the real score for, lets say, SS and IN of a specific skater. Lets call those values SS_R and IN_R.
Now, we have a pool of trained judges, and each of them produces their own estimate of those values. Those are the scores the judges give. Of course, the score each judge gives will not necessarily be equal to SS_R and IN_R. The assumption current skating system makes is that, if we have a very huge number of judges, the average of the scores they give will be very close to SS_R and IN_R. Lets call this the estimated score, SS_E and IN_E
Now, we expect that, if all is done fairly and correctly, that the difference SS_E - IN_E will converge to SS_R - IN_R as we increase the number of judges.
Even worse, notice that, since this is a difference, it should be less affected by homecooking, reputation scoring and so on, because those will tend to cancel out.
TL;DR: Average *MUST* conserve the difference between 2 categories.
Now, if practice does not work like this, it means that practice does not match the theory - for example, that the scores given by all judges are somehow biased, and this bias is not same for all the categories.
For example, lets take a look at the ladies SP at worlds 2017 - we have a lot of skaters of quite different levels here, all on same competition, judged by same pannel. We can use this database to estimate some quantities. Based on this sample of 37 skaters, we have:
- the average difference between highest and lowest scored PC category is 0.43.
- the median is also 0.43, which means that, for 50% skaters, ALL THE PCs SCORES FALL IN A 0.43 range.
- for 80% of skaters, this range is of 0.5
- the highest observed difference was 0.82

So, I have a question. I understand that PC categories usually correlate, and you cannot really score 10 in transitions and 0 in skating skills. But is that accurate? Is that correct that most skaters have their skills very evenly distributed in PCs department, all around some level? Is it not possible for a skater to have SS 9 and IN 8, for example - because, based on ACTUAL DATA this seems like something VERY unlikely to happen.

Now, there are 2 possible answers:
1. Yes, it is very unlikely for a skater to have their REAL SS equal to 9, and their REAL IN equal to 8 (notice, not the scores given by judges).
2. No. Which means the judges scores do not represent reality. Which means scoring system is bad.

But it's fans who are using those arguments not judges.

Click to expand...

A good scoring system should not be that debatable. For real, if people who watch all FS competitions including juniors for years and know the rules by head cannot come anywhere close to some agreement, that means, in best case, that the variance of our measure (score) is too high and depends on too many factors other than the actual performance. Everywhere outside FS, this means a measure is bad and should be replaced by something more accurate.

If they make sense and have value for the competitors, but not for someone who just tuned in knowing nothing about skating, does that mean that the needs of the casual viewers should take precedence over the needs of the athletes who have dedicated their lives to the sport?

Click to expand...

Technically, skaters get paid because people watch them. So I totally see how the needs of people who buy tickets and merchandise, pay to watch it on TV/online and so on are relevant.
Also, here, I dont mean that the scoring system should be easy or primitive. It should just make sense with what a person watching it sees.
Take soccer for example. Where are quite a few rules and nuances, but if you take a person who never watched it, you can explain the basic principle in 1-2 sentences. This explanation, would enable this person to understand the score and predict the "official" score accurately after watching a game. This is a good scoring system, because the viewer dont really need to know all the bullet points by head, and spend hours thinking if they apply.

Its not like the viewer should be able to question the score judges give.
The viewer should be able to intuitively understand it with minor explanation (that does not involve reading 100500 pages on the internet and blablabla), and be able to identify a good and a bad performance.
Take gymnastics: you see two girls, one falls from the bars, the other doesnt. Is the score the judges give usually match with your expectation that the first one will lose and the second one will win?
Right now, it does not happen. Take an example: Polina's FS at NHK. So there is this viewer. He sees a clean performance and the 74 at the TES box, 4 points above the TES of the previous best. Then, there are replays, and suddenly he finds out that this clean performance with 74 TES scores 4 points less than a performance with lower tes, a fall and a messy landing.