Justifying claims of under/over scoring

[drivingmissdaisy]

Personally, I blame the IJS. Under ordinal judging I was perfectly content to say, I thought this performance was better than that, Sometimes the judges agreed, sometimes they didn't, and sometimes there was a split panel with some judges agreeing with my opinion and others having a different view.

Regardless, i could always list what I thought were cogent reasons for my opinion, while also listening to what others thought. The viewer felt engaged and invested.

Now…well, we can't argue with the computer that adds up the scores, so all we can do is say meaningless things like, "I thought she should have gotten 8,75 instead of 8.25 in choreography, or "I didn't think he satisfied the bullet point for smooth knee action," or "82.3 for that!? I wouldn't give it a tenth more than 80.4."

I'm confused as to what the difference is between a judge giving a skater 5.9 vs. an 8.75. Yes, IJS does give the impression of being more precise when it may not be, but it has also steered skaters towards more intricate programs and better separates programs of different content levels. Under 6.0 skaters could still get 5.8's with 5 clean triples, but 7 triples still wouldn't get you a 6.0 on technical merit. A skater could also get 5.9's with 7 triples and mediocre spins. With IJS, those differences are better reflected in the scores.

 

[Procrastinator]

I'm one of those people who has tried to learn more about the code so that I can make reasoned comments, especially as I have never skated. However, while learning about levels and GOEs is quite easy, PCS are not at all transparent and it is impossible to discern why they're awarded. Nationality of the judges affects them, reputation of the skater affects them, and even event affects them (GPF sp and worlds sp seem, to my eye, to always have deflated component scores).

My complaints about PCS are mainly directed towards Chan and Hanyu, namely the fact that they are awarded high 9s when they fall or stumble or have lots of room for improvement in the P/E and INT categories especially. Even though they are leagues ahead of the field, it does not make sense why (according to the PCS rubric), they are to be rewarded with high 9s instead of low 9s or high 8s when the rest of the competitors are earning high 7s and low 8s anyway so there would still be proper differentiation if they weren't so generously scored.

Also, PCS are so difficult to judge between competitions. Take Sochi 2014 for instance and the top 3 skaters. People were lambasting Sotnikova of course for earning 9s but the other two performances were not exactly perfect without room for improvement either and were still thrown out 9.75s etc for INT and were not hammered for lack of transitions even relative to other competitors.

 

[Mathman]

I'm confused as to what the difference is between a judge giving a skater 5.9 vs. an 8.75. Yes, IJS does give the impression of being more precise when it may not be, but it has also steered skaters towards more intricate programs and better separates programs of different content levels. Under 6.0 skaters could still get 5.8's with 5 clean triples, but 7 triples still wouldn't get you a 6.0 on technical merit. A skater could also get 5.9's with 7 triples and mediocre spins. With IJS, those differences are better reflected in the scores.

Forget the 5.8s and 5.9s. That was just window dressing. The old system was an ordinal system. At the end of the day it was th judges' responsibility to say, this skater was best, this one second best, etc.

This had an immediacy and intimacy that allowed the viewer, whether expert or naif, to wade right in. Great fun.

Now I find myself saying, well, if I care enough I guess I can wait until the protocols are posted and pore over the numbers. But mostly I don't -- the moment's gone.

 

[Alex D]

PCs are very subjective, I have a dance background and I believe, I try to express myself pretty well when I mention PC´s, especially in Ice Dance, but even with my background, you will find people who have a different idea of PC´s, as we saw with Adelina at Rostelecom. Personally, I try to make my point clear, argue as good as I can on the background that I have, but I do not expect that everyone will agree there, as honestly, it is an individual thing and nor am I right or wrong - it is subjective.

The whole TES, honestly - I do not have the bright knowledge of others there, like Mrs. P and the other lady who I sadly forgot the name now.... but I guess, even there you might find subjective influences?

What the ISU is giving us, are §§ and just like in law, you can work them in many ways.

That said, I agree with the general idea of this thread, but as mentioned, not even the ISU judges actually use the §§ for scoring as INT is not CH and TR is not SS and so on.

 

[Mathman]

I also wish that broadcast networks and commentators do not use Presentation Marks or Artistic Impression for PCS, implying "Artistry" as basis of PCS…

I agree. It would be easy for the commentators to say that the first mark is the total points for individual jumps, spins and footwork sequences, while the second mark is for the quality of skating sustained throughout the program.

That way people would not complain, when a skaters gets high marks, "What was so all-fired artistic about that?" The commentator could explain that what gave that skater a 90 in program components was speed, ice coverage, quality of basic stroking, steps and turns, and (my favorite ) mastery of the full skating vocabulary, together with effective presentation.

 

[moriel]

I agree. It would be easy for the commentators to say that the first mark is the total points for individual jumps, spins and footwork sequences, while the second mark is for the quality of skating sustained throughout the program.

That way people would not complain, when a skaters gets high marks, "What was so all-fired artistic about that?" The commentator could explain that what gave that skater a 90 in program components was speed, ice coverage, quality of basic stroking, steps and turns, and (my favorite ) mastery of the full skating vocabulary, together with effective presentation.

They still need to separate BV and GoE to make "total points for individual jumps, spins and footwork sequences, while the second mark is for the quality of skating sustained throughout the program" work =(
Wish it was like 3 scores, (1) sheer technical stuff - jumps, difficult positions etc, (2) skating quality , (3) and artistic impression, choreo etc.

 

[drivingmissdaisy]

They still need to separate BV and GoE to make "total points for individual jumps, spins and footwork sequences, while the second mark is for the quality of skating sustained throughout the program" work =(
Wish it was like 3 scores, (1) sheer technical stuff - jumps, difficult positions etc, (2) skating quality , (3) and artistic impression, choreo etc.

I actually like that each element is specifically judged for quality. It makes it easier to hold the panels (if not the individual judges) accountable for the marks given to each move. It has also helped me better understand how elements are evaluated. For example, I used to think a good jump was one that had good height and a clean, controlled landing. However, a good jump can be made "better" with a difficult entry, a difficult exit, or a varied air position. I look for these things when I watch programs now and it has helped me better appreciate skaters like Courtney Hicks, who I don't think would do well at all under 6.0 but has tremendous scoring potential under this judging system.

 

[moriel]

I actually like that each element is specifically judged for quality. It makes it easier to hold the panels (if not the individual judges) accountable for the marks given to each move. It has also helped me better understand how elements are evaluated. For example, I used to think a good jump was one that had good height and a clean, controlled landing. However, a good jump can be made "better" with a difficult entry, a difficult exit, or a varied air position. I look for these things when I watch programs now and it has helped me better appreciate skaters like Courtney Hicks, who I don't think would do well at all under 6.0 but has tremendous scoring potential under this judging system.

Oh, i dont even mean each element would stop being judges for quality... Something more like just giving 3 numbers instead of 2 and re-grouping them differently so there is no this weird PCs mix which is quality + artistry thing. So kinda we would understand it better... Like this guy has baaaaaaaad tech and fell half of his jumps, but other than that the quality of his skating is great, and he is also a very artistic one.

 

[gkelly]

I'm one of those people who has tried to learn more about the code so that I can make reasoned comments, especially as I have never skated. However, while learning about levels and GOEs is quite easy, PCS are not at all transparent and it is impossible to discern why they're awarded. Nationality of the judges affects them, reputation of the skater affects them, and even event affects them (GPF sp and worlds sp seem, to my eye, to always have deflated component scores).

PCs are very subjective, I have a dance background and I believe, I try to express myself pretty well when I mention PC´s, especially in Ice Dance, but even with my background, you will find people who have a different idea of PC´s, as we saw with Adelina at Rostelecom. Personally, I try to make my point clear, argue as good as I can on the background that I have, but I do not expect that everyone will agree there, as honestly, it is an individual thing and nor am I right or wrong - it is subjective.

I figure with PCS, there aren't right and wrong answers, just better and worse ones.

Better scores especially for Skating Skills and some aspects of Performance/Execution and Choreography would come from judges or fans who had a close up seat in the arena, decreasing with less advantageous seats and then a big decrease with watching on video vs. live, though the quality of the video would also make a difference.

Better scores from those who have more knowledge and experience with the rules, with relevant knowledge about skills, and with the range of quality expected between the weakest and the best possible skaters, especially with consistent benchmarks for what constitutes poor, weak, average, good, or superior within the skating field as a whole.

For P/E, CH, and Interpretation, fans with extensive performing arts background might have more relevant knowledge about many of the criteria than official judges without that background.

I think there's plenty of room for disagreement with each other and with the official judges -- I love conversations along the lines of "I would have given this skater Y score instead of X for this component, because..." with well thought out reasons. When we share our reasons, we can all learn from each other and bring more knowledge to our responses in the future. I love those kinds of conversations.

I think we should consider that those we disagree with might have had good reasons of their own for coming up with a different number. We can ask each other here. We can only guess what the judges on the panel were thinking. My inclination is always to give the benefit of the doubt -- maybe it looked different from their vantage point, or they were prioritizing different bullet points or bringing different outside knowledge to bear on their decisions. Not necessarily better or worse. If we could discuss specific scoring decisions with the judges probably we could learn from them and they might learn from us to consider some aspect that fits the written PCS guidelines and stood out more to us.

Scoring the elements is a lot more objective under IJS than under 6.0 judging.

PCS is still just as subjective -- but broken down somewhat to allow for showing distinctions where one skater is stronger in one group of skills and weaker in another. And there are also more explicit criteria we can point to in discussing why we think a given performance deserved a given score.

 

[Vernella]

This is a very fair criticism. However, given that skating is a judged sport, the panels are forced to assess athletes relative only to those skaters they have seen, because you can't judge someone against a skater who hasn't yet performed. If Mao skates among skaters who earns 6's, the 8's she earns are much stronger marks. When you get to the last group, where every skater has obvious strengths, their marks will be high and clustered together. What I'm saying is that I don't think it's intentional; I don't believe panels go out seeking to punish skaters merely because they skate early. Yes, fans try to use PCS discrepancies as bona fide evidence of cheating when their favorites lose, but a lot of the variation can be attributed to skate order. Because this effect is somewhat egalitarian, impacting every skater, and not intentional but rather a result of humans judging an event, I accept it.

But that's not all that happens... There are skaters not so strong in PCS that get rewarded only because they skate later... If Ashley doesn't have better SS than Mao her score should be lower even if she skates later, this is the problem. Of course being so objective it becomes possible to compare competitions is still too difficult.

Or we take away the PCS and openly replace it with a reputation score.

 

[YesWay]

There are skaters not so strong in PCS that get rewarded only because they skate later...

This is exactly the kind of unsubstantiated claim I wish to avoid.
People wheel out that tired old cliche/excuse, every time their favourite skater doesn't finish as high as they'd like.
Or the skater they hate, scored more than they'd like.

Skaters are in earlier groups because they are not high scorers, and/or they they have not scored highly in previous competitions.

Where is the proof that skaters automatically get lower PCS for skating in an earlier group?
I'm sure it happens... but I DON'T believe it happens as often as some people claim.
And how did Boyang Jin get such a great SP score this weekend, etc?

As per the OP - If you wish to make a case for a skater being under or overscored in PCS, my view is - you have to go into the protocols and point out where exactly his/her skating deserved more or less points vs a) the criteria for components, and b) the performances of other skaters at the same competition. Otherwise, your complaint is unsubstantiated, and becomes just another part of the "background noise"...

 

[YesWay]

I love conversations along the lines of "I would have given this skater Y score instead of X for this component, because..." with well thought out reasons. When we share our reasons, we can all learn from each other and bring more knowledge to our responses in the future. I love those kinds of conversations.

Yes, indeed. But the point of the OP is, those conversations are so few, vs. the number of opinions given with no reasoning or supporting argument/evidence... :-/

 

[Mrs. P]

This is exactly the kind of unsubstantiated claim I wish to avoid.
People wheel out that tired old cliche/excuse, every time their favourite skater doesn't finish as high as they'd like.
Or the skater they hate, scored more than they'd like.

Skaters are in earlier groups because they are not high scorers, and/or they they have not scored highly in previous competitions.

Where is the proof that skaters automatically get lower PCS for skating in an earlier group?
I'm sure it happens... but I DON'T believe it happens as often as some people claim.
And how did Boyang Jin get such a great SP score this weekend, etc?

As per the OP - If you wish to make a case for a skater being under or overscored in PCS, my view is - you have to go into the protocols and point out where exactly his/her skating deserved more or less points vs a) the criteria for components, and b) the performances of other skaters at the same competition. Otherwise, your complaint is unsubstantiated, and becomes just another part of the "background noise"...

I tried to explore the PCS topic by starting this thread: http://www.goldenskate.com/forum/showthread.php?56694-Reputation-judging-or-threshold-judging

I feel as you do, the "reputation judging/earlier group/etc" argument is a tired one and I think it's worth seeing if perhaps there are other judging trends at play here. But it's very, very difficult. If you read the thread there's plenty of noise.

 

[cl2]

There are countless claims of under/over scoring on these boards.
But how many are made with a reasoned supporting argument?

What do I mean by a "reasoned argument"? As I see it, if people want to make a case for under/over scoring, it should be made in terms of:

• the protocols for the competition (tech calls included),
• the GoE bullet points that a skater should or should not have been awarded for each element, and
• the PCS criteria they met or didn't meet for each component.

...Or they at least need to have a reputation for being able to do so in those terms, if called upon.

Without that - it seems to me a complainer doesn't actually know what they are complaining about. They may be actually be right, but since their complaint is unsubstantiated - it's more likely just wishful thinking, bias, favouritism, hot air, sour grapes, etc... and I can't take it seriously.

Unfortunately, the number of times I see a "reasoned argument" is very, very small.

And the few that are posted, may be buried under large numbers of "unsubstantiated" posts. The sort where people think that PCS stands for "Popularity Contest Score" ie. the points a skater "deserves", because "I like that performance". Or that skaters were "held back" or "boosted" by being in an earlier or later group, by having a particular nationality/fed/reputation, by competing at a certain location, etc etc... as if these things are all givens (which they aren't), and they happen at every competition (which they don't).

And I wish this would change... more "reasoned argument" posts would be educational for all... help people understand scoring better generally... understand when scores are actually reasonable, even if they seem too high or too low at first... vs when there really IS a reason to pick up the pitchforks and burning torches and form an angry mob. Because as it is, it feels like it's "pitchforks and burning torches" for every... single... competition. I see no proper justification, which makes it very tiresome.

Is there any hope?

I'd love to see "official GS" reviews of scoring, that examine the protocols in these terms, and indicate how reasonable/justifiable the scoring was... I know there are a few GS posters who (I think) have the knowledge/experience required to do such a thing, because I have occasionally seen posts like that... but I guess it would take quite some time and effort to do it on a regular basis, so perhaps not feasible?

Is there somewhere else to turn for "reasoned arguments" or assessments? I have been impressed in the past by Yukiko Okabe's commentary on skating competitions - she is an ISU judge and technical controller, and her insights into a skater's performance vs what judges are looking for... is valuable, very educational and sometimes surprising. I wish that type of broadcast were made more frequently, and more accessible. I feel it helped my understanding of scoring immensely. But the only ones I've seen were made months after the event, and only on Japanese TV in Japanese language.

Are there other FS commentators who are equally "authentic" or knowledegable, impartial, and provide that level of detail in commentary? Or bloggers, or whatever? Chris Howarth (an active FS coach) on British Eurosport seems quite good (unlike Simon and Nicky, bless :-D) but he's not there for every competition, and doesn't always go into a lot of detail, presumably in the interests of keeping things light and entertaining and unintrusive?

I just now came in to read this thread, and applaud the OP for raising a grievance I've been having for some time. And the irony of the first reply is that it's exactly the kind of unsubstantiated claim that the OP was referring to!

I generally don't have a problem with scoring anymore, mostly because I accept that there are a few things aside from what the skater does on the ice that affect the scores. In particular, skating early always negatively impacts the score, particularly on PCS. Posters here often say "I don't understand why skater X had 2 points higher PCS than skater Y in one event, then 2 points lower in this other event." Well, it's usually because the skater who skated much earlier than the other didn't get as high PCS. Even today, people wonder how Ashley can have better SS than Mao. She doesn't. She just skated later. People need to accept that scores between different events cannot be compared for this reason.

With due respect, just to explain my grievance about the nature of the first reply to the OP, a good substantiation of an observation is typically *not* "X and Y always happen simultaneously, therefore judges are biased". Rather, a good analysis almost always are more nuanced, e.g. "X and Y always happen simultaneously, but there're hidden factors A, B and C that we should consider as well; but I devised a controlled test to show that A and B are not correlated with X, while on the other hand, being better at C arguably causes higher X and Y, so with our current information we cannot reject the hypothesis that judges are fair."

 

[Mathman]

... a good analysis almost always are more nuanced, e.g. "X and Y always happen simultaneously, but there're hidden factors A, B and C that we should consider as well; but I devised a controlled test to show that A and B are not correlated with X, while on the other hand, being better at C arguably causes higher X and Y, so with our current information we cannot reject the hypothesis that judges are fair."

This would be so much more fruitful and easier to do if the judge's scores were not anonymous and randomized.

 

[cl2]

This would be so much more fruitful and easier to do if the judge's scores were not anonymous and randomized.

No doubt we as outside observers have less information than, say, the "insiders". But let us not delude ourselves as to the quality or certainty of our speculations, especially knowing that it's drawn from imperfect information.

 

[Alex D]

I tried to explore the PCS topic by starting this thread: http://www.goldenskate.com/forum/showthread.php?56694-Reputation-judging-or-threshold-judging

I feel as you do, the "reputation judging/earlier group/etc" argument is a tired one and I think it's worth seeing if perhaps there are other judging trends at play here. But it's very, very difficult. If you read the thread there's plenty of noise.

I still believe it is an issue.

Look at Hongo CoC. This girl got 0.60 less for Choreography and INT, while her program was better than of Mao Asada who skated 12th and last. Sure it was only one number, but it is so typical for the current scoring system. You have a huge advantage going last or having a big name.

What Hongo did at COC, the way her elements were always on beat, the Irish dance steps, her expression and the open arms - what justified the 0.60? SS I agree, MAO is just better, she has deeper edges, she is faster and smoother in her transitions (even though she can do better than at COC) - but CH and Int, sorry - to me it was not fair judged and that I am still so angry about it, oh man I hate being angry

It seemed like the judges just gave her automatically 0.50 more than Hongo, without actually analyzing the programs.

 

[gkelly]

Should several of us try to analyze the programs?

Maybe some of us will see differences in Asada's favor that you didn't notice.

Maybe we can speculate about advantages she may have had in those areas that aren't as obvious on video as in person.

Maybe you can explain in detail why you thought Hongo's program was better.

We can each agree that we might miss something or prioritize some criteria differently than each other or than the judges.

Maybe some of the judges thought Hongo was better in those areas but the way the averaging worked out Asada came out ahead.

Maybe some judges did miss some important qualities that we can see because we're better at analyzing musical interpretation and all the Choreography criteria that do come across on video than they are.

We can still respect the process while disagreeing with specific results.

Or we can just assume that our own assessments are the only right ones and anyone who disagrees with us, official or unofficial, must be wrong. But if fans disagree with each other about specific programs, does that mean we have to question each other's honesty?

 

[Perdita]

This is how ISU explains PCS:
http://static.isu.org/media/104183/program-component-explanations.pdf
And how they translate into score:
http://static.isu.org/media/146428/program-components-overview-2014.pdf
And some FAQ about the differences with 6.0 system: (it looks very useful but I haven't read it all...:hopelessness
http://static.isu.org/media/129337/new-judging-system-faq.pdf

Every argument on PCS should be based on these ISU criteria with reference to each criterion, not what each fan thinks as PCS. That will make a solid, constructive discussion, I believe (I suppose that the judges have special occasions to learn how to look at each criterion, so we fans should not assume to have 'the judge's eye view' even with these references though)

And not to mention that the judges don't agree with each other anyway!

 

[Mathman]

Every argument on PCS should be based on these ISU criteria with reference to each criterion, not what each fan thinks as PCS. That will make a solid, constructive discussion, I believe (I suppose that the judges have special occasions to learn how to look at each criterion, so we fans should not assume to have 'the judge's eye view' even with these references though)...

Still, I think there is also room at the table for fans who just want to say, "Oh that performance was so beautiful and sublime; it touched my heart; she hit every note of the music; she deserves a 10 in PCSs and I don't care what some old ISU document says!" :yes: