2017_18 ISU Judging Anomalies

[icybear]

I don't think it's as egregious as the Chinese judge. I'm assuming for Vincent he was marked lower on PCS than Fernandez/Hanyu/etc. (but not to the degree that people expect) and surpassed them on TES.

Zhou's BV was 18 points higher than Hanyu and 26 points higher than Fernandez.... and that partially contributed to that judge scoring him higher, obviously that judge marked Hanyu/Fernandez a bit more conservatively on GOE and Zhou a bit more generously. It was out of line judging, but given national bias and the fact that Zhou had much higher tech content than Hanyu/Fernandez it's hardly unexpected.

Also, a judge giving a +2 instead of a +1 on a quad/3A means 1 more point added to that judges "individual score". So if they add 1 or more extra GOE to each of their skater's elements, that adds up. For example, on the 4Z+3T, the US judge gave 2.4 points of GOE versus someone like judge 6 giving 0 points of GOE. For the 4Tx, judge 2 gave 2.2 points of GOE versus judge 6 giving 0 points.

At least when judge 2 was an anomaly they might be 1 GOE extra. Compare that to judge 7 for Jin -- giving +3s across the board and a slew of 9.50's. Like, that Chinese judge didn't care - if Jin stayed on his feet, it was a +3.

So ISU are investigating this when all that was affected was a difference of 4th or 5th place which is trivial. I guess if you're not an ISU protected country a touch of berserk judging will have you criminalized whereas if you're protected you could steal a gold medal and have a million signature petition protesting it and the ISU will turn a blind eye. If they want to investigate something, investigate how Uno quad loop wasnt called for when it was clearly under rotated and costed Fernandez a sliver medal rather than investigating into the scoring of a skater who received nothing.

 

[Metis]

Yeah, me too. That's why I hope they consider it wholistically -- Chen was way, way overmarked on PCS for SP and LP, so Jin should've finished 4th even without the Chinese judge.

Jin’s 4Lz GOEs make me hope the Rink Monster is Jabba the Hutt, that he’s napping underneath the judging area, and that he wakes up for a snack during the short programs tonight.

 

[chopinskate]

So ISU are investigating this when all that was affected was a difference of 4th or 5th place which is trivial. I guess if you're not an ISU protected country a touch of berserk judging will have you criminalized whereas if you're protected you could steal a gold medal and have a million signature petition protesting it and the ISU will turn a blind eye. If they want to investigate something, investigate how Uno quad loop wasnt called for when it was clearly under rotated and costed Fernandez a sliver medal rather than investigating into the scoring of a skater who received nothing.

Oh, but it wasn't within the 30 minute time limit!

This is totally within the 30 minute time limit! That's just another name for three days, right?

 

[icybear]

Oh, but it wasn't within the 30 minute time limit!

This is totally within the 30 minute time limit! That's just another name for three days, right?

They've just changed the rule yesterday.

 

[CanadianSkaterGuy]

Yeah, me too. That's why I hope they consider it wholistically -- Chen was way, way overmarked on PCS for SP and LP, so Jin should've finished 4th even without the Chinese judge.

All the guys were way overmarked, TBH (with IMO some exceptions who were underscored/correctly scored like Uno, Zhou, Rizzo, Vasiljevs, Chan). But Jin's scoring by the Chinese judge was the most egregious example of overscoring from the men's competition, bar none.

 

[chopinskate]

But Jin's scoring by the Chinese judge was the most egregious example of overscoring from the men's competition, bar none.

True.

(Uno's jump GOEs are a discussion for another time, as is Chen's PCS )

 

[Metis]

All the guys were way overmarked, TBH (with IMO some exceptions who were underscored/correctly scored like Uno, Zhou, Rizzo, Vasiljevs, Chan). But Jin's scoring by the Chinese judge was the most egregious example of overscoring from the men's competition, bar none.

America may be too lazy to be a hegemon these days but we can still give China a run for its money in “inflating our designated medal-winner’s scores.”

Boyang somehow ended up undermarked overall while Chen did a variation on Plushenko’s Skating From Jump to Jump: NOW WITH MORE CROSSOVERS. (The spins were good, though.)

 

[CanadianSkaterGuy]

So ISU are investigating this when all that was affected was a difference of 4th or 5th place which is trivial. I guess if you're not an ISU protected country a touch of berserk judging will have you criminalized whereas if you're protected you could steal a gold medal and have a million signature petition protesting it and the ISU will turn a blind eye. If they want to investigate something, investigate how Uno quad loop wasnt called for when it was clearly under rotated and costed Fernandez a sliver medal rather than investigating into the scoring of a skater who received nothing.

Well, arguably, Fernandez's lip was ignored, which could have adversely affected his tech score too. Also, there are some liberties he received (I mean, a +3 for a 2S that was supposed to be a quad... ) +3 on his opening quad which was a tad wild on the landing was a bit generous too. Not that Shoma himself didn't get some leeway with the GOE scoring of his landings. All I'm saying is, both are top guys... if you point out the errors of one, you can point it out for the other. And Javier's goal was to medal, which he did. Uno also did 3 quads to Javier's and beat him by almost 10 points on TES. Not to mention, I thought PCS wise Shoma and Javier should have been a bit closer together.

I think the silver was deserved.

 

[chopinskate]

Boyang somehow ended up undermarked overall while Chen did a variation on Plushenko’s Skating From Jump to Jump: NOW WITH MORE CROSSOVERS. (The spins were good, though.)

Regarding his PCS, I think Boyang was undermarked given what Nathan's score was there. If Nathan had been marked fairly, I'd probably knock off a full mark on transitions and composition for Boyang.

Comparing the jump GOEs for the Top 5, on the other hand, would be quite a revelation.

 

[CanadianSkaterGuy]

America may be too lazy to be a hegemon these days but we can still give China a run for its money in “inflating our designated medal-winner’s scores.”

Boyang somehow ended up undermarked overall while Chen did a variation on Plushenko’s Skating From Jump to Jump: NOW WITH MORE CROSSOVERS. (The spins were good, though.)

Chen did skate from jump to jump, but it's not like Jin had spectacular artistry himself. Not to mention, he fell. I think 86 PCS for him was totally suitable, and perhaps even a bit generous.

It's ironic with Jin though -- he's so used to his home scoring being lukewarm to inflating him (see: his CoC stints), that it's unorthodox to see a Chinese judge actually score him on the other end of the spectrum so blatantly biased in his favour.

 

[chopinskate]

Chen did skate from jump to jump, but it's not like Jin had spectacular artistry himself. Not to mention, he fell. I think 86 PCS for him was totally suitable, and perhaps even a bit generous.

Sure, but this highlights Nathan's SP PCS issue -- 42 PCS with all three jumping passes botched (little more than a point behind Boyang). Completely turned everything off after the 3A.

Nathan just has more backing from the judges.

 

[slider11]

Nathan Chen has extensive ballet training which is evident in his posture and arm and leg positions. His choreography was excellent in both programs. He's no Patrick Chan in terms of interpretation but he's getting there. I disagree that Nathan was overscored on components.

 

[chopinskate]

Nathan Chen has extensive ballet training which is evident in his posture and arm and leg positions. His choreography was excellent in both programs. He's no Patrick Chan in terms of interpretation but he's getting there. I disagree that Nathan was overscored on components.

This doesn't have anything to do with his performance on the night, though. :confused2:

Nathan has improved on PCS a lot. However, PCS is supposed to be judged for the event. In the event, Nathan wasn't performing too well. Of course, the judges retained their inertia with his PCS (though reducing it a little because of the falls in the SP/ tentativeness that even they were able to spot in LP + skating in group 2).

Funnily enough, 87-88 is an appropriate PCS for Nathan's best performance this season.

 

[chopinskate]

Nathan Chen has extensive ballet training which is evident in his posture and arm and leg positions. His choreography was excellent in both programs. He's no Patrick Chan in terms of interpretation but he's getting there. I disagree that Nathan was overscored on components.

Also, I disagree about his posture. The line he creates with his neck and his arms is not good. The arms themselves are held well, but he has a weird thing going on with his neck. Legs are fine, though.

 

[TryMeLater]

Well, they do, unfortunately, see each other’s marks, but let’s assume each judge is completely walled off from one another, and no one has shared their opinions or preferences regarding the skaters prior to judging. If you think your scores for a given skater may be high enough to be the marks that are thrown out but you aren’t certain of that outcome, you have two choices: roll the dice and hope your marks make the average (in which case someone either skated phenomenally or another judge just went for max possible GOEs and PCS inflation) or you could try to remove as much risk as possible and adjust your scores so they come in just high enough to avoid looking utterly crazy while also maximizing the chance your scores are the one removed as the high in the average. The latter is the safer move; even with peeking, you don’t know exactly what everyone else is doing, and despite your own inflation, you may still come in under the highest score if someone else adds 20+ points to a skater’s average PCS while you went with a more conservative 18.5. On the GOEs, you’re trying to make sure you give the max possible value whenever possible, along with the lowest deductions stipulated, and throwing in an extra one or two +3 GOE scores makes it likelier a scoresheet that is quite generous goes into the average (mandatory deductions only, highest positive GOEs that the skater could have pretended to earn).

Why is it optimal? Because if you dice-roll but you make the average, the higher scores removed from the average might have been more advantageous to keep (say, +2-5 points higher in total), but because you played it “safe,” you removed marks that could have easily passed as legit, and you brought down the average. (This is the nightmare when scores are very close for the top skaters; if you wanted a given skater to win, losing any points is minimally sub-optimal and potentially changes podium order.) On the other hand, if you deliberately go for the highest marks you can “get away” with (or even not get away with — who cares about perceptions of bias?), you can predict to a much greater degree of certainty what you’d need to mark down to get your scores thrown as the high. (Do you know the skater’s SB and PB? Congrats, you’re basically done.) And if they somehow aren’t the high? (This is the case for not inflating to the point of lunacy; this is also why none of this takes coordination between two or more parties, just common sense by one.) You achieved the best possible outcome. A winner is... uh, not “you,” our hypothetical judge, but the skater they favor.

This strategy is obviously not optimal in an environment in which each judge trusts one another to score each element appropriately and according to the rules, but there’s a lot going on, including humans being frankly terrible with anchor numbers (the 7.7 problem when 5 is “average”), positive GOEs being awarded for basic execution rather than demonstration of mastery, etc. As an individual, you can’t control the average, but by deliberately removing yourself from it, you can “uncap” the high end of the score distribution, which is huge. Submitting a set of more conservative and honest-with-some-inflation scores risks making a set of far less generous scores the discarded high, which is why it’s “optimal” (from a purely rationalist standpoint) to adjust your marks in such a way that you can be reasonably certain that yours are now the highest in the set, as that preserves all scores that fall closer to your end of the distribution for the average. (You could also try to be the “low” in the average if you suspect the scores are going to cluster toward that end; in that situation, trying to “stop the bleeding” by preserving all scores above yours for the average may do more for the overall outcome if the highest score is going to be a true outlier due to a horrific skate.)

You’re right that this is hard to manipulate, at least in theory, due to what game theory calls “private information” and issues of “concealed preference.” But I’m not talking about “guaranteeing” an outcome — just what the most rational, optimal choice is for an individual judge given how the system is currently implemented. (Rational as in “rational choice,” as in game theory; the whole system is utterly irrational on a macro level.) There’s also the fact that judges aren’t exactly closed books (they have left plenty of data behind in the form of prior scores), that the composition of the judging panel is known to the judges (including nationality, which an economist already did an analysis of in terms of how it effects judges from the same country as a given skater), and scoring itself is an “iterated game” — the short and the free programs take place over two days. Even if a given judge knew nothing about anyone else on the panel, it only takes the first flight or two to figure out where the group naturally averages, who the sticklers are, etc. By the free skate, judges have a ludicrous amount of data just from observation (seeing how scoring went in the short).

Again: this isn’t about guaranteeing podium order, judges conspiring with one another, or anything so involved. (Though it doesn’t take a genius for the average judge to figure out that some of their fellows are going to undermark certain skaters just by observation; if you’re the only person giving a +1 GOE to Hanyu’s 3A, you really aren’t even trying to blend in.) It’s about making the most of the terrible system we’re stuck with and the fact that improper use of GOEs and PCS scores as a whole have made PCS marks functionally ordinal rankings (would love to test to see if I could find various correlations but yeah). Because GOEs aren’t being assigned uniformly across competitions or even in competitions (let alone accurately per ISU’s own rules), we can all read the scores and see the inflation and general “what fresh hell is this” in the numbers... as a judge, it would be irrational to go by the book unless you actually known the other judges and have solid cause to believe the majority are going to assign points as the system intended. Absent that, if you think a skater deserves to win... you could give them the score they earned, or you could all but ensure your score is dropped from the average, leaving space for all the other ridiculous values to go in.

As a US resident, I half want to believe that’s what the American judge was doing with Chen’s scores, but let’s be honest: “rational choice,” “probabilistic forecasts,” “Pareto equilibrium” — these are not words found in the language of far too many of our people. The guy was likely throwing darts at a chalkboard, though on Chen’s score, those darts landed exactly where he needed them to... it’s either proof of concept or a broken clock being right.

As a final note on optimization, if every single judge goes for “try to be high enough to be thrown out of the average without being conspicuous” for their preferred skaters, I wonder what happens to PCS values.... My point being: if enough judges are rational actors, a +20 PCS jump from one competition to another may not seem quite so insane. I don’t think that’s the whole explanation, but it’s a hypothesis that has yet to be disproven, and it may well account for some of the variance. It’s a thought!



Well, yeah, this is what would fall under “irrational” and “not optimal play.” The point of inflating a score to deliberately be the highest in the average is to do so to preserve higher marks behind yours, not to signal LOOK AT ME. He could have been attempting optimization (based on his history: LOLNO) and this is what happens when your read on the judging panel’s average is way off, but the hilarious level of PCS inflation is a dead giveaway that no real thought was put into those marks. Though, ironically, they probably don’t rise to my “top 5 worst” from this OWG so far.

Rizzo’s PCS marks in the team event free skate were truly a thing of wonder when he skated immediately after Kolyada (and he and Tanaka wound up with an identical PCS score). I actually don’t have an issue with some of Kolyada’s PCS values, but the transition marks were the classic “7.7” issue.

Now, that is a long post.
I do agree with your hypothesis - however i'm not sure that you can assume independence in the sense you have written.
Judges have meeting prior to an event, so they are biased towards something. I think that this kind of thinking by a judge will take time, and they don't have enough time to process it correctly.

 

[miso]

Also, I disagree about his posture. The line he creates with his neck and his arms is not good. The arms themselves are held well, but he has a weird thing going on with his neck. Legs are fine, though.

This. In time, hopefully someone corrects this. And his artistry.

 

[gkelly]

Metis:

I guess the question is, what do international judges see their task?

1) The primary goal is to evaluate the skating according to skating criteria as accurately as possible (with the caveat that all humans are biased and individual judges' unconscious biases will affect their evaluations even if they do their very best to be impartial).

2) The primary goal is to evaluate the skating according to skating criteria as accurately as possible. However, there is also an additional goal of boost the results of the home country skaters. To achieve this secondary goal, it is often necessary for judges to make exceptions to the primary goal of accurate evalution. They may always giving the favored skaters benefit of every doubt, always rounding up borderline scores, and occasionally giving them one score higher than they deserve according to honest evaluation; they will do the opposite for the favored skaters' rivals; they may make deals with other judges to favor that country's skaters and disfavor those skaters rivals as well.

3) The primary goal of each judge is to achieve the highest possible placement for their country's skaters. The "game" is not for skaters to compete to be judged the best, but for judges to compete to manipulate the placements best. The competitors are the judges and the skaters are playing pieces. The expertise that judges need to win this game is intimate knowledge of game theory and strategy, with only just enough knowledge of skating criteria to maintain a fiction that they are judging the skating.

Are referees and the technical committee and assessment commission also players in the game, or are they the officials that regulate the judges' and maybe tech panels' moves? Do the players need to worry about these higher arbiters?

Is winning defined by amassing the most medals for the home team? In that case, getting the best skaters as playing pieces would help significantly. Should the federations be considered the primarily players, applying strategy to develop the best skaters and send them to events as well as developing/choosing judges who are the best players of the scoring game?

Or do judges win the game by boosting their skaters the maximum number of places above what they actually deserve, in which cases there is more room for success the weaker the skaters actually are?

 

[Nikidom]

LOL judges panel think that Med jumps and spins better than Zagitova in OG SP, almost all GOE higher than Zags, thats real anomaly to bring Med to gold no matter what it cost.

 

[chopinskate]

LOL in what universe is Med's StSq on the same level as Carolina Kostner's? They both got a perfect 6.0.

And Med outscored Satoko there.

 

[chopinskate]

You know something's up when even BuzzFeed cottons on.