Mona Adolfsen on being an ISU Judge and the scoring system

In the run-up to the biggest figure skating event of the season, the 2019 World Championships, the International Skating Union (ISU) judge and technical controller Mona Adolfsen (President of the Norwegian Skating Association) tells about her motivation to work as an ISU judge as well as the peculiarities of the new ISU judging system.

At the 2017 World Figure Skating Championships, held in Helsinki, Adolfsen (Judge No. 8) awarded Evgenia Medvedeva, the first Russian woman ever to successfully defend her world title, a perfect score of 10.00 in "Performance" as part of the component score. She was the only lady in the competition to receive 10.00s at the event.

Since then, some significant changes to the ISU judging system have been implemented. The biggest changes that are in effect from this season are to the way that elements are now evaluated. For example, the Grade of Execution (GOE) has been changed to a minus-5 to plus-5 range. In addition, a 10 percent bonus is added to the base value of a jump when performed after the halfway point of a program (for the last jump element in a short program and for three jump elements in free skating).

“Very often it takes some time before we get the changes completely 'under the skin,'” said Adolfsen. "With the changes going from +/- 3 to +/- 5, the differences in each individual element can be greater. It will be an advantage, especially when the figure skaters make several mistakes in a single element. At the same time, we see that the differences sometimes become greater between the scores awarded by the judges.”

With this new scale, differences in scores from judges can occur for various reasons.

“The explanation for such differences is that judges are not completely used to the new scale," Adolfsen pointed out. "There may be a different interpretation of the guidelines as well. One may press the wrong button when entering scores. Some judges may believe they have seen, for example, not a fully rotated jump or landing it on two feet, or other things. Sometimes one judge throughout the competition awards lower scores than all the other judges. At the same time, it must be said that if there was a requirement that everyone should award exactly the same score, then only one judge would be needed.”

Click to expand...

The ISU regulations say that in a program containing Falls or Serious errors, the score nine-fifty (9.5) or higher should not be awarded for Skating Skills, Transitions and Composition and the score nine (9.0) or higher should not be awarded for Performance and Interpretation. However, sometimes we see that judges do not follow this rule and award such scores for components in programs with falls and serious errors as seen below.

(Click to Enlarge)



“I do not know what lies behind the scores for the components when they are as high as they are in the programs performed with falls and obvious errors,” admitted Adolfsen. “We have now clear guidelines, but they have only become clearer this season. It is good for the sport. I wouldn’t like to comment on the way the judges award high scores in such situations, as it will be wrong of me as a judge. We shall not comment on such things in accordance with our code of ethics as well, but what I can say is that there is no kind of exception to whom it applies.”

Click to expand...

More...

What's your take?

Very interesting interview, thank you!

I’m surprised (in a positive way) to hear that judges get 4 day seminars on components (but when and how often?).

I’m sure some people will be glad to hear that obvious prerotation affects score and to have clarification that under rotation is judged by comparing position of toe pick at take off and landing.

Regarding too high components for programs with serious errors, I kind of wonder why the scoring system can’t be programmed to automatically flag it and tell the judge to enter a lower component score

Tavi... said:

Very interesting interview, thank you!

I’m surprised (in a positive way) to hear that judges get 4 day seminars on components (but when and how often?).

Click to expand...

Generally four. In 2018, the ISU held four for singles/pairs and a separate (fifth) one for ice dance.

I’m sure some people will be glad to hear that obvious prerotation affects score and to have clarification that under rotation is judged by comparing position of toe pick at take off and landing.

Regarding too high components for programs with serious errors, I kind of wonder why the scoring system can’t be programmed to automatically flag it and tell the judge to enter a lower component score

Click to expand...

That's a good idea. Like trying to withdraw too much money from an ATM

Very interesting to hear pre rotation and under rotation described by an actual judge.:agree:

As well as her take on the way PCS is awarded

I think if they want a mandatory deduction for a serious error, they need to have it applied like the fall deduction is right now (maybe just increase the fall deduction). Otherwise, it's just an invitation for it to be applied unevenly. Plus the rule as it currently stands interacts weirdly with high PCS versus lower PCS skaters scores. If a skater who typically scores 9.5 in components and a skater who typically scores 7.5 both fall, I don't see why the former skater should have a mandatory PCS deduction and the second should not.

I think the rule re 9.5 or 9.0 for certain components is open to misinterpretation.

As a native English speaker I read it (when read normally/quickly) as meaning if there's a fall or serious error then those are the values that apply. However a lawyer looking at the language might say they mean if there's more than one fall or serious error then those are the values that apply. Hence it could be that judge Adolfsen is confused (which is not a good thing), and that the values in the protocol are OK (or perhaps they're not - it's what the ISU meant).

Also there's a rule that if a program contains a fall or serious error then 10 shall not be awarded to any of the components, which would also suggest that the second rule is where there's more than one error, whereas the protocol example only has one serious error, a fall.

Finally the wording also says 'should' not be awarded rather than 'shall not be awarded' hence further reason for a judge not to follow the official guidelines.

Hopefully judge Adolfsen's comments plus others will reach the ISU, and wording will be tightened so that there's no grounds for misinterpretation.

Re judges entering values that should not be possible, it should be possible to tweak the ISU's computer program to avoid at least some of them. If there's a < or << or e or ! symbol against an element then there must be a max GOE that is possible, and it should be pretty easy to change the program to prevent a judge entering a value that's impossible. Similarly if a fall were marked against an element behind the scenes even if it doesn't appear on the official protocol against a particular element.

However for other serious errors it wouldn't be possible for the technical panel to say whether these are serious or not, it would be down to the individual judges. However you could say that if a judge has given an element a -4 or -5 then by definition it must have had some sort of serious error. Hence you could tot these up plus any where there's a symbol (< or << or e, ! wouldn't count as a serious error) or fall behind the scenes, and if there's 2 or more then prevent components of 9.5 or above being entered for SS, TR and CO, and 9.0 for PE and IN.

I also thought the example in the article of judges misjudging Components was bad for the same reason. None of the marks in that protocol actually violate the PCS guidelines as written, and the official language reads very deliberately to me.

Glad to see that she clarified the UR standard, where the toepick touches the ice. I’m not sure the PR was really cleared up; should I read that as skate blade (for edge jumps)/toepick (for toe jumps)? I would tend to think turning on the blade for a toeloop is worse than turning on the pick.

Otherwise, I was disappointed that this piece was basically just a regurgitation of the rulebook rather than interesting insights into the judging process. What insights there were seemed shallow, and also very oddly worded, like the GOE assessment:

We assess the criteria and at the same time the criteria for the minus GOE,” said Adolfsen. “At the same time, we must make an overall assessment of a performance before we establish the GOE.”

Click to expand...

or her comments about PCS:

That means that a skater can get the score 7.50 in Transitions and 9.50 in Performance, but at the same time, it often has to be a particular spectrum performance where the skater gets a full score, for example, in Compositions.

Click to expand...

What exactly is this even supposed to mean?

She didn’t really address any questions I would have about judging and that I’ve seen fans comment on many, many times, like why we never actually see a skater getting a 7.50 for TR but then a 9.50 for PE, for instance.

Some feel that there may be a pre-formed opinion based on a skaters’ titles and that those titles may have an effect on scores awarded by judges.
”As it stands in our regulations, as well as we are reminded at the start of each competition, we shall judge regardless of how the skaters performed earlier, and it also applies to performances by skaters in the ongoing competition.”

Click to expand...

I mean, what was she even supposed to say to this question, really? But still. “I know people are concerned about judges breaking the rules, but the rules say they shouldn’t.”

Honestly, the general thrust of this article sort of gives me massive Fail vibes.

Miller said:

I think the rule re 9.5 or 9.0 for certain components is open to misinterpretation.

As a native English speaker I read it (when read normally/quickly) as meaning if there's a fall or serious error then those are the values that apply. However a lawyer looking at the language might say they mean if there's more than one fall or serious error then those are the values that apply.

Click to expand...

The notes in the Scale of Values for the 2018-19 season spell out the difference quite clearly: for a program containing one fall or serious error, judges are not permitted to award a score of 10.00 for any Component, and for a program containing two or more falls or serious errors, they are not permitted to award a score of higher than 9.50 for SS, TR and CO, or higher than 9.00 for PE and IN. This applies for singles and pairs; I can't remember what the rule is for dance. The protocol used as an example only has one fall/serious error noted on it (the fall leading to missed combination on the 4T), so if the fall/missed was treated as a single serious error rather than two (one fall and one missed combo, which doesn't seem to be a good idea as it's impossible to both fall on the first jump and complete the combo) the higher cap on the component scores was applicable and the range of marks seems to be fine. If it had combined a fall on the 4F with the (fall) 4T+COMBO notation, they'd have been restricted to scores within the lower cap.

Judge Adolfsen might have misspoken on this point in the interview as a result of a misunderstanding, or just because technical conversations in second languages are always hard! But it was a very interesting interview nonetheless. :thumbsup:

Here is another protocol (with more than one error) that might be easier to avoid misinterpretation.

“If it is very clear that the rotation is done on the ice before the take-off, the jump is downgraded and evaluated using the scale of values for the jump of one rotation less,” Adolfsen explained. “The position of the toe pick/skating blade on the take-off is considered here, not the body position as some may believe. The position of the blade where the toe pick hits the ice is considered to assess whether the jump is under-rotated or not.”

“We cannot see the jumps in slow motion, otherwise there will soon be many jumps to be considered and may be downgraded,” she continued. "Again, there are small margins, and here it has been decided that it must be visible in normal speed to consider whether the jump is pre-rotated or not. The rule for pre-rotations is not as strict as for landing of jumps as it is physically impossible to not have any pre-rotation.”

Not that this will satisfy everybody who constantly gripes about pre-rotation, but here's the pertinent part.

CanadianSkaterGuy said:

“If it is very clear that the rotation is done on the ice before the take-off, the jump is downgraded and evaluated using the scale of values for the jump of one rotation less,” Adolfsen explained. “The position of the toe pick/skating blade on the take-off is considered here, not the body position as some may believe. The position of the blade where the toe pick hits the ice is considered to assess whether the jump is under-rotated or not.”

“We cannot see the jumps in slow motion, otherwise there will soon be many jumps to be considered and may be downgraded,” she continued. "Again, there are small margins, and here it has been decided that it must be visible in normal speed to consider whether the jump is pre-rotated or not. The rule for pre-rotations is not as strict as for landing of jumps as it is physically impossible to not have any pre-rotation.”

Not that this will satisfy everybody who constantly gripes about pre-rotation, but here's the pertinent part.

Click to expand...

Funny, I can usually see it in real time. I must have eagle eyes :reye:

Harriet said:

The notes in the Scale of Values for the 2018-19 season spell out the difference quite clearly: for a program containing one fall or serious error, judges are not permitted to award a score of 10.00 for any Component, and for a program containing two or more falls or serious errors, they are not permitted to award a score of higher than 9.50 for SS, TR and CO, or higher than 9.00 for PE and IN.

Click to expand...

Note that this is inclusive... so for 2 or more serious errors (pop/fall) the SS/TR/CO cannot be 9.50, 9.75, 10.00; and the PE/IN cannot be 9.00, 9.25, 9.50, 9.75, or 10.00.

gsk8 said:

Here is another protocol (with more than one error) that might be easier to avoid misinterpretation.

View attachment 559

Click to expand...

Yes, because this skater had 2 major errors (a fall, and a pop), the majority of these judges scored this particular skater incorrectly/too high according to the aforementioned rules:

Judge 1: PE 9.00/IN 9.00 (8.75 is the max)
Judge 2 (the worst offender - overscoring every component beyond the max): SS 9.50/TR 9.50/CO 9.75 (9.25 is the max)/PE 9.25/IN 9.50
Judge 3: appropriately scored
Judge 4: SS 9.50/PE 9.25/IN 9.25
Judge 5: appropriately scored
Judge 6: PE 9.25/IN 9.00
Judge 7: IN 9.00
Judge 8: PE 9.50/IN 9.00
Judge 9: appropriately scored

If these judges who overscored had their PCS reduced to the maximum allowed, the skater would have received 9.21 for SS (instead of 9.25); the same TR of 9.00 (even if the 9.50 was dropped to 9.25 it's thrown out anyways), 8.71 for PE (instead of 8.93), the same CO of 9.18 (even if the 9.75 was dropped to the max of 9.25), and 8.72 for IN (instead of 8.96).

Enforcing the max PCS rules, this skater's total max PCS should have been 89.64, instead of 90.64.

narcissa said:

Funny, I can usually see it in real time. I must have eagle eyes :reye:

Click to expand...

Well, then join the ISU and become their most proficient tech specialist!

“If it is very clear that the rotation is done on the ice before the take-off, the jump is downgraded and evaluated using the scale of values for the jump of one rotation less,” Adolfsen explained. “The position of the toe pick/skating blade on the take-off is considered here, not the body position as some may believe. The position of the blade where the toe pick hits the ice is considered to assess whether the jump is under-rotated or not.”

“We cannot see the jumps in slow motion, otherwise there will soon be many jumps to be considered and may be downgraded,” she continued. "Again, there are small margins, and here it has been decided that it must be visible in normal speed to consider whether the jump is pre-rotated or not. The rule for pre-rotations is not as strict as for landing of jumps as it is physically impossible to not have any pre-rotation.”

Click to expand...

On the other hand, if we bold these parts, she seems to be clearly claiming:

1. Excessive prerotation is an error (though obviously some amount of prerotation is allowed, a fact that no one disputes)
2. UR is measured by when the toepick hits the ice, not when the rest of the blade does. (Relevant to some discussions of whether so-and-so URed at such-and-such event.)
3. If there were more slow motion replay, there would most likely be more tech calls. Ergo, some skaters are likely getting away with not fully rotating their jumps. (The way it's worded makes it sound like the ISU wants to avoid increasing the amount of tech calls and so that's why they're avoiding more slow motion replay? Tbh, it makes it sound like cheated jumps are such a rampant problem that the ISU doesn't want to address it for fear of ruining FS's legitimacy as a sport.)

I would like to echo Narcissa that it's not clear what "visible at normal speed means", because different people are better or worse at spotting pre and under rotation. I mean, on the low end of visual recognition ability, people who don't watch figure skating regularly can't even distinguish jumps from each other at normal speed, whereas on the higher end, some people are able to call URs fairly accurately at normal speed, so I see no reason they couldn't call PR as well, if they were to train themselves to do so. Plus, your recognition ability also depends on factors like where your focus is, how awake you are, and so on, so it seems a very unclear standard. Nor is it clear why the fact that a skater prerotates faster ought to get away with prerotating versus skaters who prerotate slower. After all, the problem is not really the speed at which you do it--it's whether you achieve a certain amount of rotation in the air. It's unclear why someone who only manages 810 degrees in the air should get the same credit as someone who manages 1000, just like it would be unclear why someone who runs only 90m should get the same credit as someone who runs 100.

To take a non competitive example to illustrate my point, I recently saw someone jump a "single toeloop" with 270 degrees pre-rotation. Now, it was really obvious he was prerotating, because he rotated slowly, but if he had managed to do the prerotation faster would you credit him with a correct toeloop, considering he only rotated a quarter turn in the air?

Shanshani said:

On the other hand, if we bold these parts, she seems to be clearly claiming:

1. Excessive prerotation is an error (though obviously some amount of prerotation is allowed, a fact that no one disputes)
2. UR is measured by when the toepick hits the ice, not when the rest of the blade does. (Relevant to some discussions of whether so-and-so URed at such-and-such event.)
3. If there were more slow motion replay, there would most likely be more tech calls. Ergo, some skaters are likely getting away with not fully rotating their jumps. (The way it's worded makes it sound like the ISU wants to avoid increasing the amount of tech calls and so that's why they're avoiding more slow motion replay? Tbh, it makes it sound like cheated jumps are such a rampant problem that the ISU doesn't want to address it for fear of ruining FS's legitimacy as a sport.)

I would like to echo Narcissa that it's not clear what "visible at normal speed means", because different people are better or worse at spotting pre and under rotation. I mean, on the low end of visual recognition ability, people who don't watch figure skating regularly can't even distinguish jumps from each other at normal speed, whereas on the higher end, some people are able to call URs fairly accurately at normal speed, so I see no reason they couldn't call PR as well, if they were to train themselves to do so. Plus, your recognition ability also depends on factors like where your focus is, how awake you are, and so on, so it seems a very unclear standard. Nor is it clear why the fact that a skater prerotates faster ought to get away with prerotating versus skaters who prerotate slower. After all, the problem is not really the speed at which you do it--it's whether you achieve a certain amount of rotation in the air. It's unclear why someone who only manages 810 degrees in the air should get the same credit as someone who manages 1000, just like it would be unclear why someone who runs only 90m should get the same credit as someone who runs 100.

To take a non competitive example to illustrate my point, I recently saw someone jump a "single toeloop" with 270 degrees pre-rotation. Now, it was really obvious he was prerotating, because he rotated slowly, but if he had managed to do the prerotation faster would you credit him with a correct toeloop, considering he only rotated a quarter turn in the air?

Click to expand...

I understand the quote that they can´t do slow motion on one jump deciding if it will be downgraded or not, because then they probably have to watch over almost all the other jumps, at least the ones that questionable, and that could really vary from skater to skater. And that would be very time consuming.

Getting judges scores is already taking time and they have to be on time not to delay the whole event. A lot of competetion allready start later then planned. So, as far as I understand, they have the rules so that it´s fair for all skaters. Let´s say that each skater has about a certain amount of time doing their program and getting the scores before the next skater starts. Would it be fair if a skater has a lot of questionable jumps in a FS, but they only have only time to rewatch say like 3 of the jumps? Two of them get downgraded after watching the slow motion, the last doesn´t. The are about 3 more jumps that could be checked, but they don´t have time. So they just check 3, downgraded 2 and the rest, we don´t know. They give the scores. So the next skater skates almost perfect, but has 3 jumps that needs to be checked. All of them get downgraded, but there were no other questionable jumps, everything else was super clean. So the last skater made 3 small mistakes, get downgraded to all of them, the other skater had a lot of more questionable jumps, but due to time, they just had time to check 3, but there could be more. Would that be fair to the other skater?

If you are going to use slow motion to downgrade jumps, you have to be consistent and do it every time, by every skater where they are some sort of question. Some skaters have troubles with all the jumps, imagine novice skaters, they are new to this, or skaters who are very nervous and have a really bad skate, and all jumps in the FS has to be checked. It would take forever! This practice would lead to more delays, some events are packed with diciplines from start to end, doing both novice, juniors and skaters at one day, there are barely time between ice resurfing before the next competitions. It would also not be fair to the one skating next, having to wait extra long to go on the ice, because they use so much time on watching the skater before in slow motion.

It´s not that can´t do it, it´s just too time consuming if they are going to be fair. It won´t be fair if they do slow motion, but only have time to check 2-3 jumps, that would benefit skaters who have a lot more mistakes, and not the ones having almost none. And I think most people would agree that we don´t want them checking everything due to waiting.

Yeah, unfortunately the technology doesn't exist and judges aren't about to pull out their protractors. I think Mona outlines that there is essentially leeway, and they only apply a downgrade to an obvious PR. Also, nowhere does she mention that the PR is taken into consideration when assessing whether the landing is UR or not. So a skater could have zero prerotation and a 90 degree turn on landing and still get a UR compared to a skater with 150 degrees pre-rotation and a landing with a 45 degrees turn (which is within a quarter turn).

What I would be curious to know is whether she considers the takeoff on toe jumps based on where the pick is inserted, or where the pick leaves the ice... because a toe axel clearly starts with the skater inserting their pick facing forwards, whereas some toe jumps involve the pick pre-rotating on the ice before the skater goes up, which could be fine based on the leniency (and it sounds like judges are pretty lenient about that because if they started legit scrutinizing skaters for pre-rotations then you'd get tons of downgrades, especially in the ladies/pairs event where PR is more common).

Miller said:

However for other serious errors it wouldn't be possible for the technical panel to say whether these are serious or not, it would be down to the individual judges. However you could say that if a judge has given an element a -4 or -5 then by definition it must have had some sort of serious error.

Click to expand...

In practice, that is going to be true most of the time.

In theory, though, it's entirely possible for a judge to end up with a score of -4 or -5 for an element that had multiple minor errors or no actual "errors" but was just very poor quality.

Of course, a skater who has very poor quality on multiple elements is not likely to be earning PCS in the 9s to begin with.

I'm not talking about poor quality by elite standards, but as would apply across all skaters at all levels. Imagine, for example, a double axel by a novice skater that is telegraphed, entered with little speed and exited with less, small, and poorly positioned. That could be landed with adequate rotation on one foot on a back outside edge and still legitimately scored as -4. But it's hard to imagine that skater's performance would be under consideration for PCS above 9.

Every time I hear about PSC and falls, I remember Carolina, 2018 Europeans and her breathtaking skating between falling.

Also, a recent event - Evgenia's last win. Who cares that she has shaky jumps, a fall and simple transitions and is constantly regressing? She has a reputation and titles, so her components will never be low.

Helena b said:

Every time I hear about PSC and falls, I remember Carolina, 2018 Europeans and her breathtaking skating between falling.

Also, a recent event - Evgenia's last win. Who cares that she has shaky jumps, a fall and simple transitions and is constantly regressing? She has a reputation and titles, so her components will never be low.

Click to expand...

gsk8 said:

That's a good idea. Like trying to withdraw too much money from an ATM

Click to expand...

Exactly. Like as soon as a singles or doubled quad/triple, or a fall, or a jump/spin/footwork with lower than -4 GOE, or two falls or a downgrade (on any triple or quad or 2A) is recognized the computer should literally prevent the judges from going any higher than the proscribed PCS. It would be funny to see a judge try to input a 9.75/10.00 and be denied and prompted to judge more fairly.