Judges & Referees Webinar for GOE +5 to -5

[chaser]

The point of a seminar, or webinar in this instance, is to learn something.
Streaming the Congress was a great move. Likewise, webinars are also a good move. But, they have to serve a real purpose, other than providing paid jobs for the production team.

Do the singles and pairs ISU Judges that were there have more information that they take to their respective countries? And what if they don’t have more information, or have not absorbed it properly, or are poor presenters? The end result becomes a fractured interpretation. That’s why a good webinar that truly explains what will be required for each of the eleven scores (-5 to +5) could really assist.

It’s fair to say that +5 and -5 are identifiable for any high level judge. The range that could be a killer for the skaters is the range +2 to +4 and -1 to -4. If the guidelines (with unclear bullet points) are open to interpretation country by country and judge by judge, the end result could become a lottery. Giving 0 and +1 doesn’t look to have been a problem in years gone by. It’s everything in between that has danger written all over it.

Maybe watching the singles webinar a second time will be more beneficial than the first go around. I didn’t get how the bullet points were said to be only guidelines. What’s that about? They needed proper explanation. Maybe the ISU should have practised a webinar of this sort with all the main players watching it back before leaping in. They looked unprepared and indecisive.

Oh, and using current competitive skaters doesn’t sit well.

I guess the incremental PC scoring going up by group has been put in the back of the sock drawer with all the other lost socks.

 

[draqq]

Well, it's inherent to evaluation of figure skating since the beginning, made slightly more precise and certainly more granular since the beginning of IJS, but not to the point of matching the precision of scoring in less complex or less qualitative sports.

6.0 short program judging had mandatory deductions (many of which included a numerical range, as do many of the GOE reductions), but other than that the guidelines for scoring programs under 6.0 were much less specific than they are under IJS. Judges go through years of trial judging and judging lower level events as they work their way up to international and if possible ISU judging appointments. During that process -- and before, if they were skaters themselves -- they get a sense of what is "average" or "good" or "very good" for each kind of element and for each overall program-wide quality or criterion (program component). What to expect as average for novices or juniors vs. seniors. For men vs. women.

That standard is always going to be a work in progress as judges get more experience watching skaters at all levels, and as the field of skaters as a whole tends to improve dramatically at some kinds of skills and to let other skills that are no longer considered as important to fall by the wayside.

And then they're supposed to consider all the good points and all the bad points in coming up with their scores.

Under 6.0 there was one technical score for everything in the whole program -- judges had to weigh the good qualities and the errors or weaknesses and the difficulty of the content and balance it all out to come up with one score. There were no explicit guidelines about what was more important and what was worth less. Each judge had to decide that for themselves, influenced by what they learned from more experienced judges through the trialing and early career judging process.

With IJS, it's just the good qualities and errors or weaknesses of one element at a time -- with tech panel/scale of values taking care of the difficulty determination. So that's a lot more precise than 6.0 judging. But there are still caveats that the range of quality from very poor to outstanding is continuous but the scores available for GOEs are discrete integers, errors can occur with varying degrees of severity (which is reflected in the rules where a range of deductions/reductions is suggested), and positive qualities can also occur with a range of quality from above average to good, very good, excellent, and outstanding. (The +GOE bullet point rules either +3 or +5 do not explicitly allow for rewarding one point for "good" and more for "excellent" execution of the same quality, as the earliest GOE guidelines ca. 2003-05 did -- perhaps calling the bullet points "guidelines" is a way to allow for that flexibility.)

True, the 6.0 system was also very vague. Sure, they had mandatory deductions in the short program on the technical side, but judges could be quite lenient or severe, or some judge might have caught a flutz or double-footed landing while others didn't. I think the IJS system is overall an improvement; I just think that it needs to take a step further and provide data to clarify some of the adjectives used to assess GOE.

Clarity in the form of spelling out ranges of positives, or giving video examples of poor, fair, average, good, excellent? Stating explicitly where judges have room to exercise judgment or to balance out strong positives, mild positives, strong weaknesses/errors, and mild weaknesses, vs. where there is only one correct final GOE?

Up to a few years ago there used to be a number of errors that required negative GOE or required -3 GOE especially in short programs, but then the rules/guidelines were explicitly loosened to allow judges to offset reductions for errors with positive qualities.

Now the only mandatory GOE is for doing only one valid jump in the SP jump combination slot.

Providing video examples would be a good start for clarity (for the viewers too), not only for qualitative GOE purposes but also for assigning levels to elements. For instance, what are the judges actually looking for when assigning levels to footwork sequences and the key points in the compulsory portion of the (now) rhythm dance?

As for the webinar itself, I would have liked to have seen more video examples of negative GOE elements (or elements that have both positive and negative GOE), as opposed to explanations of each positive GOE bullet point.

If you're going to measure height and distance on jumps or rotational speed on spins, should the cutoff between "good enough to earn this bullet point" or not be the same for a 4'10" woman or a 6'2" man? Or even skaters of the same sex and skill level but with a full foot of height difference?

Or should the technology just tell judges the number of inches/centimeters or the number of RPMs and let judges draw their own mental guidelines – to which they could then be more consistent in drawing their own lines as to when to award the bullet point.

Spin centering is less dependent on the size of the skater, so that could be more absolute. But could the technology account for the inevitable change of circle size that occurs with a switch to a forward edge while spinning, so that change-edge spins aren’t penalized for a slight change of centering that would be penalized in a backward-edge-only spin?

Similarly, could the technology have different settings to assess the rideout on the first jump landing of a combination depending whether the subsequent jump is a loop or a toe loop?

Etc.

I’m sure it would theoretically be possible to account for all these variables. But at what point does the gain in precision justify the cost of developing and implementing the systems and technology?

Would it make sense just to determine certain aspects of a performance that can be measured absolutely and just plug those numbers directly into the scoring with no consideration of the size of the skater or the exact element being performed, and let the judges concentrate on judging all the qualities that can’t be measured objectively at all?

The point of the technology in my mind would be providing precise information and data to judges (and the viewers), not as a sort of mandatory requirement where this skater has to, say, achieve X meters to get a +2 on a jump, or something like that. The judges will indeed have to translate the given numerical info into the GOE, accounting for gender, size, and such. It's not that I think the judges aren't knowledgeable or all that inconsistent really, but it's just to provide some factual, measured grounding and context to some of the GOE marking. I'm the kind of person who loves to see replays of top-scoring elements where it's clear just how far a lady went in a throw jump or how much height a skater got on a 3A.

I think that ultimately this kind of technology will help the skaters improve on the GOE side of every element. All of them already know that they need to work on more speed/ice coverage, fast spins, higher jumps, etc. but the data would help them (and us) figure out what it will actually take to get those coveted +2s, +3s, and so on. And with enough data over time, the judges themselves will come to determine the general numerical ranges that they expect to see for a "fast" spin, "good" ice coverage, etc.

 

[gkelly]

Providing video examples would be a good start for clarity (for the viewers too), not only for qualitative GOE purposes but also for assigning levels to elements. For instance, what are the judges actually looking for when assigning levels to footwork sequences and the key points in the compulsory portion of the (now) rhythm dance?

Judges do not assign levels to footwork sequences or identify key points in the rhythm dance. Judges don't even know which levels have been assigned.

That is the job of the technical panel. Different individuals (for the most part) who receive different training.

As for the webinar itself, I would have liked to have seen more video examples of negative GOE elements (or elements that have both positive and negative GOE), as opposed to explanations of each positive GOE bullet point.

My understanding is that this webinar was only intended to educate judges about how to use the different numbers (+5/-5 vs. +3/-3), not how to identify good and bad qualities of elements. The latter is part of the ongoing training of each judge that takes place over years of trial judging, seminars/judges' schools, and judging at lower levels (and also their own skating for those who were skaters themselves.

I still have a lot of questions about the subtleties of the differences between how to apply these +5/-5 vs. +3/-3 guidelines. I would prefer to see more in-depth discussion of how to handle multiple errors, errors on elements that also meet one or more positive bullet points, whether or where there is room to reflect differences between "good" and "excellent" on the positive side, etc. In that sense, I'd say that these singles/pairs webinars only scratched the surface of the new numerical subject.

But using the numbers is not nearly as deep a subject as identifying quality or errors in the first place, and I don't think that was what they were intended to address. That kind of education primarily takes place in person and even if there were a days-long webinar that covered everything that might be covered at a trial judge experience or in-person seminar, it would still be inadequate to educate by video only about what the elements look like or sound like from a fixed ice-side position.

The point of the technology in my mind would be providing precise information and data to judges (and the viewers), not as a sort of mandatory requirement where this skater has to, say, achieve X meters to get a +2 on a jump, or something like that. The judges will indeed have to translate the given numerical info into the GOE, accounting for gender, size, and such. It's not that I think the judges aren't knowledgeable or all that inconsistent really, but it's just to provide some factual, measured grounding and context to some of the GOE marking. I'm the kind of person who loves to see replays of top-scoring elements where it's clear just how far a lady went in a throw jump or how much height a skater got on a 3A.

I think that ultimately this kind of technology will help the skaters improve on the GOE side of every element. All of them already know that they need to work on more speed/ice coverage, fast spins, higher jumps, etc. but the data would help them (and us) figure out what it will actually take to get those coveted +2s, +3s, and so on. And with enough data over time, the judges themselves will come to determine the general numerical ranges that they expect to see for a "fast" spin, "good" ice coverage, etc.

OK. So it's just a matter of developing inexpensive technology to derive these measurements, that can get used in all sorts of rink configurations without interfering with the skating or other functions of running the event. And making the information available to the judges quickly at the end of each program (I recommend not flashing it on their screens during the performance because it will distract them from watching the next element or the in-between skating). Getting the info to the judges afterward should be the easy part.

 

[CanadianSkaterGuy]

"Empty" and "boring" first half of the Alina's program being used by ISU as an example of musicality? LOL

A skater can exhibit musicality (as in movements match the musical structure/rhythm pattern) but their program can still be empty and boring.... it's just one aspect of artistic expression and interpretation.

In Alina's case, the criticism is that the first half was her going through movements to essentially use up time before the 2nd half when she puts the jumping passes. And yeah, often times I did feel like her step sequence at the beginning while nicely executed and in time with the music (i.e. with musicality), didn't really contribute much to the performance from an artistic standpoint because it did (IMO, and others' opinions) look like she was just going through the motions.

 

[Danny T]

Just had some time to watch the webinars. Agree with everyone that the Ice Dance video is by far and beyond the superior one. Not that I'm ever gonna try and self-score ID (even that's too much drama for me & I only watch ID for the aesthetics not scoring discourse anw ) but the video was very detailed and comprehensive. I like the speakers - they are really kind to skaters with negative features, and they try going for some humour so it's not too dry and monotonous.

As for the singles & pairs, well, Bianchetti needs to retire yesterday already. Freaking bicycle analogy :disapp: (I don't buy the 'bullets are guidelines only so judges have room for evaluation' argument anyway, ice dance also has their own single and double features allowance and they are very specific, not some IKEA smoke)

 

[Baron Vladimir]

^^^
We can see and that in ice dance webinar! a clear example where they cant decide is it better to give 0 or +1 GOE. Im just saying thats a normal situation in process of FS judging. Individual judges should have different opinions sometimes and final result will be something beetwen those different opinions.

 

[Henni147]

Okay, I've just looked at a scoresheet of Stephen Gogolev from Minto Summer Skate 2018:
https://mobile.twitter.com/gogogogolev_/status/1023232214136434688/photo/2

Let's have a closer look at the raw sum of GOE without considering the scale of values:
J1 -4
J2 +8
J3 +12
J4 +17
J5 +20
J6 +10

How can you justify a difference like that?!? :shocked:

 

[chopinskate]

Because the judges haven't changed so why will the judging bla bla?

 

[cohen-esque]

Well, sure, but it’s not usually so bad. I’m giving the judges more credit and assuming they really do need to get used to the new GOE system. Look at the range of scores for 2003 Skate America, where the judges were similarly getting a grip on the (original) new system: http://www.isuresults.com/results/sa2003/sa03_ladies_fp_scores.pdf

 

[gkelly]

Okay, I've just looked at a scoresheet of Stephen Gogolev from Minto Summer Skate 2018:
https://mobile.twitter.com/gogogogolev_/status/1023232214136434688/photo/2

Let's have a closer look at the raw sum of GOE without considering the scale of values:
J1 -4
J2 +8
J3 +12
J4 +17
J5 +20
J6 +10

How can you justify a difference like that?!? :shocked:

Judges are not supposed to be thinking in terms of the total net GOE for the whole program. They're just supposed to evaluate each element on its own merits -- according to their own standards which should be consistent across all skaters, and according to their own understanding of the rules.

Some judges tend to be very strict applying GOE reductions and stingy about applying pluses for all skaters. Other judges tend to be more lenient on the minuses and more generous on the pluses. (And others might be more conservative or more adventurous in both directions.) If you have one judge who leans toward the stricter, more punitive side of things and another who is more inclined to look for qualities to reward, their total net GOEs may end up very different from each other. As long as each judge is consistent in the application for all skaters, they are doing their jobs fairly. It becomes a cliche to say "That's why there are 9 (or 6 in this case) judges."

If every judge thought identically, there would be no need for a whole panel and one person's marks would be just the same as another's.

For now, in this first month of the new GOE rules, I expect that most judges are in the process of recalibrating their understandings of how the new guidelines match what they're seeing on the ice and how to apply the numbers. There are some differences from the old guidelines that might cause some judges to say "I could have found 4 positive bullet points under the old rules, but with the new ones I can really only justify 1." Or if they're balancing positive and negative bullets to end up at a score in the middle, they might find the negatives more often outweighing the positives with the new point values.

I expect that individual judges will become more assured in their own application of the new guidelines as they get more practice in the coming months. And also that conversations among judges will result in more shared understandings of when it's OK to award positives that now have higher bars according to the new wording, how much it's OK to balance errors in one part of an element with strong points elsewhere, etc.

So I expect judges' approaches to using the new numbers to become more standardized with more practice and more sharing of strategies, for there to be fewer examples of such wide differences among judges by this winter, or by next summer.

But I don't expect all judges to think in lockstep by then either. That's not how the system is intended.

(Another thing to take into account is that judges at these summer competitions may not all be international judges with as high a level of experience.)

 

[Henni147]

I agree that the judges are not familiar with the new system yet and it's likely that these judges at the Summer Skate have less experience.

On the other hand it appears like J1 and J5 have watched two entirely different programs, no? I'm really curious how the scoresheets will look like in December :angry2:

 

[Andrea82]

Okay, I've just looked at a scoresheet of Stephen Gogolev from Minto Summer Skate 2018:
https://mobile.twitter.com/gogogogolev_/status/1023232214136434688/photo/2

Let's have a closer look at the raw sum of GOE without considering the scale of values:
J1 -4
J2 +8
J3 +12
J4 +17
J5 +20
J6 +10

How can you justify a difference like that?!? :shocked:

(Another thing to take into account is that judges at these summer competitions may not all be international judges with as high a level of experience.)

In the Free Skating, none of the judges have International or ISU status


In SP they used a different panel where
J1 was an ISU judge
J3 was an International judge
J2-J4-J5 couldn't judge in international events in Singles (J2 is an international judge for Ice Dance)

 

[4everchan]

I agree that the judges are not familiar with the new system yet and it's likely that these judges at the Summer Skate have less experience.

On the other hand it appears like J1 and J5 have watched two entirely different programs, no? I'm really curious how the scoresheets will look like in December :angry2:

not at all...

these judges marked within their range... one is more severe the other is more generous...keep tallying up the numbers for all skaters. I wouldn't be surprised if you found out that judge 1 was more severe across the board while judge 5 was more generous... and that would be fine...

what is NOT fine is when a judge who is normally severe is very generous for ONE skater and mean for all the others... or vice versa.

 

[Henni147]

SP judges
J1 is an ISU judge
J3 is an International judge
J2-J4-J5 can't judge in international events in Singles (J2 is an international judge for Ice Dance)

wait...this is for the FS given the elements liste in the link
In the Free Skating, none of the judges have Internaitonal or ISU status

Thanks for clarification! In that case we should wait indeed

 

[Ice Dance]

I think people are definitely having conversations about how to apply the new scoring and the impact it is having on results. A couple comments I overheard at Lake Placid this week, where there were some very experienced judges & also both national & international competitions with different dance panels.

One person said "they"--no idea who they is, it could have been people studying protocols or giving skaters feedback or who knows--were discussing the scoring for hours one night.

A couple people were talking about the negative GOE--that it could have a huge impact or unexpected impact. (Not too surprising. I've heard quite a bit of discussion about giving out +4's & +5's in the various threads this season but not much if anything about the expanded range of negative GOE & how it should be applied).

My guess is that these kind of discussions are going on everywhere this summer and will likely be going on all over the world throughout the preseason.

 

[yude]

In the Free Skating, none of the judges have International or ISU status


In SP they used a different panel where
J1 was an ISU judge
J3 was an International judge
J2-J4-J5 couldn't judge in international events in Singles (J2 is an international judge for Ice Dance)

Thank you for your information!