Judging math details

moriel said:

As a side note, keep in mind that, appearently, the judges DO NOT DIFFERENTIATE the different PCs categories, basically just giving some score for all fields, with minor variation.
So all you people who keep saying "omg, the PCs categories are a necessity, they are needed to differentiate the subtle aspects", the judges dont.

Click to expand...

Yes, because all of them are dependable/based on same thing - blade work and body work. Thus, we got one total PCS score in the end, presented to audience. edit: but making 5 categories of it is more useful to skaters themselves, because it (through those minor variations between categories) gives them recommendations in which part they can work more (for example when Mirai get 7.75 in PE and 8.25 for IN that means she could skate with more awarness of the audience while skating, when Ashley get 8.25 in SS and 9 in PE that means she could work more on speed and edges, when Tukt get 8.50 in SS and 7.75 in TR that means she could work more between the elements to make her programme looks as a whole etc etc)

moriel said:

As a side note, keep in mind that, appearently, the judges DO NOT DIFFERENTIATE the different PCs categories, basically just giving some score for all fields, with minor variation.
So all you people who keep saying "omg, the PCs categories are a necessity, they are needed to differentiate the subtle aspects", the judges dont.

Click to expand...

Agreed. Five might offer more information to skaters and perhaps the five categories are well enough defined to be distinct from one another. But it's a different story whether the average judge is distinguishing between the five given the environment and time limitations. Remember that the brain automatically takes shortcuts to save on resources. The judges are not trying to do their jobs badly, but corners get cut when you're given so much to do.

My questions lie more with whether the ISU has asked an expert to come in to determine whether the number of judging tasks assigned can be completed well (remember we are deciding competitions on hundredths of a point) within the couple minutes that the judges have to make their assessment. Every category might be well intentioned but the actual result in practice may leave something to be desired.

Baron Vladimir said:

Yes, because all of them are dependable/based on same thing - blade work and body work. Thus, we got one total PCS score in the end, presented to audience. edit: but making 5 categories of it is more useful to skaters themselves, because it (through those minor variations between categories) gives them recommendations in which part they can work more (for example when Mirai get 7.75 in PE and 8.25 for IN that means she could skate with more awarness of the audience while skating, when Ashley get 8.25 in SS and 9 in PE that means she could work more on speed and edges, when Tukt get 8.50 in SS and 7.75 in TR that means she could work more between the elements to make her programme looks as a whole etc etc)

Click to expand...

There is almost no feedback for the skaters concerning thr different components . After a casual skate judges usually do not differentiate between the 5 components. Usually they decrease the value for transitions by half a point and that's it. It should not be that way but it currently is.

After a rough skate you can sometimes see that the scores for skating skills and choreography are still high but the value for performance is significantly lower. But you do not have to be master to judge those things.

I recommend to have a look at the marks given for Evgeni Pluschenko at the Sochi Team event. This was a highlight of professional and non-professional judging.

^^^^

They are not, nor supposed to be that much distinctive from one another, like some of you think, and you can see that through the judges scores. They are named and defined differently in order to give more detailed guidelance about PCS score and help judges and skaters, not because they are so different (like i said their basics are the same and they all refer to skating during whole programme). If they decrease transitions for most of the skaters, that means there is parts of their programmes which could be filled with something to make programme looks like one unity. Judge who gave Pluschenko 5 for TR and 8,5 for the rest was too biased (and probably wrong), but his point is the same.

Baron Vladimir said:

^^^^

They are not, nor supposed to be that much distinctive from one another, like some of you think, and you can see that through the judges scores. They are named and defined differently in order to give more detailed guidelance about PCS score and help judges and skaters, not because they are so different (like i said their basics are the same). If they decrease transitions for most of the skaters, that means there is parts of their programmes which could be filled with something to make programme looks like one unity. Judge who gave Pluschenko 5 for TR and 8,5 for the rest was too biased (and probably wrong), but his point is the same.

Click to expand...

That in itself may be a problem. When I grade something with that many factors I often have to go over it more than once because it's hard for the brain to focus on that many things at once.

To the extent that a judge is assessing five categories, they at least need to be distinct enough to make distinguishing calls in a time crunch. Otherwise you end up with anchored scoring like we have now--where the judges pay more attention to certain categories and judge the rest based on the one or two. Again, if the judges had more time I would be more trusting of their ability to judge such overlapping categories, but it is hard for humans to achieve it in such a short time.

That's why I argue for simplification. Not because I don't think the categories are useful or that they don't capture important parts of skating. But if in practice the PCS is really coming down to one or two categories then it might be time to change something about the equation.

GGFan said:

That in itself may be a problem. When I grade something with that many factors I often have to go over it more than once because it's hard for the brain to focus on that many things at once.

To the extent that a judge is assessing five categories, they at least need to be distinct enough to make distinguishing calls in a time crunch. Otherwise you end up with anchored scoring like we have now--where they judges pay more attention to certain categories and judge the rest based on the one or two. Again, if the judges had more time I would be more trusting of their ability to judge such overlapping categories, but it is hard for humans to achieve it in such a short time.

That's why I argue for simplification. Not because I don't think the categories are useful or that they don't capture important parts of skating. But if in practice the PCS is really coming down to one or two categories then it might be time to change something about the equation.

Click to expand...

In today practice probably yes, because most of the skaters learned how to skate every part of the judges task for the programme to their own maximum. But every skater could have something happening when its time to compete which can change some of those marks, from competition to competition. At the end, we must ask judges if current judging is too complicated for them, but i think more categories (than one or two) are helping them because its telling them for what to look and make their judgment more objective. And they already had a lot of expirience with them in their practice, which makes it easier to judge every next time. At the very end those categories are again one mark presented to audience.

Baron Vladimir said:

In today practice probably yes, because most of the skaters learned how to skate every part of the judges task for the programme to their own maximum. But every skater could have something happening when its time to compete which can change some of those marks, from competition to competition. At the end, we must ask judges if current judging is too complicated for them, but i think more categories (than one or two) are helping them because its telling them for what to look and make their judgment more objective.

Click to expand...

Oh I completely agree that you want to keep the judges honest by preventing the categories from being too broad and meaningless. But even there we might have better ways to achieve that. As moriel has been arguing it would keep the judges more honest if they were just judging very small discrete categories and then the computer was doing the job of building the scores. Ultimately I think a better compromise can be reached than what we're doing currently because I actually would like the actual day of competition to make a difference. That is I want PCS on P/E to be lower on days where the performance wasn't quite there. But I think the anchoring effect often gets in the way.

Judges never have the opportunity to judge anything down to a hundredth of a point. For PCS, they don't even get tenths of points any more -- only quarter points.

Base values and GOEs can break down TES to tenths of points.

The hundredths come from the averaging and factoring.

Which brings us back to the math questions from the beginning of this thread.

Most decisions between most skaters in most events are relatively clearcut, decided by whole points or more. Differences in tenths of points are also common. Rarely, some close decisions do come down to tenths of points.

To avoid results decided by hundredths of points, we could:

*Maintain all decimal places in the computer until the very end of the calculations, and round only the total TES and the total PCS, to tenths only. (Deductions are already whole numbers so they wouldn't add decimal places.) The effect would be that the same number of decisions currently decided by hundredths of points would end up being ties.

*Multiply all the point values by 10, and round to tenths all along. This would just move the decimal point in the same calculations and have the effect of elite scores being in the thousands instead of hundreds. No scores would ever be reported beyond the first decimal place.

*Multiply all current element base values by 7, rounded to whole numbers; use 9 judges for every single competition (this would be a hardship at local levels); and then drop high and low scores from the panel but just add up the remainder with no averaging. Again, elite scores would be in the thousands.

*Use 12 judges at elite events, drop high and low, and then there would be 10 scores to average so the averaging would just move the decimal point without creating additional decimal places -- and then round to tenths (This would be even more of a hardship and so would never be adopted at local levels, which would therefore still have the current messier decimal places. But keep in mind that, although elite scores with the current scale of values range in the hundreds, at middle levels scores are in double digits, and at the lowest levels total final scores might be less than 10, so decimal places are what makes the difference.)

Which brings us to the other question, of how much difference judges should be showing between different component scores for the same performance.

In theory, the whole range of scores 0 to 10 is available for all competitions. We're only going to see 9s and 10s from the very best in the world at those components, so already that full range is not relevant to most skaters.

At various competition levels, there's probably a minimum level of skating skill we would expect to be able to execute the elements expected at that level. E.g., a skater who can only manage SS score of 2.0 probably can't do double axels or triple jumps.

For other components, it is theoretically possible to be much better at performance or musical interpretation than at skating, and it's always possible to be worse than one's general skating skill level on any given day if the skater is focusing purely on technique and elements.

But for the most part, skaters develop their skills in the various component areas at approximately similar rates throughout their development as skaters. They can do their best to make up for weaker technical skills by emphasizing artistry up to a point, but eventually they will reach a limit of what they are physically able to do with their current skill level. If they don't pay much attention to artistry but do have strong technique, they will at least make the unartistic skating look better than a less skilled skater of the same presentation ability.

As Mathman and Baron Vladimir have suggested above, we could think of the whole number part of the component scores as identifying the general range of competition a skater belongs in, and the decimal places as distinguishing between their strengths and weaknesses within that range, or distinguishing between individual skaters in the same general range. (If two skaters are almost identical in ability but one is more musical than the other, then 5.0 5.0 5.0 5.0 5.5 beating 5.0 5.0 5.0 5.0 5.0 would be meaningful. Most of the time there's a little more difference between skaters and across components. But at the lower skill levels, the range of scores for a whole event might be 1.0 to 3.0 or 2.5 to 4.0, which doesn't give much room to distinguish among a field of 12 or more skaters without the decimal places (only quarter points, not tenths or hundredths, are allowed to each judge, though, remember) and multiple components.

And sometimes there will be outliers at either end, who deserve scores in a completely different range from the rest of the field. Most often, if they're much better or much worse than the field they're going to be much better or much worse in all or most areas.

At large events where entry is limited more by age than by tests passed or required minimum scores, we can see a wider range of skills in the same events. JGPs are good for that purpose. The skaters who are weaker overall tend to be weaker in most areas, and vice versa, with occasional exceptions. I'm going to make a separate thread for examples.

Overlapping between ranges within the same event is certainly allowed. If you had two skaters with similar skating skill but one was exceptionally musical and the other completely ignored the music doing moves that were not even choreographed to recognize the phrasing or style, you might get one skater with 5.0 ... 6.5 and another with 5.0 ... 3.5. Any judge awarding a difference of 1.5 between highest and lowest score for the same performance is making a statement that these performances were not well-balanced in their skills and that their interpretation should be considered in a different range than their skating skills. And that the interpretation skills of these two skaters that day was in completely different ranges -- in a freeskate with PCS factors of 1.8, the difference between the first and second skater's IN score is worth more than most triple jumps. But still with no more than 1.5 difference between high and low scores for any individual skater.

I think we're talking past each other because gkelly and Baron Vladimir are arguing that the five categories are valuable and the judging system as it currently stands is doing a good job.

I agree that the system generally does a good job but I argue that "good job" captures quite a range and doesn't mean that improvements shouldn't be made. Often my questions just have to do with wanting to see more scientific validity behind what the judging system claims to achieve. I have no problem with what it is meant to achieve but I question whether it is achievable.

If two skaters are almost identical in ability but one is more musical than the other, then 5.0 5.0 5.0 5.0 5.5 beating 5.0 5.0 5.0 5.0 5.0 would be meaningful. Most of the time there's a little more difference between skaters and across components.

Click to expand...

Let's get back to the math here and the critical difference between accuracy and precision. "Meaningful" is very relative here. In order for these distinctions to be meaningful we would have to trust that judges across the spectrum are very accurate. The judges have finely tuned instruments (quarter points) to be precise in their judgments and perhaps some judges are but given the number of variables involved I do not think they are very accurate. That is if more time were given and those 5 categories were really broken out how close would that result be to the judging done under the time crunch.

Other things like reputation, currents standings, skating order, audience reaction etc. play too much of a part for me to agree that the judging system is not in need of some reform. Take Kaori for example. I know there is variance across her programs but the changes in her skating skills scores across programs would completely "eat up" and make irrelevant that .5 difference the judges would account for with musicality. This seems to have little to do with an objective change in her skating skills between the programs. Also the range of scores often indicates that judges are not in agreement as to what a 5 or 5.5, or 5.75 means. All of those are areas where there should be a margin of error built into the system and allow for more ties like QuadThrow mentioned several pages back.

All of this to say how am I supposed to take a 5.25 vs. 5.5 when the judges don't seem to be able to agree on that difference themselves?

^^^
Well, i dont think judging system is doing job that good, i think improvements could be made, too. I was just trying to explain better how some parts of it work and why they are there, because i saw some misunderstandings about it.

This might be unrelated but I wonder of some of my concerns about margins of errors would be allayed if Figure Skating (both the organization, the competitors, fans etc.) took a more dualistic approach like alpine skiing. In that sport they definitely value things like World Championship and Olympic gold medals, but they also value the World Cup results which reward consistency.

Given that different courses, conditions, start order, can make a difference in skiing I think that approach makes sense. A gold medal might have been the result of one competitor really stepping up or it could be the result of the best skiers tackling the course during bad conditions. The prestige of the World Cup acknowledges these realities. Every analogy breaks down at some point but perhaps rewarding overall consistency more or making the rankings more prestigious is another way to account for the math issues.

GGFan said:

...making the rankings more prestigious is another way to account for the math issues.

Click to expand...

I think the ISU did try to make the season's rankings more important a few years ago. IIRC they gave a $50,000 bonus to the top skater/team in each discipline. I don't know why they dropped it -- ran out of money, I suppose.

Baron Vladimir said:

Yes, because all of them are dependable/based on same thing - blade work and body work. Thus, we got one total PCS score in the end, presented to audience. edit: but making 5 categories of it is more useful to skaters themselves, because it (through those minor variations between categories) gives them recommendations in which part they can work more (for example when Mirai get 7.75 in PE and 8.25 for IN that means she could skate with more awarness of the audience while skating, when Ashley get 8.25 in SS and 9 in PE that means she could work more on speed and edges, when Tukt get 8.50 in SS and 7.75 in TR that means she could work more between the elements to make her programme looks as a whole etc etc)

Click to expand...

Judges dont differentiate the existing difference between them.
Yes, they are supposed to be same level, but the difference the judges give is TOO SMALL.

If you take some Worlds, for example, then take the PCs for each category for each skater, and then compute the range for each skater you will see what we are talking about.
Because umm, It is plausible for example for a skater have a Very good Performance and Good transitions, right? So we would expect a difference of 1 unit happening with some frequency.
But such difference barely never happens.

moriel said:

Judges dont differentiate the existing difference between them.
Yes, they are supposed to be same level, but the difference the judges give is TOO SMALL.

If you take some Worlds, for example, then take the PCs for each category for each skater, and then compute the range for each skater you will see what we are talking about.
Because umm, It is plausible for example for a skater have a Very good Performance and Good transitions, right? So we would expect a difference of 1 unit happening with some frequency.
But such difference barely never happens.

Click to expand...

How much frequency?

I urge you to go to the Distinguishing Components thread I started and give your own scores (whole numbers only) for the two components and five programs I selected.

How much difference do you see between those two unrelated components for each of those skaters. Are there any other components that you would have scored further away from the two that are the focus of the thread?

gkelly said:

How much frequency?

I urge you to go to the Distinguishing Components thread I started and give your own scores (whole numbers only) for the two components and five programs I selected.

How much difference do you see between those two unrelated components for each of those skaters. Are there any other components that you would have scored further away from the two that are the focus of the thread?

Click to expand...

What? Difference between the HIGHEST component and the LOWEST component
At some worlds, a difference of 1 happened only once, out of 37 or 38 FS.
Half of the differences are below 0.5

Now, what does that mean? That, out of 37 skaters, ONLY 1 had a one gradation difference between his STRONGEST PCs category and his WEAKEST PCs category.

Either there is no major difference between them, so it doesnt really matter if judges give 5 scores, or give one, or the judges cannot evaluate this difference properly, which again means that there is no need in 5 categories.

As for your thread, I checked it out.
We have 2 answers so far, so lemme average:

Arijana Tirak: TR = 2.75 IN = 2.38 DIF = 0.37
Riona Kato: TR = 4.12 IN = 4.38 DIF = 0.26
Polina Agafonova: TR = 5.88 IN = 5.62 DIF = 0.26
Regina Glazman: TR = 3.38 IN = 3.25 DIF = 0.13
Christina Erdel: TR = 4.25 IN = 4.75 DIF = 0.5

So far, the difference between TR and IN for all those skaters is under 0.5. For me, that means that there is no need for the judges to give separate scores for TR and for IN, because those are roughly the same.

I think of it as going like this.

At the top we have the Total Segment Score.

This is divided into two equal pats: The Total Element scores and the Total Program-as-a whole scores.

The Total element scores comprise two parts, the Base Values (what did you do?) and the GOE (how well did you do it?). The base values carry more numerical weight, but when the majority of the skaters present more or less the same elements, the GOE rises in marginal importance, so it is not too far fetched to say that the two scores are of approximate equal weight in determining placements.

The total Program Components Scores also comprise two parts: "Technical Skills sustained throughout the program" and "Presentation Skills sustained throughout the whole program." At present these are more or less equally important, with current division being 20% of the total on Tech and 30% of the total on Presentation.

To me, the scheme has a pleasing symmetry. Breaking the "Presentation Skills Sustained through the whole program" into three parts, PR, CO and IM, might be one subdivision too many. But then again, the TES is broken down into 12 or 13 further subdivisions and no one is complaining. (Most likely because it really is common for a skater do do a beautiful triple Lutz and then to fall on her triple loop, so all the scores will not turn out to be the same. )

moriel said:

What? Difference between the HIGHEST component and the LOWEST component
At some worlds, a difference of 1 happened only once, out of 37 or 38 FS.
Half of the differences are below 0.5

Now, what does that mean? That, out of 37 skaters, ONLY 1 had a one gradation difference between his STRONGEST PCs category and his WEAKEST PCs category.

Either there is no major difference between them, so it doesnt really matter if judges give 5 scores, or give one, or the judges cannot evaluate this difference properly, which again means that there is no need in 5 categories.

Click to expand...

My personal opinion is that the gaps should be a bit larger than we tend to see in real life, but not as large as the critics who complain the judges or the system are all wrong suggest.

I'd want to see differences of 1.0 between highest and lowest component should be relatively common for individual judges (if not half the skaters, then likely at least a quarter of them) and to occur at least once in large events in the panel averages; for differences of 1.5-2.0 from individual judges to occur often enough to be unsurprising but relatively rare, and larger than 2.0, or larger than 1.5 in panel averages, to be quite rare. It really depends on what the skaters put out, though. Some rare performances really will be that unbalanced, and judges shouldn't be afraid to reflect the differences.

For the majority of performances, however, I do think the differences between high and low scores will usually end up within 1.0. Now I'll explain why I think it is still valuable to continue to use separate scores:

We can think of the whole number part of each component score as the judge saying "This is the general skill level this performance represented." What separates the skaters into top, high middle, low middle, and bottom groups, or maybe more tiers than that at a large championship. Within each tier, there will be several skaters of generally similar skill level.

So judges need ways to distinguish among skaters who are generally in the same range. They might think "These three skaters really stand out from the rest of the field . . . they're exceptional skaters within the history of the sport and, these were exceptional performances. These should be our medalists today. What a great day of skating! These three skaters really deserve 9 or better in all areas." (Of course that thought is phrased in hindsight, whereas judges award the scores for each performance as it finishes.)

And similar thoughts for the Very Good and Good and Above Average etc. tiers, which might have 10 or more skates in the same tier at a large event.

We don't want judges to give the exact same score of 9 (or 10, or 7, or whatever) for "Program Components" or "Artistry" or "Presentation" or whatever we want to call it. There needs to be some way for judges to distinguish among those three Outstanding performances, or among ten Good performances.

They're not making direct comparisons between skaters under IJS, although judges might remember some of the scores they gave to specific earlier skaters and use those to benchmark their later scores in the same event.

They do often have a sense that "That was a great performance!" vs. "Wow, that is one of the best performances I have ever seen in all my decades of watching/judging figure skating!!!" Just using decimal places (maybe .1 instead of 0.25, as was the case in the 6.0 system) would allow for distinctions between great and super-great, or between Good and almost-Very Good.

But sometimes a judge might think "That was an outstanding performance. He never wavered in his commitment to the performance or to the music, not even when he fell on that quad. He really used every note and got every bit of nuance out of that music -- one of my all-time favorite pieces of music interpretation ever. But he did have that one fall. And his posture really isn't that great, which kind of distracts from the performance despite the exceptional commitment and charisma. For me, that was a 9.75 on Interpretation, but only 9.0 or maybe 9.25 for Performance."

A different skater who also excelled that day might have different strengths. "Wow, that was one of the most complex, well-constructed programs I've ever seen, with something interesting going on every moment. And today's performance was really foot perfect. I'd have to go to 9.5, 9.75, maybe even 10 for Transitions and Performance and Composition. But I will say that the middle section was a lyrical waltz, and even though the skater did a great job of skating lyrically and conveying the emotion of the story he was telling, I didn't really see the 1-2-3 rhythm or any typical waltz lilt in the knee action or upper body during that section. So I don't think I could go higher than 9.0 for Interpretation."

Lower in the standings, or at lower level competitions, the thoughts wouldn't be so superlative in phrasing. But the types of thought processes might be similar.

The skaters and the audience will never know exactly what the judge was thinking about each component. But they will know that, of these two outstanding performances, this judge thought the first skater was strongest in music interpretation and the second skater, though very strong overall, was least strong in that particular area. And I do think that is useful information, even if not very precise and even if other judges end up with different strengths and weaknesses because they reward the component criteria slightly differently.

As for your thread, I checked it out.

Click to expand...

Thanks

So far, the difference between TR and IN for all those skaters is under 0.5. For me, that means that there is no need for the judges to give separate scores for TR and for IN, because those are roughly the same.

Click to expand...

These were only 5 skaters from a field of 32, and I deliberately chose skaters with clear differences in overall skating skill. (YMMV regarding the transitions and interpretation.)

Among those 32, for example, there were 9 skaters whose five component scores all started with 4 in the ones column, with different scores in the decimal places.

If judges just awarded 4.0 to all those skaters for "Program Components" as a whole, they would all end up with the same score. Or there would be some differences from averaging if some judges were out of line and awarded 3 or 5 to some of them (which would then put those skaters into a different tier in that judge's marking).

So how should the judges, and the system, distinguish among those 9 skaters who were, overall, somewhere between Fair and Average, but some of those 9 were clearly better than others?

Well, I dont expect gaps of 2-3, but i surely expect a lot of gaps of 1, because thats pretty common - a skater have good something and then very good something. Or a skater having an outstanding interpretation, but just good skating skills.

I dunno if this sounds plausible. Just, from a not entirely qualified point of view, I kinda see such differences in real skaters.

moriel said:

Judges dont differentiate the existing difference between them.
Yes, they are supposed to be same level, but the difference the judges give is TOO SMALL.

If you take some Worlds, for example, then take the PCs for each category for each skater, and then compute the range for each skater you will see what we are talking about.
Because umm, It is plausible for example for a skater have a Very good Performance and Good transitions, right? So we would expect a difference of 1 unit happening with some frequency.
But such difference barely never happens.

Click to expand...

They are too small because skaters training all of them at the same time. When you are practicing the whole programme you are practicing all the categories equally. They are not independable one of another. They are just different perceptions on the same programme. The point in differention of them may be in day of the competition when some aspects of PCS categories may differ from skaters own maximum he can perform in trainning. And scores of categories are telling them what aspect went wrong in day of the competition... Judges task is not to compare lets say Zhenya and Alina SS and performance, they are giving marks for every skater independently, the same way professor is giving marks to their students. It is like giving your students to write 5 poems on different subject, and then judged them. The one who has better writing style will get better marks, no matter what theme of the poem is. But some differencies in details of the different poems exists, because student may be more inspired by one theme, and another find borring, and give more in one of them, so the mark will be lets say B+ for one and B- for another. You cant totally fail in writing one poem, and totally shine in writing another, when your knowledge of writing poems is only one. But its important to you as a writer to get feedback on them and see where you didnt give your maximum.

I disagree.
Yes, skaters train them all at the same time. Still, there are visible differences for most skaters. There are very few skaters who are actually well rounded.

But ok, suppose they are all the same because skaters train everything, and they need the 5 different categories to give skaters feedback. But why? since all the cats get the same score, this feedback is as useful as if there was just 1 number, period.