COP: How important is judge selection?

[carriecmu0503]

I wish the GP judges were all the same judges with only a few substitutes as an exception. I know it's relatively unrealistic but I'd appreciate the consistency.

This would have judges working all over the world for 6 weeks straight, 8 if you count the GPF. Please remember that judges are volunteers who take time away from their "real" lives to judge at events. They are not paid beyond reimbursement for expenses. Many judges use their vacation time from work to judge, but it is unrealistic to think that they would have 6-8 weeks of paid leave that they could take in one chunk to judge the entire GP series. Also, most judges judge a lot more than the international events. Even judges who judge GP/ Worlds/ Olympics are judging at local events and test sessions throughout the year. Let's be realistic about what we expect of judges.

 

[OS]

What Mathman calculated is what is the probability of the same European countries be chosen again and again. But your question was about any European countries in the panel, where the point is just to be 8 European countries of 9 of them in the panel, not the same 8 European countries. So that is not right result for your question. And its just funny how you use some numbers which are totally different from the thing you trying to prove, to prove the same. But you can keep trying.. just use logic sometimes

You sure? That is not what I have asked, or understood had been calculated (ie/ Any 8 from the 17).
Since you seem so clued in, what would be your calculation be then? Based on 5 consecutive years of probabilities on 8/9 EU fed representation based on the same pool of 26 judges with European represent 65% at 17 representation every year per average?

Assume EU federation vs Rest of the world federation representation 65%: 35 % proportion, the repeating of same EU Fed representation also sticks out

2 X same EU fed = 4 / 5 years (5% chance probability each, made even rarer when you consider there's 2)
6 x same EU fed = 3 / 5 years
4 x same EU federation = 2 / 5 years
(Left out other EU fed who only represented 1/5 years)

vs the total number of all non-EU feds representation in last 5 years

China = 0 / 5 years
Japan = 1 / 5 years
US = 1 / 5 years
Korea = 1 / 5 years
Canada = 2 / 5 years
Australia = 1
South Africa = 1

There are just too many inconsistencies with natural probabilities over the last 5 years ladies to be truly random.

 

[Baron Vladimir]

You sure? That is not what I have asked, or understood had been calculated (ie/ Any 8 from the 17).
Since you seem so clued in, what would be your calculation be then? Based on 5 consecutive years of probabilities on 8/9 EU fed representation based on the same pool of 26 judges with European represent 65% at 17 representation every year per average?

Assume EU federation vs Rest of the world federation representation 65%: 35 % proportion, the repeating of same EU Fed representation also sticks out

2 X same EU fed = 4 / 5 years (5% chance probability each, made even rarer when you consider there's 2)
6 x same EU fed = 3 / 5 years
4 x same EU federation = 2 / 5 years
(Left out other EU fed who only represented 1/5 years)

vs the total number of all non-EU feds representation in last 5 years

China = 0 / 5 years
Japan = 1 / 5 years
US = 1 / 5 years
Korea = 1 / 5 years
Canada = 2 / 5 years
Australia = 1
South Africa = 1

There are just too many inconsistencies with natural probabilities over the last 5 years ladies to be truly random.

I perfectly understand what you asking. Mathman already calculated that is almost 20% chance of 8/9 European panel in that precise pool of judges. So chances are around 20% per year if we are using the same judging poll. I already gave you my opinion that 7/9 European panel would be rightfull, using facts Eppen provided (there is around 75% European judges in the poll, while its only 25% judges from non European countries... if there is more non European countries in some poll, like in example you provided, there can rightfully be even 3 non European judges in the panel). But 8 European judges in the panel of 9 is still possible outcome, as it is 2 North American judges of 9, even both solution are not the best one giving current state of judging polls. The problem you are talking about is not in probability, but in a little amount of countries outside of Europe with official judge' represents... Anyway, the problem could be easily solved if ISU start using 3 different polls, so 7 judges can be chosen from Euro poll, 1 from NA and Oceania poll (USA, CAN, MEX, AUS NZL), and 1 from Asia poll (CHN, KOR, JPN, KAZ, PHI). They can even go further and divide Europe in 7 regions, so choose one Euro judge from ex USSR (RUS, UKR, BLR, GEO), second from Baltic (EST, LAT, LTU, FIN), third from Scandinavia and Benelux (SWE, NOR, DEN, BEL, NED), fourth from Balkan (HUN, BUL, SLO, ROU), 5. from central Euro (AUT, POL, SVK, CZE, SUI), 6. from south/Mediterranean (ITA, ESP, CRO, TUR), 7. from west (FRA, GBR, GER) for example. E: And, also, when other different countries in Asia can send trained official judges to represent them (for example), they should give Asia as a continent 2 slots in the panel, leaving Europe with 6 etc etc

 

[OS]

I perfectly understand what you asking. Mathman already calculated that is almost 20% chance of 8/9 European panel in that precise pool of judges. So chances are around 20% per year if we are using the same judging poll. I already gave you my opinion that 7/9 European panel would be rightfull, using facts Eppen provided (there is around 75% chance that European judge will be selected in the panel, and 75% of 9 judges is 7 judges). But 8 European judges in the panel of 9 is still possible outcome, as it is 2 North American judges of 9, even both solution are not the best one giving current state of judging polls. The problem you are talking about is not in probability, but in a little amount of countries outside of Europe with official judge' represents... Anyway, the problem could be easily solved if ISU start using 3 different pools, so 7 judges can be chosen from Euro poll, 1 from NA and Oceania poll (USA, CAN, MEX, AUS NZL), and 1 from Asia poll (CHN, KOR, JPN, KAZ, PHI). They can even go further and divide Europe in 7 regions, so choose one Euro judge from ex USSR (RUS, UKR, BLR, GEO), second from Baltic (EST, LAT, LTU, FIN), third from Scandinavia and Benelux (SWE, NOR, DEN, BEL, NED), fourth from Balkan (HUN, BUL, SLO, ROU), 5. from central Euro (AUT, POL, SVK, CZE, SUI), 6. from south/Mediterranean (ITA, ESP, CRO, TUR), 7. from west (FRA, GBR, GER) for example. E: And, also, when other different countries in Asia get official judges (for example), they should give Asia as a continent 2 slots in the panel, leaving Europe with 6 etc etc

You can't simply discount consecutive years of improbability as if they are isolated incidents, unrelated to each other. It would be like saying winning 5 consecutive jackpots is not a big deal because apparently winning jackpot once, the chance is not that low. I mean can't you at least admit these consecutive overwhelming majority panels are highly unusual and hugely improbable?

It is exactly the type of pattern that can snowball minor slants into bigger slants over time that establishes inflation like we have experienced the last couple of years with selective European skaters, with the rest of the field catching up usually 1, 2 competitions later as system tryings to realign itself, and actually left some skaters even more behind if they are not part of the power fed makeup. It is the same principle behind how you can boost momentum for skaters who are only assigned to the favourable event leading up to he WC (e.g home events, home region events, friendly competitions) that tends to only benefit power fed skaters, hence create inflations. Kostner used to do tons of senior B European events which ensures she build up good PCS cushion and buzz that had become her strength, and save her many times over the years even when her tech failed on the day. The skaters themselves still need to deliver of course, but it helps if they have momentum, build a bigger safety net and have the judges panel on their side.

The idea about proportional representation from 3 pools of judges would be a good start to ensure there is at least equal representation covering the rest of the world - especially important, given they are in the minority group while covering at least 3 continents. Although in the long run, ISU should aim to increase the pool of judges to at least 30 (add 4 more federations to the current 26), aiming for approx 15 European judges random selection from Europe, and 15 Non European judges randomly selected from the rest of the world to make the panel more in balance 50:50 not 65:35. Take initiatives to train, develop judges diversity program, encourage to bring in a minor federation's participation more to become the overall ISU representation. Not given any chance to judge is part of the reasons the pool is kept small.

 

[gkelly]

Again, remember that judges are volunteers and train primarily at their own or at their federations' expense.

It's easier for judges in Europe to get international experience because in most cases there are other countries a few hour's drive or train ride away. Judges from Australia or the Philippines don't have that option.

I believe the ISU does subsidize training to some degree (e.g., free seminars?) for officials from newer federations with developing programs. But still, where is the best place to hold these trainings that's most convenient for the largest number of small federations? Usually in Europe.

It's no one's fault; it's just a fact of geography and economics.

 

[plushyfan]

Adelina's jumps were judged to be strong across the board in Sochi. Her 3T-3T in the SP, her 3Lo in the LP and her 2A-3T in the LP received +2 and +3 from every single judge. In the SP and/or LP, there were judges from KOR, RUS, ITA, USA, JPN, and CAN. The result was controversial, but agreement that those jumps were solid is unquestionable and in no way a reflection of the panel composition.

the judges scored uniformly, regardless of their nationality in both skater's case ( Yuna and Adelina)

 

[Sam-Skwantch]

This would have judges working all over the world for 6 weeks straight, 8 if you count the GPF. Please remember that judges are volunteers who take time away from their "real" lives to judge at events. They are not paid beyond reimbursement for expenses. Many judges use their vacation time from work to judge, but it is unrealistic to think that they would have 6-8 weeks of paid leave that they could take in one chunk to judge the entire GP series. Also, most judges judge a lot more than the international events. Even judges who judge GP/ Worlds/ Olympics are judging at local events and test sessions throughout the year. Let's be realistic about what we expect of judges.

Well that's why I said it would be relatively unrealistic

My point was understated admittedly. What I think bugs me most about the GP is how many variances we can see from event to event which are mostly TP related but not entirely. So I'd rather for the sake of consistency see the skaters face the same judges when establishing Season Bests that can affect skater's eligibility and to a lesser extent scoring perceptions and potential among fans and judges.

Too bad the ISU can't properly pay officials though like other major sports can afford to do. Only in a perfect world I guess. It is hard for me to imagine in today's day and age the funding isn't there for a six week job thar could properly compensate it's judges.

 

[Mathman]

What Mathman calculated is what is the probability of the same European countries be chosen again and again.

No, these are the probabilities that in each of n consecutive years some combination of 8 European countries are selected. They could be a different 8 each year.

The probability that the same 8 European countries are selected each year is vanishingly small. Even for two years, if 8 particular European countries are chosen the first year, then the (conditional) probability the the same 8 are chosen the next year is .00000224.

The only assumptions are that each year there are 26 countries in the titoal pool (they don't have to be the same countries every year) of which 19 are European (not necessarily the same 19 countries every time), and that the draw each year is independent of past draws. If there are a different number of countries in the pool from one year to the next, the results are off a little bit, but not by much.

It's the "so many years in a row" that is so hard to achieve, probabilistically. If you flip a coin the probability of getting heads is 50%. but getting heads ten times in a row -- that's unlikely. (This is not the same as asking, "given that I already got 9 heads in a row, what is the probability of getting heads on the tenth trial." That would be 50%.)

 

[Mathman]

[Figure Skating] is governed like an 'old boys' club at their own discretion, practically run like an Oligarch with their clansmanship that holds majority power.

I think that this is the problem more so than the fact that many judging panels have a lot of Europeans. "Europe" is surely too large and diverse to raise concerns that skaters from other parts of the world will be disadvantaged. I don't think that a judge from Spain would automatically favor
Kolyada over Uno just for reasons of geography. (Of course, he will favor Fernandez over both.)

I also am pretty sure that the ISU employs statisticians to pour over all of these mathematical matters. I wouldn't be surprised if the "oligarchs" do have their own agendas, though.

I agree with all the maths but one thing to consider... these numbers work in a real aleatory pool...

we know that this is not the case here.

That is the big question. We don't really "know" what other factors may come into play. We make guesses and accusations when our favorites don't win, but we don't have much actual information about what goes on in the back corridors of the ISU or in the judges' heads.

 

[Baron Vladimir]

No, these are the probabilities that in each of n consecutive years some combination of 8 European countries are selected. They could be a different 8 each year.

The probability that the same 8 European countries are selected each year is vanishingly small. Even for two years, if 8 particular European countries are chosen the first year, then the (conditional) probability the the same 8 are chosen the next year is .00000224.

The only assumptions are that each year there are 26 countries in the titoal pool (they don't have to be the same countries every year) of which 19 are European (not necessarily the same 19 countries every time), and that the draw each year is independent of past draws. If there are a different number of countries in the pool from one year to the next, the results are off a little bit, but not by much.

It's the "so many years in a row" that is so hard to achieve, probabilistically. If you flip a coin the probability of getting heads is 50%. but getting heads ten times in a row -- that's unlikely. (This is not the same as asking, "given that I already got 9 heads in a row, what is the probability of getting heads on the tenth trial." That would be 50%.)

What you calculated first is possiblity that in panel of 9 judges 8 are European in a poll of 26 judges with 19 of them are European. Lets say its 17% possiblity. Now you need to calculate how possible is that more different outcomes with 17% possibility to happen each year would be reproduced in 4 of 5 year... so you can't use same 17% by simply multiple it, because it can be more different outcomes in the pool which gave you the same 17% possiblity to happen each year... With multiple it you are only calculating probability of 17 things out of 100 possible will be chosen again next time..

 

[zounger]

The reason why we have more European and big federation judges than the above math calculations are showing us (which are correct I think), is in the mechanics of the actual judging draw.

For example according to the rule 521 (page 90, 4c). "A Judge must not serve in more than one (1) discipline per ISU Championships;". So if a "small" federation has only one qualified judge then this judge can not be present in others disciplines. The big federations have plenty of qualify judges so they may end up with a judge in each discipline.

We have to include that fact as well in our calculations to make it more realistic. Also, we need some "history" of the previous championships according to the 4a of the same rule.

 

[Baron Vladimir]

You can't simply discount consecutive years of improbability as if they are isolated incidents, unrelated to each other. It would be like saying winning 5 consecutive jackpots is not a big deal because apparently winning jackpot once, the chance is not that low. I mean can't you at least admit these consecutive overwhelming majority panels are highly unusual and hugely improbable?

It is exactly the type of pattern that can snowball minor slants into bigger slants over time that establishes inflation like we have experienced the last couple of years with selective European skaters, with the rest of the field catching up usually 1, 2 competitions later as system tryings to realign itself, and actually left some skaters even more behind if they are not part of the power fed makeup. It is the same principle behind how you can boost momentum for skaters who are only assigned to the favourable event leading up to he WC (e.g home events, home region events, friendly competitions) that tends to only benefit power fed skaters, hence create inflations. Kostner used to do tons of senior B European events which ensures she build up good PCS cushion and buzz that had become her strength, and save her many times over the years even when her tech failed on the day. The skaters themselves still need to deliver of course, but it helps if they have momentum, build a bigger safety net and have the judges panel on their side.

The idea about proportional representation from 3 pools of judges would be a good start to ensure there is at least equal representation covering the rest of the world - especially important, given they are in the minority group while covering at least 3 continents. Although in the long run, ISU should aim to increase the pool of judges to at least 30 (add 4 more federations to the current 26), aiming for approx 15 European judges random selection from Europe, and 15 Non European judges randomly selected from the rest of the world to make the panel more in balance 50:50 not 65:35. Take initiatives to train, develop judges diversity program, encourage to bring in a minor federation's participation more to become the overall ISU representation. Not given any chance to judge is part of the reasons the pool is kept small.

They are not highly unusal, they are just unusal... There are also a lot of countries from Europe, part of FS, which are not involved in judging (because they dont have one), and that number is almost the same like the number of countries from rest of the world not involved too. So, its highly unfair not to include other European countries (Greece, Ireland, Cyprus, Moldova etc) just to make panel less European... What you want is like to put African or European judge in American football game, when American football is a sport playing only in America and normal logic tells you that only America has judges to do the job... So, the only problem is your preconception that 8 judges from Europe, which made 'European panel' how you called it is a bad thing... But according to your logic, people should complain and call panel of judges with 2 North American judges in the panel - 'American panel' (because it can be only 3 judges from NA in the poll of 40, if we consider all countries in the world who can send the judge), or panel with 3 Asian judges we should call 'Asian panel'... Balance cant be 50%/50% where there is so much more European judges involved in FS... Its not ISU fault why there is only 3 judges in a poll from countries which can represent NA and 8 which can represent Asia, opposite of 30 judges representing different European countries in general... The problem is not in probability but in your preconceptions, and some people already stated what they are... Its OK to disagree with 8 European judges of 9 in the panel (when obviously better solution is 7 of 9) but to say how that is not possible to happen and how that happened only to bring some specific outcome(s) is just not right...

 

[Mathman]

The reason why we have more European and big federation judges than the above math calculations are showing us (which are correct I think), is in the mechanics of the actual judging draw.

For example according to the rule 521 (page 90, 4c). "A Judge must not serve in more than one (1) discipline per ISU Championships;". So if a "small" federation has only one qualified judge then this judge can not be present in others disciplines. The big federations have plenty of qualify judges so they may end up with a judge in each discipline.

We have to include that fact as well in our calculations to make it more realistic. Also, we need some "history" of the previous championships according to the 4a of the same rule.

It always helps to check the rules.

What's even worse, from a statistics point of view, is that actually 13 judges in each discipline are chosen from the pool. Then later there is a draw among the 13: some will judge the short program only, some will judge the long program only, and some will judge both.

Anyway, before anyone gets too paranoid, the results for the Pyeongchang Olympics miraculously are relatively stacked against European countries, at least in ladies and men.

The 13 countries chosen for the ladies panel are Canada, USA, Japan, Kazakhstan, South Korea, China (6 non-European countries) and Russia, Belgium, Slovakia, France, Germany, Hungary, Latvia (7 European countries). It looks like Medvedeva will just squeak through.

For men it is Canada, USA, Japan, China, Uzbekistan, Kazakhstan, Australia (7 non-European countries) versus Spain, Russia, Israel, Latvia, France, Czech Republic (6 European countries). Looking good for Hanyu, Uno, Jin, Chan and Chen, bad for Fernandez. Kazahkstan seems to the big power broker here. :yes:

For pairs and dance the split is more along the lines of expectation. Both 13 member panels have 9 European members and 4 non-European.

 

[4everchan]

It always helps to check the rules.

What's even worse, from a statistics point of view, is that actually 13 judges in each discipline are chosen from the pool. Then later there is a draw among the 13: some will judge the short program only, some will judge the long program only, and some will judge both.

Anyway, before anyone gets too paranoid, the results for the Pyeongchang Olympics miraculously are relatively stacked against European countries, at least in ladies and men.

The 13 countries chosen for the ladies panel are Canada, USA, Japan, Kazakhstan, South Korea, China (6 non-European countries) and Russia, Belgium, Slovakia, France, Germany, Hungary, Latvia (7 European countries). It looks like Medvedeva will just squeak through.

For men it is Canada, USA, Japan, China, Uzbekistan, Kazakhstan, Australia (7 non-European countries) versus Spain, Russia, Israel, Latvia, France, Czech Republic (6 European countries). Looking good for Hanyu, Uno, Jin, Chan and Chen, bad for Fernandez. Kazahkstan seems to the big power broker here. :yes:

For pairs and dance the split is more along the lines of expectation. Both 13 member panels have 9 European members and 4 non-European.

well... Kazahkstan and Uzbekistan are former Soviet "states" so... they may be part of 4CC but are they really not European?

 

[Tulipstar]

well... Kazahkstan and Uzbekistan are former Soviet "states" so... they may be part of 4CC but are they really not European?

The Soviet Union was not just situated in Europe

 

[4everchan]

The Soviet Union was not just situated in Europe

i know that.. and Russia isn't really either...

my point is that if we consider that European judges favour European, including Russian skaters, then we need to consider its former republics as "European" in the conversation we are currently having.

 

[Tulipstar]

i know that.. and Russia isn't really either...

my point is that if we consider that European judges favour European, including Russian skaters, then we need to consider its former republics as "European" in the conversation we are currently having.

If we consider that.

If geography is so important, then it makes much more sense that Kazakhstan would favour China, who is their neighbour (!) and not Spain, which is about eight countries away from them, for instance. Kazakhstan chooses to be part of the Four Continents, so I think we should respect that they identify themselves as belonging there and not in Europeans. And if we stay on the respectful side, perhaps we can consider that they might not favour anyone because of geographical or cultural reasons and strive to judge fairly.

I also find it strange that we are focusing so much on the ladies, which is the one discipline in which 'Europe' has been dominant the last few years. In all other three, they are not. How can that be if European judges dominate and they are all so eager to favour each other? Perhaps because they are not nearly as culturally similar or inclined to favour each other as some suggest.

 

[Neenah16]

If we consider that.

If geography is so important, then it makes much more sense that Kazakhstan would favour China, who is their neighbour (!) and not Spain, which is about eight countries away from them, for instance. Kazakhstan chooses to be part of the Four Continents, so I think we should respect that they identify themselves as belonging there and not in Europeans. And if we stay on the respectful side, perhaps we can consider that they might not favour anyone because of geographical or cultural reasons and strive to judge fairly. I also find it strange that we are focusing so much on the ladies, which is the one discipline in which 'Europe' has been dominant the last few years. In all other three, they are not. How can that be if European judges dominate and they are all so eager to favour each other? Perhaps because they are not nearly as culturally similar or inclined to favour each other as some suggest.

Come on Tulipstar. Why are you trying to bring logic to this discussion, we were having fun cherry picking data and making interesting conspiracy theories based on that data. I am personally very excited and looking forward to the part were aliens involvement is revealed and we finally find out why judges from Mars are inflating Medvedeva's PCs :hap85:

Joking aside, you are correct. There is a lot wrong with the analysis done here (at least the math is correct), and I am disinclined to accept any results that are reached by picking partial data to support a hypothesis rather than using the full set of data to make a conclusion

So before accepting any offered "truths" and analysis, I would really like a satisfying answer to these questions:
1) Why only the ladies?
2) Why a 5 year period and not more or less?
3) Why would European countries support skaters from other European countries in a competition where they are going against each other?
4) Why not consider other reasons for inflation (other than geography)?
5) What is the selection process for judges? What factors can naturally influence it? How much weight those factors have when considering the probabilities?
6) How do we define a balanced judging panel? Why? What proof do we have that a specific make of the panel is balanced or fair? (basically, if your hypothesis is that there is tampering in the selection process and that the panels selected do not have the right people, you need first to show who are the right people and why they are the right people)

These are the questions I have off the top of my head, and will probably think of more later, but they should be enough to start really looking into the issue logically and with a real scientific approach (meaning your feelings have nothing to do with it)

Wish I had time to do the research myself. This could be an interesting paper :think:

 

[moriel]

Also people started bringing former Soviet states.
Just a note - they are not always on friendly terms, and being a former Soviet state does not mean automatically that a judge from that country will vote for Russia. See Ukraine and Poland as examples

In general, i think more than location, the important factor is a country having their skaters in a competition, because judges tend to give higher grades for their own skaters, and lower grades for their direct competition. This is not only about podium chances, this goes all along top 10 (actually, in the study I made last year, there was a judge that, in FS, gave higher scores to own skater who was in 8th and lowballed competition - ranked 7th. Interestingly, this didnt happen in SP when the own skater was not threatened by this competitor just yet).
So, basically, I don´t think KAZ will support Fernandez or russian ladies - they will support Denis and Elizabet, for example =) And so on.

 

[Baron Vladimir]

Also people started bringing former Soviet states.
Just a note - they are not always on friendly terms, and being a former Soviet state does not mean automatically that a judge from that country will vote for Russia. See Ukraine and Poland as examples

In general, i think more than location, the important factor is a country having their skaters in a competition, because judges tend to give higher grades for their own skaters, and lower grades for their direct competition. This is not only about podium chances, this goes all along top 10 (actually, in the study I made last year, there was a judge that, in FS, gave higher scores to own skater who was in 8th and lowballed competition - ranked 7th. Interestingly, this didnt happen in SP when the own skater was not threatened by this competitor just yet).
So, basically, I don´t think KAZ will support Fernandez or russian ladies - they will support Denis and Elizabet, for example =) And so on.

So then, right think should be not to include judges who have their own representatives. Ladies judging panel should be for example MEX, NZL, UZB, LTU, BUL, AUT, NOR, GBR, ESP. Mens JAR, PHI, TPE, BLR, FIN, CRO, SVK, POL, TUR etc etc