COP: How important is judge selection?

[OS]

COP: How important is judges selection?

For years, I have felt COP can also stand for Conduct of Points. Points itself may imply absolute value but are actually meaningless, they can vary from competition to competition. The power and influence lies more with the judges than the skaters.

Moving on from the Japan Open thread where I made an analysis of of past 5 consecutive years in the ladies, where against mathematically probability, we see EU judges consistently makes up the overwhelming majority judge panel during the ladies FS final, with an average of 8/9 representation and 100% representation 9/9 in 2015. This is despite 4 out of 5 world champions had all took place outside Europe (and thus disprove the the standard believe if you have competition in the region, you are more likely to get judges from that region.)

What also stood out, is the overwhelming majority REPEATED EU federations representation (judged as many as 4 times) out of 5, while non European Judges from the rest of the world, only makes up an average of 1 time every 5 years, including US, and zero time for China. The question I pose to the forum is

1) How important is judge selection?
2) Is proportionate representation and diversity important?
3) Can competitions be slanted by judge panel?

4) Do you feel there should be proportionate representation at world events, and what that mean? For example, should there be minimum 3 continents representation.

5) Finally, what would be the ideal and fairest panel look like and how to achieve it? By randomised draw? Qualification? Judging history? Avoid conflict of interest?

 

[gkelly]

Good questions. I have no answers. I know very little about how officials are chosen for international events.

I think there are several different ways in which the composition of the judging panel can have an effect on scoring:

*Cultural preferences among different skating communities. There are numerous different groups within Europe so Russian vs. French vs. British preferences, for example, might cancel each other out.

*Familiarity with the skaters. European judges are more likely to have seen not-yet-medal-contender European skaters at European Championships and at senior B event the majority of which take place in Europe, and to be less familiar with American or Canadian or Japanese or Australian skaters first emerging on the world scene. American and Canadian and Japanese and Australian judges will be familiar with their own country's new international skaters, but less likely with each other's. They may have seen some at Four Continents.

*Developing consensus among judges about what makes good skating, what to notice especially to reward or penalize. Aside from official discussions which all referees should promote -- in English which is the official language of the ISU -- different language groups might have unofficial discussions in their own common languages. This may not involve discussing specific skaters at all, but maybe general issues like "Do skaters with harder programs deserve higher PCS for the same level of Performance/Composition/Interpretation?" or "How much do falls affect the overall impression of a program?"

*Outright collusion. Which can just as well happen between officials from different continents. Although knowing each other well from often serving on panels together, and speaking the same language, would make it more likely.


At least, the skaters have more control with IJS than they did under 6.0. Judges have no control over the base value part of the technical score, which can make up to 50% or more of the total score. Technical panels have some control, but except in so-called gray areas the calls are driven by what each skater actually does on the ice.

 

[Mathman]

5) Finally, what would be the ideal and fairest panel look like and how to achieve it? By randomised draw? Qualification? Judging history? Avoid conflict of interest?

I think a random draw is best. If you give someone the power to hand-select judges, no matter by what criteria, that's just asking for more trouble.

That said, there are certainty statistical red flags that make me think that something more is going on than just a pure random draw. I am not alarmed that a lot of European countries are drawn -- there are more European countries in the ISU and more countries that have qualified ISU judges. Still, it seems like the biggest skating countries are drawn more often than probabilities predict, and not always to the advantage of Europe. The USA sits on judging panels more often than does Lichtenstein or Moldavia.

My understanding of the current procedure is that there is a random draw of countries well before the event; then each nation that was chosen nominates an individual judge. The draw is conducted in front of many witness. If the ISU is cheating somehow (by stuffing the ballot box, for instance) I think word would quickly get out. It may be that for major events like Worlds, some countries have no ISU certified judges and decline to participate in the draw, upping the odds for everyone else.

 

[eppen]

Fact checking is something that gets done less and less these, I guess. But considering mathematics and geography, I have to say I am a little suprised at OS’s results. The winners for ladies in the 5 past Worlds (I guess this is the competition we are talking about) come from Russia (3, Tuktamysheva, Medevedeva x2), Japan (1, Asada) and South Korea (1, Kim). Russia as a country belongs to Europe despite its large Asian part (list of European countries eg here: https://en.wikipedia.org/wiki/Europe#List_of_states_and_territories), so that is 3/5 winners from Europe and 2/5 from Asia.

In the men’s category, the victories in the past 5 years to Canada (1, Chan), Spain (2, Fernández) and Japan (2, Hanyu). Europe, Asia and North America represented in the continental level. In the pairs, the results have been to Russia (1), Germany (1), Canada (2) and China (1), again Europe, Asia and North America represented. And in ice dance we have, USA (1), Canada (1), Italy (1), and France (2). North America and Europe represented.

The countries and larger regions that the top skaters come from probably does not surprise anyone, but the judges also score 29 other skaters in the singles categories and almost the same number of teams in the other categories. In that sense it is perhaps better to look at the big picture instead of concentrating only on the very top. Now, I did not have the time to calculate the countries for all the 5 past Worlds, so I just took the latest one as an example. In recent years, the number of skaters from budding skating countries like Brazil or the Philippines has increased, so the diversity of countries is more likely to be visible in the most recent competitions.

The judges for the 2017 Worlds were selected by the customary draw in 2016 (http://isu.org/communications/519-2048-judges-draw-by-number-championships-2017-rev/file) and 26 countries participated (not necessarily to all the disciplines, btw): 17 European countries (AUT, BEL, BLR, CZE, FRA, GER, GBR, ITA, LAT, LTU, NOR, RUS, SLO, ESP, SWE, SUI, UKR), 6 Asian countries (CHN, ISR, JPN, KOR, TUR, UZB), 2 North American (CAN, USA) and Australia. 17-9 for Europe means that more European judges were likely to be drawn.

ISU member countries with ISU level judges can enter the draw. You have to have a rep at the place where the draw is made. It is usually good if the country has more than one ISU level judge because there are different disciplines and one judge can rep his/her country only a certain number of times in ISU competitions. Some 25-30 countries usually participate and 13 get drawn from that pool. The final panel gets drawn an hour or so before the start of the competition. These days, the panel is changed for the FS so that 4 judges from the SP panel are replaced by the remaining 4 (this happens also by draw before the FS competition starts). (The origins of the judges can be checked eg here: http://www.isu.org/communications/12127-isu-communication-2111/file)

So, the idea is (I guess) that every member country can participate and gets an equal chance of getting a judging assignment drawn. If you look at the list of judges, you will notice that not every country has judges at ISU level (not even every European country), but not surprisingly the countries who have world class skaters tend to also produce judges at international and ISU level. I don’t think there would have been that many Spanish judges say 15 years ago, but now that they have skaters in all disciplines and they have also qualified judges for every discipline. Ideally,the final panels in the major competitions represent a selection of the active countries with skaters participating in the competitions.

Now back to 2017 Worlds…

Men: Asia 12 countries (ARM, AZR, CHI, TAI, GEO, ISR, JPN, KAZ, MAL, PHI, KOR, UZB), Europe 12 (BEL, CRO, CZE, FIN, FRA, GER, ITA, LAT, RUS, ESP, SUI, UKR), North America 2 (CAN, USA) and Australia. 15:12 non-European:Europe (1,25:1).

The judges came from Asia with 4 judges (SP: ISR, UZB, CHI; FS: COR), Europe with 8 judges (SP: AUS, BEL, GER, GBR, ITA, SPA; FS: RUS, UKR). CAN had a judge in the FS. 5:8 non-European:European judges (1:1,6). Here, the panel features slightly more European judges than would be proportionally ok considering the origins of the skaters.

Ladies: Asia 7 countries (ARM, CHN, TAI, JPN, KAZ, SGP, KOR), Europe 17 countries (AUT, BEL, CZE, EST, FIN, FRA, GER, HUN, ITA, LAT, NOR, RUS, SLK, SLO, SWE, SUI, UKR), Americas 3 countries (BRA, CAN, USA). 10:17 non-European:Europe (1:1,7).

The judges came from Asia 2 countries (JPN, KOR), Europe 8 countries (CZE, FRA, GBR, NOR, SUI; FS: LAT SLO, SWE), CAN and AUS. 4:8 between non-European and European. Considering the origins of the skaters a good representation of judges countries (1:2).

Pairs: Asia 3 countries (CHN, KOR, JPN), Europe 13 countries (GER, RUS, FRA, ITA, CZE, AUS, CRO, LTU, BLR, HUN FIN, GBR, SUI), North America 2 countries (CAN, USA) and Australia. 6:13 non-European:European (1:2,17).

The judges came from Asia 2 countries (CHN, JPN), Europe 7 countries (BLR, SUI, RUS, FRA, GER; FS: LTU, ITA), North America 2 countries (USA; FS: CAN) and Australia. 5:7 between between non-European and European (1:1,4). Here, Europe underrepresented based on skater origins.

Ice Dance: Asia 7 countries (ISR, CHN, TUR, KOR, JPN, ARM, AZR), Europe 15 countries (FRA, RUS, ITA, DEN, POL, UKR, GEO, ESP, GER, GBR, FIN, CZE, BLR, LAT, LTU), North America 2 countries (CAN, USA). 9:15 non-European:European (1:1,67).

The judges came from Asia 3 countries (ISR, KOR, TUR), Europe 9 countries (RUS, FRA, CZE, ESP; FS: UKR, GER, GBR), North America 2 countries (USA, CAN). 5:9 non-European:European (1:1,8). I’d say a fairly good representation of the origins of the skaters.

So, looking at the whole array of participants and judges, the situation looks actually quite ok. There could be more diversity in the mens towards non-European regions (and in time it might come, if the small feds continue to be represented in skaters), there could also more Europeans in the pairs event panel. The proportions probably vary a bit every year but I should expect overall similar results from previous years.

Last year, when the scores could be connected with the judges and their respective countries, I looked at the results from the GP series which has judges always from the participating countries with some additional ones to complete the 9-judge-panel if needed. The national bias was clear in that if there was a major player for a country, their judge tended to score their own higher and the major opponents slightly lower. Like, the Japanese judges were always giving good scores to Hanyu and Uno and lower scores to Fernández, Chan etc. There was no indication of continental block voting, like Europeans always scoring say Fernández high (he usually got good scores from Eastern European and Asian judges, Spain is in Western Europe, but Western European judges did not in general score Javi high). North Americans hardly ever gave each other skaters high scores. This excercise can be repeated now and there is the JGP series to look at also.

For the non-Europeans it seems also sometimes to be hard to understand that Europe is not a uniform block (despite the European Union), but it really consists of dozens of nation states with varying languages, cultures, histories, traditions etc. I would find it amazing if say a Polish FS judge would have the language skills and the time to follow the junior scene in all of Europe, eg in Finland or in Italy or in Latvia, in addition to his/her own country. The emerging top skaters from different countries come out maybe in smaller international competitions, but national scenes are hard to follow already because of the variety of languages used.

In addition, I started thinking about the dispersion of the Soviet FS community in the 1990s to not just Eastern European but to also a large numner of Asian countries (among them Uzbekistan, Kazakstan, and even Israel)... Asia is definitely not a uniform block either.

So, I guess I am asking based on the numbers presented above (albeit just from one major competition), what is really wrong with the current system? Does it enable geographical block voting? Or is that just a thought pattern from the 6.0 era? It is a whole lot more difficult to control 18 scores given by 9 judges with the trimmed mean treatment than the 2 scores from the old system.

What is the diversity that gets asked in the first post? Each European country is different from the others, why is that not enough? Supporting emerging countries in their effort to establish also judges is a good suggestion, though.

If all the judges came from countries with no skaters in the competition, would it change the results radically considering how the ISU trains their judges?

How much do the personal preferences of a judge inform their scoring as compared to their possible national preferences/obligations?

E

 

[4everchan]

in some sports, judges from the same country as a finalist are not allowed in... perhaps it's too complicated in figure skating.. but what if we took away judges from coutnries represented in the final flight... or even final two flights?

woudln't that reduce national bias?

 

[Ares]

in some sports, judges from the same country as a finalist are not allowed in... perhaps it's too complicated in figure skating.. but what if we took away judges from coutnries represented in the final flight... or even final two flights?

woudln't that reduce national bias?

Maybe it would but I can imagine it being very difficult to apply. Different judges in different flights? Such an inconsistency would make the whole event look untrue. The national bias surely exists but there are cases that suggest otherwise. USA judge scored Michelle the lowest of them all here. The only one who put her in the 2nd place. 5.6 in technical? 1996 SP - Worlds. I see a bigger problem in reputational judging myself and rewarding high PCS just because of high Tech content.

https://www.youtube.com/watch?v=1imuQWeIi4Q&t=37s

 

[Neenah16]

Whole post

Thank you very much for doing the research, I wanted to do it myself but did not have the time, your effort and time is really appreciated.

I personally think that there is no issue with the judges selection. Meaning I don't think anything illegal is happening based on the procedure used and the countries participating in the draw. Maybe there are better ways to choose the judges but treating them as groups who think and act the same based on the continent they come from is not it.

 

[Hevari]

Maybe it would but I can imagine it being very difficult to apply. Different judges in different flights? Such an inconsistency would make the whole event look untrue.

Maybe not different judges in different flights but no judges from countries represented in final two fligths at all. Seems that 4everchan meant something like this... And I just add - no tech panel members from countries represented in final two flights as well

 

[Ares]

Maybe not different judges in different flights but no judges from countries represented in final two fligths at all. Seems that 4everchan meant something like this... And I just add - no tech panel members from countries represented in final two flights as well

Maybe it could be doable.

 

[Hevari]

Thank you very much for doing the research, I wanted to do it myself but did not have the time, your effort and time is really appreciated.

I personally think that there is no issue with the judges selection. Meaning I don't think anything illegal is happening based on the procedure used and the countries participating in the draw. Maybe there are better ways to choose the judges but treating them as groups who think and act the same based on the continent they come from is not it.

For me it seems that ISU should assign judges personally and not assign countries, who's MF's will then assign a judge. Same way that is used in soccer for example

 

[Ares]

For me it seems that ISU should assign judges personally and not assign countries, who's MF's will then assign a judge. Same way that is used in soccer for example

It's a good idea as the countries would not be able to pick the most favourable judge for their interests. I still remember that Russian judge giving +2, +3 for seemingly every Lipnitskaya's jumping pass in Sochi. That's a farce.

 

[lappo]

For me it seems that ISU should assign judges personally and not assign countries, who's MF's will then assign a judge. Same way that is used in soccer for example

Totally agree with this. it worked for the Ancient Olympic Games (only one case of unfair judgement in over 1000 years of history) it can as well work now, at least for Grand Prix series, World and Olympic Games.

 

[Sam-Skwantch]

I wish the GP judges were all the same judges with only a few substitutes as an exception. I know it's relatively unrealistic but I'd appreciate the consistency.

 

[Sam-Skwantch]

It's a good idea as the countries would not be able to pick the most favourable judge for their interests. I still remember that Russian judge giving +2, +3 for seemingly every Lipnitskaya's jumping pass in Sochi. That's a farce.

This is a pretty inaccurate thing to say unless you are just mixing up Julia and Adelina.

http://www.isuresults.com/results/owg2014/index.htm

Not to mention there is no way to know which judge is which in this event. Also...look at the top scores. There was one judge for each of them (except Mao for some reason :ddevil: ) that gave higher marks than the average. I think a reasonable logic could conclude and make a case for that being the same person. Maybe even one of the judges added in the FS but wasn't a part of the SP panel. These things can't be definitively answered.

 

[Hevari]

I wish the GP judges were all the same judges with only a few substitutes as an exception. I know it's relatively unrealistic but I'd appreciate the consistency.

I don't understand this... You mean you want all six GP's to be judged by the same panel? Right?

 

[Sam-Skwantch]

I don't understand this... You mean you want all six GP's to be judged by the same panel? Right?

Yes

 

[gkelly]

Totally agree with this. it worked for the Ancient Olympic Games (only one case of unfair judgement in over 1000 years of history) it can as well work now, at least for Grand Prix series, World and Olympic Games.

Were there judged sports in the ancient Olympic Games?

 

[lappo]

Were there judged sports in the ancient Olympic Games?

Totally OT. Not judged in the same ways FS is judged, but there was a panel of 10 judges (the Hellanodikai, notice that even the name is associated with the concept of justice "dike"). They were first of all supposed to ensure that the athletes were divided according to the proper class ages (the young, the middle and the old) for each discipline. Then they had to establish who was the winner in the different disciplines of running and equestrian running (there were no cameras so they had to sit right in front of the finishing line and do it with their eyes); the same with the throwing of javelot and the discus and the long jump. Finally, there were several kind of different boxing games and each of them had different rules about which kind of moves were permitted or not and they had to ensure that the the two athletes would not use them. The Hellanodikai were citizen of Elis (the city that administered and hosted the Games) and were known for their fairness and held in the greatest estime, literary accounts often refers to them in the most honorable ways.

Edited to add: of course, Greeks were no saints and there were several bribery attempts between coaches and athletes and the Hellanodikai had to investigate about every suspicious case. If they found someone guilty they stripped him of the Olympic crown and fined him; the fines were used to erect statues of Zeus in the sanctuary of Olympia.

 

[4everchan]

Maybe it would but I can imagine it being very difficult to apply. Different judges in different flights? Such an inconsistency would make the whole event look untrue. The national bias surely exists but there are cases that suggest otherwise. USA judge scored Michelle the lowest of them all here. The only one who put her in the 2nd place. 5.6 in technical? 1996 SP - Worlds. I see a bigger problem in reputational judging myself and rewarding high PCS just because of high Tech content.

https://www.youtube.com/watch?v=1imuQWeIi4Q&t=37s

i didn't mean different judges in different flights though... simply that final flights skaters would see their national judge removed.... i think diving is like that. Of course, for them, the scores are erased from prelims, to semis to the final.... but I don't think it would be a crime to have a completely different panel for the LP, that would not have the top contenders' countries represented... It would balance things out...

and yes, sometimes a judge from your own country can screw you over... but most likely it's the opposite...

and finally, changing the panel for the LP would reduce the risks of collusion....

 

[Hevari]

Yes

Why so? I don't see any advantages of that...but see disadvantages and risks of collusions