Page 3 of 3 FirstFirst 1 2 3
Results 41 to 53 of 53

Thread: Judging bias on the Grand Prix post CoP revision (numbers!)

  1. #41
    Spectator
    Join Date
    Sep 2018
    Posts
    3

    1 Not allowed!
    Quote Originally Posted by Shanshani View Post
    Correlation may not necessarily be causation, but that doesn't mean it can't be causation either. Can you come up with a competing explanation for why, for instance, Olga Kozhemyakina (I pick on this judge a lot, but she's a good example because she judges a lot of competitions), consistently overscores Russian skaters and underscores non-Russian skaters other than nationalistic bias?
    Quote Originally Posted by Shanshani View Post
    What do these skaters have in common that could explain her overscoring, other than them skating for the Russian federation?
    "Russian style"? Skaters from the same country might have similarities in their styles, skaters and coaches from the same country see each other more often, they have local competitions (and a lot in Russian case), even common training camps/schools.
    So Russian judge might just like "Russian style of figure skating". F.e. all Eteri skaters definitely have a lot in common. For me definitely there is a difference between any Russian girl and any Japan girl, do not know about Men/pairs/dance.
    Even more, judge might prefer Russians just because they are Russians AND because he/she likes "Russian style". And separate completely legal bias (like that style) from flag-related (OUR figure skater!) bias is not an easy task to do even for the judge herself, if she ever wants to perform such self-reflection.

  2. #42
    On the Ice
    Join Date
    Dec 2016
    Posts
    454

    3 Not allowed!
    Think I'll just link in the Buzzfeed artcile I referred to in my post above. It's absolutely crushing, 1,600 performances analysed, 27 judges identified (from 10 countries) where the chances of their scores occurring at random are less than 1 in 100,000. 16 of the judges went to the Olympics, and 2 of them (identified beforehand by the article) i.e. Feng Huang and Weiguang Chen of China ended up getting banned. Now someone tell me national bias is not endemic in skating.

    https://www.buzzfeednews.com/article...mplon/the-edge

  3. #43
    Shepherdess of the Teal Deer Metis's Avatar
    Join Date
    Feb 2018
    Posts
    1,961
    Country: United States of America

    2 Not allowed!
    Quote Originally Posted by Miller View Post
    Think I'll just link in the Buzzfeed artcile I referred to in my post above. It's absolutely crushing, 1,600 performances analysed, 27 judges identified (from 10 countries) where the chances of their scores occurring at random are less than 1 in 100,000. 16 of the judges went to the Olympics, and 2 of them (identified beforehand by the article) i.e. Feng Huang and Weiguang Chen of China ended up getting banned. Now someone tell me national bias is not endemic in skating.

    https://www.buzzfeednews.com/article...mplon/the-edge
    Even better: you can read the underlying research from economics professor Eric Zitzewitz, which has been the data-driven elephant in the room since 2010: it’s extremely readable!

  4. #44
    On the Ice
    Join Date
    Mar 2018
    Posts
    209

    0 Not allowed!
    Quote Originally Posted by Draculus View Post
    "Russian style"? Skaters from the same country might have similarities in their styles, skaters and coaches from the same country see each other more often, they have local competitions (and a lot in Russian case), even common training camps/schools.
    So Russian judge might just like "Russian style of figure skating". F.e. all Eteri skaters definitely have a lot in common. For me definitely there is a difference between any Russian girl and any Japan girl, do not know about Men/pairs/dance.
    Even more, judge might prefer Russians just because they are Russians AND because he/she likes "Russian style". And separate completely legal bias (like that style) from flag-related (OUR figure skater!) bias is not an easy task to do even for the judge herself, if she ever wants to perform such self-reflection.
    I notice you conveniently left out the list of skaters I referenced. Do you really think there is one 'Russian style' which encompasses skaters as diverse as Sergei Voronov, Dmitri Aliev, Zabiiako/Enbert, Evgenia Medvedeva, Maria Sotskova, Stanislava Konstantinova, Boikova/Kozlovskii, Tarasova/Morozov, and Pavliuchenko/Khodykin? These skaters encompass a diverse range of program types, skating styles, and skating schools. Personally, even within the same discipline I don't find Voronov and Aliev to be similar skaters at all. If they were all from the same school, maybe you would have more of a point, but skaters from different schools have very distinct styles.

    To take an example from ladies since it's the discipline you're most familiar with, sure Eteri skaters may have similar styles, but what about Elizaveta Tuktamysheva? Her programs are the opposite of Eteri-style programs--quite empty of transitions and choreographically simple, much more reliant on the charisma of the skater than on filling the program with motion. On top of that, Alina and Liza have very different jump technique as well. The idea that she and Alina share the same 'Russian style' seems pretty silly to me--they are more different than they are the same. But nonetheless, they are both overscored by Russian judges (though not Kozhemyakina, she hasn't had the opportunity yet due to being more of a mens/pairs judge).

    And to conclude, I think I will just quote Miller:

    Quote Originally Posted by Miller View Post
    Think I'll just link in the Buzzfeed artcile I referred to in my post above. It's absolutely crushing, 1,600 performances analysed, 27 judges identified (from 10 countries) where the chances of their scores occurring at random are less than 1 in 100,000. 16 of the judges went to the Olympics, and 2 of them (identified beforehand by the article) i.e. Feng Huang and Weiguang Chen of China ended up getting banned. Now someone tell me national bias is not endemic in skating.

    https://www.buzzfeednews.com/article...mplon/the-edge

  5. #45
    Spectator
    Join Date
    Sep 2018
    Posts
    3

    0 Not allowed!
    Quote Originally Posted by Shanshani View Post
    Do you really think there is one 'Russian style' which encompasses skaters as diverse as Sergei Voronov, Dmitri Aliev, Zabiiako/Enbert, Evgenia Medvedeva, Maria Sotskova, Stanislava Konstantinova, Boikova/Kozlovskii, Tarasova/Morozov, and Pavliuchenko/Khodykin?
    I have no idea. I do not see why not, if judge doing it for 20 years... Human brain is very good at finding similarities. Even when it is wrong. And with huge experience she might see something I do not.
    Quote Originally Posted by Shanshani View Post
    but what about Elizaveta Tuktamysheva? Her programs are the opposite of Eteri-style programs--quite empty of transitions and choreographically simple, much more reliant on the charisma of the skater than on filling the program with motion. On top of that, Alina and Liza have very different jump technique as well.
    The idea that she and Alina share the same 'Russian style' seems pretty silly to me-
    Its fine. But you switching registers. In a world of correlations, statistics and P-value there is no place for "silly". Either we take common sense - and I completely agree, that there are judges who are nationality-biased. Russian nationalism is quite strong. And numbers are not required in this talk, they help but just a little.
    Or we take statistics, P-value of the hypothesis, etc. In this world to correctly calculate P-Value it is required to go through all variants. And to decline "russian style" something more that "silly" is required.
    For me personally in GPF last years there was a cluster of "russian ladies". May be it is silly, may be in blind experiment my guess russian/not russian would be 50%. But it is impossible to make such experiment so I can just give my opinion, that success rate would be >50%.

  6. #46
    On the Ice
    Join Date
    Dec 2016
    Posts
    454

    1 Not allowed!
    Quote Originally Posted by Shanshani View Post
    But nonetheless, they are both overscored by Russian judges (though not Kozhemyakina, she hasn't had the opportunity yet due to being more of a mens/pairs judge).
    She did however do Ondrej Nepela - Rika Kihira 135.39 for her 8 triple, 2 Triple Axel free skate, clean but for a mistake on the final 3S (GOE -0.77 from the judges, so a 2 point swing compared with normal).

    Also I notice she is on the list of 16 worst offenders in the Buzzfeed article, as is Yuri Guskov of Kazakhstan (Elizabet Tursynbaeva was in strong contention after the SP) - he gave Rika 137.25, plus the difference between Rika's Free Skate and Elizabet's was only 11.3/11.4 points from both judges despite Elizaveta having 3 URs and a fall in her 7 triple, non-3A Free Skate - http://skatingscores.com/2019/cssvk/ladies/long/tss/

    Also this is how the list of 16 worst judges on the Buzzfeed article are performing this year - note that Shanshani's database in the OP doesn't include all competitions, but it is noticeable how one or two seem to be keeping a low profile - I wonder if the article has had some effect on some federations. I have also numbered the number of own skaters and other nationality skaters so you can see how much of a sample size we're talking about, the bigger the sample size the more notable the results obviously.

    The points difference is that between how they scored their own countries skaters vs other country skaters i.e. column E in the judges database - https://docs.google.com/spreadsheets...Q7k/edit#gid=0

    Canada - Nicole Leblanc-Richard - no main competitions judged (2 Challengers, 1 JGP)
    Canada - Jeffrey Kukasik - no main competitions judged, 1 Challenger

    China - Weiguang Chen - no competitions judged, currently banned
    China - Feng Huang - no competitions judged, currently banned
    China - Tianyi Zhang - no main competitions judged, 1 Challenger, 1 JGP

    Israel - Anna Kantor - +13.49 (2 own skaters/40 other)

    Italy - Walter Toigo - +7.82 (but +10.81 in only competition with Italian skater) - 1 own skater/19 other

    Kazakhstan - Yuri Guskov - +8.04 (1 own/17 other)

    Russia - Maira Abasova - +5.15 (6 own/36 other)
    Russia - Elena Fomina - +11.79 (8 own/18 other)
    Russia - Olga Kozhemyakina - +8.96 (11 own/40 other)

    Spain - Marta Olozagarre - +3.21 (2 own/28 other)

    Turkey - Tanay Ozkan Silaoglu - no competitions judged (Agafanova/Ucar retired)

    United States - Lorrie Parker - no competitions judged
    United States - Sharon Rogers - +5.20 (GP Finland, has also done 2 Challengers not in Shanshani's database)

    Uzbekistan - Saodat Numanova - no main competitions (Misha Ge retired) - 1 JGP.

    Interesting though how few competitions, relatively speaking, those judges specifically identified by the Buzzfeed News article have done.

  7. #47
    Medalist
    Join Date
    Feb 2014
    Posts
    1,039
    Country: Italy

    2 Not allowed!
    Quote Originally Posted by Miller View Post
    Turkey - Tanay Ozkan Silaoglu - no competitions judged (Agafanova/Ucar retired)
    She judged at Bosphorus Cup a couple of weeks ago. She will in 2019 European and World Championships panels as Turkey has been drawn and she is their only ISU qualified judge.


    Quote Originally Posted by Miller View Post
    Israel - Anna Kantor - +13.49 (2 own skaters/40 other)

    She will be at 2019 Worlds and Junior Worlds. Israel has been drawn for both Men and Pairs and they have 2 judges. So Kantor and Zaydman will one judge Men and other one Pairs. Alexey Beletsky will judge Ice Dance as Israel has been drawn also for ID at Worlds and he is the only one left.

    Quote Originally Posted by Miller View Post
    Uzbekistan - Saodat Numanova - no main competitions (Misha Ge retired) - 1 JGP.
    She will judge Pairs at 4 Continents. But there won't be skaters from Uzbekistan.



    Quote Originally Posted by Miller View Post
    China - Feng Huang - no competitions judged, currently banned
    Feng managed to get promoted as ISU Technical Controller this summer. I believe ISU suggested they won't nominate him this season

  8. #48
    Medalist
    Join Date
    Dec 2012
    Posts
    1,439

    0 Not allowed!
    @shanshani I've already told you on another forum how much I like the statistics you compiled (together with the judges NAMES)! Maybe you can consider adding another column with the deviation and bias of a judge for their countries top competitor. Because usually a judge will overscore their countries no 1 and 2 competitor, but won't do the same with skater no3. So the non-bias shown with no3 will then decrease the end bias in your table. And while some judges do overscore all their competitors, I usually don't care that for example the kazhakstan judge doesn't overscore their 2nd single ladies skater. What matters to me, is how much bias are they showing towards Tursynbava because she is their hope for a podium or top finish.

  9. #49
    Dummy
    Join Date
    Mar 2015
    Posts
    6,048

    1 Not allowed!
    Quote Originally Posted by Shanshani View Post
    I notice you conveniently left out the list of skaters I referenced. Do you really think there is one 'Russian style' which encompasses skaters as diverse as Sergei Voronov, Dmitri Aliev, Zabiiako/Enbert, Evgenia Medvedeva, Maria Sotskova, Stanislava Konstantinova, Boikova/Kozlovskii, Tarasova/Morozov, and Pavliuchenko/Khodykin? These skaters encompass a diverse range of program types, skating styles, and skating schools. Personally, even within the same discipline I don't find Voronov and Aliev to be similar skaters at all. If they were all from the same school, maybe you would have more of a point, but skaters from different schools have very distinct styles.

    To take an example from ladies since it's the discipline you're most familiar with, sure Eteri skaters may have similar styles, but what about Elizaveta Tuktamysheva? Her programs are the opposite of Eteri-style programs--quite empty of transitions and choreographically simple, much more reliant on the charisma of the skater than on filling the program with motion. On top of that, Alina and Liza have very different jump technique as well. The idea that she and Alina share the same 'Russian style' seems pretty silly to me--they are more different than they are the same. But nonetheless, they are both overscored by Russian judges (though not Kozhemyakina, she hasn't had the opportunity yet due to being more of a mens/pairs judge).

    And to conclude, I think I will just quote Miller:
    While russian skaters are all different, i do feel that there are certain major "streams" such as "russian style" and "north american / western style".
    There are some differences in actual skating, the program / choreography / interpretation and so on. Of course, it is a very wide thing, and there is a continuous transition between those two. Of course it is a very generic thing, as any binary classification.

    I would say that this is somewhat more visible in pairs and dance rather than in singles, but well.

  10. #50
    On the Ice
    Join Date
    Mar 2018
    Posts
    209

    0 Not allowed!
    Quote Originally Posted by Draculus View Post
    I have no idea. I do not see why not, if judge doing it for 20 years... Human brain is very good at finding similarities. Even when it is wrong. And with huge experience she might see something I do not.

    Its fine. But you switching registers. In a world of correlations, statistics and P-value there is no place for "silly". Either we take common sense - and I completely agree, that there are judges who are nationality-biased. Russian nationalism is quite strong. And numbers are not required in this talk, they help but just a little.
    Or we take statistics, P-value of the hypothesis, etc. In this world to correctly calculate P-Value it is required to go through all variants. And to decline "russian style" something more that "silly" is required.
    For me personally in GPF last years there was a cluster of "russian ladies". May be it is silly, may be in blind experiment my guess russian/not russian would be 50%. But it is impossible to make such experiment so I can just give my opinion, that success rate would be >50%.
    Unfortunately, it's really not possible to do a blind experiment where you could unambiguously prove causation, so inference to best explanation is the best that we can do here, and I think that the most reasonable explanation is bias. We can argue about the mechanism that causes this bias--for instance, whether it's a conscious attempt to manipulate scores, or more of a semi/un-conscious "boost" that the judge isn't being particularly deliberate about applying--but imo there are no alternative explanations that come close on the ladder of plausibility. Moreover, if it's not national bias, but rather a totally benign preference for certain styles originating from the same culture, why don't we see similar levels of bias across judges from different feds? Japanese judges are less "biased" overall than Russian judges--is that because there's no distinctive Japanese style or because Japanese people have no preference for their own cultural styles over others? That seems pretty implausible to me. There are some smaller feds that don't exhibit any bias at all--is that because they don't like their own skaters' style?

    It seems to me that in order to establish the alternative stylistic preference hypothesis, you not need to prove that there is a substantial stylistic difference between Russian and other skaters (which, imo, the burden is on you to show that--since you bring up science, the fact that you *think* you could blindly identify Russian skaters without prior knowledge of their identities doesn't cut it scientifically), but also you need to explain why there isn't a similar effect for judges of other nationalities. So sure, if you want to be scientific about it, maybe I'll retract my statement that this alternative hypothesis is "silly" and instead substitute the statement that this alternative hypothesis requires a lot of evidence and explanation of counter evidence to become viable, and that my own hypothesis that it is caused by some sort of bias mechanism is a lot more parsimonious. I also don't understand your distinction between common sense and statistics--statistics don't interpret themselves, there's always some judgment involved regarding what kind of effects are likely, what kinds of explanations are plausible, and so forth.

    (Also, for the record, I am aware of p values and so forth, and have in fact done significance testing on judge biases in the past. The results for many judges were very significant--which matches the results that have been brought up by Miller. I may get around to doing some testing once the data set has built up enough. Of course, this doesn't prove any particular causal mechanism, but if you notice a correlation between houses being left unlocked and houses being burgled, isn't it protesting a bit too much to say that "well it's only correlation, not causation"? Better to just lock your houses instead.)

    @shanshani I've already told you on another forum how much I like the statistics you compiled (together with the judges NAMES)! Maybe you can consider adding another column with the deviation and bias of a judge for their countries top competitor. Because usually a judge will overscore their countries no 1 and 2 competitor, but won't do the same with skater no3. So the non-bias shown with no3 will then decrease the end bias in your table. And while some judges do overscore all their competitors, I usually don't care that for example the kazhakstan judge doesn't overscore their 2nd single ladies skater. What matters to me, is how much bias are they showing towards Tursynbava because she is their hope for a podium or top finish.
    Hm, sounds interesting. I will consider that. Fortunately, there are official ISU world rankings so I won't have to judge myself who the top competitor is.

  11. #51
    On the Ice
    Join Date
    Mar 2018
    Posts
    209

    1 Not allowed!
    Also, an update: the database for grand prix judges for the 2018-2019 season is complete, and may be found here.

    The database linked in the opening post is now in the process of being updated to include Senior B data. Overall, my sense was that Senior B judging was (somewhat predictably) less biased than Grand Prix judging, so I expect this new data to bring down the bias numbers somewhat. But we'll see. Unfortunately, this update process will likely be very slow and the order of competitions added will be somewhat arbitrary. But I'll try to get all of them eventually. In the meanwhile, the "Competitions" page will tell you which competitions have been added in so far. If anyone would like to help put together competition data, you may message me and it will be much appreciated.

  12. #52
    GS Supporter
    Join Date
    Aug 2011
    Posts
    8,502

    2 Not allowed!
    I'm no numbers cruncher, so whilst I've been thinking about this all season long, particularly since doing the ID PBPs, I haven't written about it. But the new judging system has ONE OVERRIDING BIAS that I find deplorable: privileging GOE over levels. Essentially the skating PTB are telling its members: you're better off doing the easier elements as long as you do it well. I'm not sure where encouraging people NOT to challenge themselves is a recipe for success in the long term. I haven't seen it work well as a long term strategy ever.

    For ID, the new scoring is a disaster. Right now the most important elements appear to be the three choreographic ones in the FD. Since they're all level 1, the tech panel has absolutely no say in judging them. It's all subjective opinions. And we know how well giving judges the chance to manipulate the scores based on "artistic" or "program components" has gone in the past. And it took about 2 competitions for the shenanigans to begin.

    I'm not hopeful for this quad at all. I mean AT ALL.

  13. #53
    On the Ice
    Join Date
    Mar 2018
    Posts
    209

    0 Not allowed!
    Quote Originally Posted by WeakAnkles View Post
    I'm no numbers cruncher, so whilst I've been thinking about this all season long, particularly since doing the ID PBPs, I haven't written about it. But the new judging system has ONE OVERRIDING BIAS that I find deplorable: privileging GOE over levels. Essentially the skating PTB are telling its members: you're better off doing the easier elements as long as you do it well. I'm not sure where encouraging people NOT to challenge themselves is a recipe for success in the long term. I haven't seen it work well as a long term strategy ever.

    For ID, the new scoring is a disaster. Right now the most important elements appear to be the three choreographic ones in the FD. Since they're all level 1, the tech panel has absolutely no say in judging them. It's all subjective opinions. And we know how well giving judges the chance to manipulate the scores based on "artistic" or "program components" has gone in the past. And it took about 2 competitions for the shenanigans to begin.

    I'm not hopeful for this quad at all. I mean AT ALL.
    It's notable that despite there being fewer points available in ID, it has the worst bias numbers out of all the disciplines including men's, which has a ton of GOE points available from quads as well as extra high PCS factoring.

    I think the GOE changes were largely aimed at fixing the issue with there being too little of a penalty for falling on hard jumps in ladies/pairs but especially men's, and also with ugly quads being rewarded too much over well-done triples, but it has the unintended consequence of throwing off the balance of subjectivity versus objectivity in ID in order to keep the scoring consistent between disciplines.

Page 3 of 3 FirstFirst 1 2 3

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •