Page 1 of 2 12 LastLast
Results 1 to 20 of 22

Thread: Nationalistic bias in figure skating judging, 2018-present

  1. #1
    On the Ice
    Join Date
    Mar 2018
    Posts
    308

    8 Not allowed!

    Nationalistic bias in figure skating judging, 2018-present

    I have put together some quite comprehensive statistics on nationalistic bias in figure skating judging, which includes every single senior level ISU competition. Unfortunately, the write-up is a bit unwieldy for a forum post as it's quite long and contains graphs and so forth, so I will direct you to this website, where you can read the whole thing.

    Here are the top line conclusions:

    The database currently contains 312 judges, 177 of which I examined for nationalistic bias (the other judges didn’t have an extensive enough judging record), 92 of which showed statistically significant evidence (p<0.05) of nationalistic bias. Of those 74 showed strong (p<0.01) evidence.
    Biased judging was widespread across most federations (shout out to Australia the one large-ish federation that was pretty clean), but some federations (notably Russia, France, Germany, and Italy) were particularly bad, though not always in the same way.

    Considering that the judges play a quite significant role in the careers of young skaters, I thought I would share this here in order to open up a conversation. Of course, I'm sure you guys have many questions as well, so feel free to ask away.

  2. #2
    Rinkside
    Join Date
    Oct 2011
    Posts
    79

    0 Not allowed!
    Thank you for the spreadsheet. Just a little thing - you should check the nationalities, some of the "Japanese " judges are Chinese and vice versa and some other nationalities are probably wrong, too.

    AUS judges are exceptional, we could see it especially during ice dance at Worlds, where they didn't get the memo that Polish team is suddenly top ten material.

  3. #3
    On the Ice
    Join Date
    Mar 2018
    Posts
    308

    0 Not allowed!
    Quote Originally Posted by Rashela View Post
    Thank you for the spreadsheet. Just a little thing - you should check the nationalities, some of the "Japanese " judges are Chinese and vice versa and some other nationalities are probably wrong, too.

    AUS judges are exceptional, we could see it especially during ice dance at Worlds, where they didn't get the memo that Polish team is suddenly top ten material.
    Hm, that sounds like the nationality-switched sheet rather than the original sheet. Let me check that they didn't get mixed up somehow. Could you tell me which judge(s) in particular?

    Edit: Are you sure the sheets aren't mixed up on your end? There's two versions linked--the one that is normal is called "Judges' sheet", but the other one is called "nationality switches", where every judge got switched to a different nationality for comparison reasons. The far right column tells you which nationality got switched for what.

  4. #4
    Rinkside
    Join Date
    Oct 2011
    Posts
    79

    0 Not allowed!
    Yeah, I am sorry, you are right.

  5. #5
    On the Ice medoroa's Avatar
    Join Date
    Dec 2017
    Posts
    205

    3 Not allowed!
    "(This example from the 2018 ACI men’s free skate is completely random, obviously.)"

    I can't tell if this is tongue-in-cheek or if you mean it. Could you elaborate?

    Also, I don't seem to see you've mapped this onto how big the staple of judges and elite skaters are in each federation. My country has no judges on that list, but this is probably more because we have few judges at an international level in the first place, and they also have no elite skaters to give biased scores to (Australia is not a "big fed" to me at all which I think is more likely an explanation for their lack of "bias" -- and Russia's size might also explain Russia's large number of "biased judges", although it might also not). Apologies if you mentioned that somewhere and I missed it.

    I'm also a little unsure what you're trying to conclude or achieve. National bias exists, this is fact. It has always been considered a fact, especially by the ISU. This is why there are judges from several different countries on a judging panel and why they are drawn randomly; this is an officially-sanctioned method to offset this bias, proving that ISU knows (and has admitted!) national bias exists. I mean, they are not dumb, regardless of how frustrating I sometimes find them. So in light of that, it would be more interesting (to me) to see how the biases fall out when the lowest and highest scores have been omitted. That would be a constructive way to look at whether ISU's method to offset bias and errors is working within reasonable limits.

  6. #6
    On the Ice
    Join Date
    Mar 2018
    Posts
    308

    3 Not allowed!
    Quote Originally Posted by medoroa View Post
    "(This example from the 2018 ACI men’s free skate is completely random, obviously.)"

    I can't tell if this is tongue-in-cheek or if you mean it. Could you elaborate?
    It's tongue in cheek. It is the one instance in which personal bias entered this project--I like Yuzu so I was going to use his scores as an example, especially one where the US judge scored him the lowest by a heftyish margin. It doesn't really matter though, the example is just there to demonstrate how the math works--the formula is the same for everyone, which you can confirm by going to the competition sheets. And in fact the US judge used in that example, Doug Williams, did not show statistically significant evidence of pro-US bias in general, so I made no accusations about him. The Japanese judge, Miwako Ando, on the other hand, did make it on to the list (although only by the least strict standard of significance).

    Hey, I had to do something to entertain myself after dealing with so much dry statistics

  7. #7
    On the Ice medoroa's Avatar
    Join Date
    Dec 2017
    Posts
    205

    0 Not allowed!
    Quote Originally Posted by Shanshani View Post
    It's tongue in cheek. It is the one instance in which personal bias entered this project--I like Yuzu so I was going to use his scores as an example, especially one where the US judge scored him the lowest by a heftyish margin. It doesn't really matter though, the example is just there to demonstrate how the math works--the formula is the same for everyone, which you can confirm by going to the competition sheets. And in fact the US judge used in that example, Doug Williams, did not show statistically significant evidence of pro-US bias in general, so I made no accusations about him. The Japanese judge, Miwako Ando, on the other hand, did make it on to the list (although only by the least strict standard of significance).
    Okay, thank you. But it matters to me, since I would like to look at the bias of the fans who put together a report on the bias of the judging.

    eta: And sorry I keep editing my replies, I'm mulling it over in my head as I type.

  8. #8
    On the Ice
    Join Date
    Mar 2018
    Posts
    308

    4 Not allowed!
    Quote Originally Posted by medoroa View Post
    Okay, thank you. But it matters to me, since I would like to look at the bias of the fans who put together a report on the bias of the judging.
    I suppose, but to be honest, I find this kind of criticism a bit lazy. As I mentioned in the comments on my blog post, I have made the entirety of my methodology and mathematics transparent to everyone who bothers to put in the effort. If you have some criticism about the project or worry about bias, you should connect it to some aspect of how the project was done. There's really no way for me to answer the criticism "You like Yuzu, so you're biased" or "You're American, so you're biased" because there is no relationship between those facts and how the project was executed, other than the fact that I decided to use one skater's scores rather than another in my explanation of how the mathematics works, which has no bearing on the results. Nor is there anything I could do (other than stop liking Yuzu or stop being American, I suppose) to answer that criticism, because it has nothing to do with how the results were found.

    Again, the same formulas were applied to everyone equally, and determinations of bias were made by comparing a judge to the other judges, and at no point did my judgment about what x skater ought to have scored enter in to it. Someone who likes Nathan, or who loves the Russian girls, or whatever will get the same results if they follow the same steps.

  9. #9
    On the Ice medoroa's Avatar
    Join Date
    Dec 2017
    Posts
    205

    1 Not allowed!
    Quote Originally Posted by Shanshani View Post
    I suppose, but to be honest, I find this kind of criticism a bit lazy. As I mentioned in the comments on my blog post, I have made the entirety of my methodology and mathematics transparent to everyone who bothers to put in the effort. If you have some criticism about the project or worry about bias, you should connect it to some aspect of how the project was done. There's really no way for me to answer the criticism "You like Yuzu, so you're biased" or "You're American, so you're biased" because there is no relationship between those facts and how the project was executed, other than the fact that I decided to use one skater's scores rather than another in my explanation of how the mathematics works, which has no bearing on the results. Nor is there anything I could do (other than stop liking Yuzu or stop being American, I suppose) to answer that criticism, because it has nothing to do with how the results were found.

    Again, the same formulas were applied to everyone equally, and determinations of bias were made by comparing a judge to the other judges, and at no point did my judgment about what x skater ought to have scored enter in to it. Someone who likes Nathan, or who loves the Russian girls, or whatever will get the same results if they follow the same steps.
    I didn't accuse you of anything. It just stood out like a sore thumb in a post which was otherwise well-presented.

    Everyone will get the same numbers with the same steps, but conclusions are always based on how a human being reads the numbers (I've worked at an ad agency, I know). It's relevant information for me if you decide to expand on your conclusions or (even more importantly) you intend to propose changes. That's seldomly, if ever, unmotivated by some form of agenda.

  10. #10
    Bona Fide Member jillredhand's Avatar
    Join Date
    Oct 2017
    Posts
    2,193
    Country: Canada

    3 Not allowed!
    Quote Originally Posted by medoroa View Post

    I'm also a little unsure what you're trying to conclude or achieve. National bias exists, this is fact. It has always been considered a fact, especially by the ISU. This is why there are judges from several different countries on a judging panel and why they are drawn randomly; this is an officially-sanctioned method to offset this bias, proving that ISU knows (and has admitted!) national bias exists. I mean, they are not dumb, regardless of how frustrating I sometimes find them. So in light of that, it would be more interesting (to me) to see how the biases fall out when the lowest and highest scores have been omitted. That would be a constructive way to look at whether ISU's method to offset bias and errors is working within reasonable limits.
    Shanshani does address this idea in the report; the methodology shows that it's not inevitable and intrinsic that all judges are biased in favor of their own skaters. There are many judges who show no statistically significant bias in favor of their own home skaters, or show some bias of a lesser degree. While it's true that some degree of nationalistic bias is natural and would probably be impossible to completely eliminate, analyses like this should make it possible to address the most egregious cases. Just because perfect impartiality may not be humanly possible does not mean that improvements cannot be made.

  11. #11
    On the Ice medoroa's Avatar
    Join Date
    Dec 2017
    Posts
    205

    1 Not allowed!
    Quote Originally Posted by jillredhand View Post
    Shanshani does address this idea in the report; the methodology shows that it's not inevitable and intrinsic that all judges are biased in favor of their own skaters. There are many judges who show no statistically significant bias in favor of their own home skaters, or show some bias of a lesser degree. While it's true that some degree of nationalistic bias is natural and would probably be impossible to completely eliminate, analyses like this should make it possible to address the most egregious cases.
    That remains to be seen. The text itself says that as the season progresses, there will be more judges added to the list while (probably) none will be removed. The text itself also says that it does not account for all methods of possible biased judging.

    Quote Originally Posted by jillredhand View Post
    Just because perfect impartiality may not be humanly possible does not mean that improvements cannot be made.
    I agree! Which is why I suggested that to me, a more constructive use of this statistic was to measure whether the fail-saves already in place were functioning satisfactorily (for some measure of satisfactory).

  12. #12
    Bona Fide Member jillredhand's Avatar
    Join Date
    Oct 2017
    Posts
    2,193
    Country: Canada

    1 Not allowed!
    Quote Originally Posted by medoroa View Post
    I didn't accuse you of anything. It just stood out like a sore thumb in a post which was otherwise well-presented.

    Everyone will get the same numbers with the same steps, but conclusions are always based on how a human being reads the numbers (I've worked at an ad agency, I know). It's relevant information for me if you decide to expand on your conclusions or (even more importantly) you intend to propose changes. That's seldomly, if ever, unmotivated by some form of agenda.
    I do actually agree with you here. Shanshani, I am extremely impressed with your work here, but that tongue in cheek example did leave a questionable impression on me too. I do trust your tranaparity and the objectivity of your methodology, but just from an optics perspective, that example and your tongue in cheek statement will probably result in people drawing unnecessary, even if wrong, adverse inferences.

  13. #13
    Bona Fide Member jillredhand's Avatar
    Join Date
    Oct 2017
    Posts
    2,193
    Country: Canada

    0 Not allowed!
    Quote Originally Posted by medoroa View Post

    I agree! Which is why I suggested that to me, a more constructive use of this statistic was to measure whether the fail-saves already in place were functioning satisfactorily (for some measure of satisfactory).
    I also agree, I'd like to know if discarding the top and bottom scores always (or mostly) gets rid of the biased judge's score. Though even if that failsafe is effective, that still doesn't mean that it wouldn't be preferable to reduce the degree of bias the judges give anyway. Discarding the high and low scores isn't meant just to eliminate national biases, it's also to make up for natural human error and disagreement in the judging.

  14. #14
    Bona Fide Member
    Join Date
    Jan 2013
    Posts
    12,012
    Country: Canada

    0 Not allowed!
    Quote Originally Posted by medoroa View Post
    Okay, thank you. But it matters to me, since I would like to look at the bias of the fans who put together a report on the bias of the judging.

    eta: And sorry I keep editing my replies, I'm mulling it over in my head as I type.
    But why would fans who put together a bias of judging ever be biased themselves?

    This is a really important point to make when seeing the source of these criticisms. There's a difference between the person presenting all data and trends and those who pick the trends that criticize the scoring in a manner that props up their faves and condemns their fave's rivals. Of course there's national bias (sometimes blatant - I recall that one Chinese judge), but sometimes skaters are marked lower or higher based on the personal preference of the judges. Important to note that there's a panel, and it's not ordinals anymore, and highs/lows are thrown out, so usually it'll balance out. Although I'm not saying that absolves bad judging but nationalistic bias certainly has less of an effect now IMO than it did under 6.0.

  15. #15
    Bona Fide Member
    Join Date
    Feb 2010
    Posts
    8,140

    0 Not allowed!
    As long as the national federations play any part in training, nominating, and/or selecting judges for international competitions, bias will exist. From the federation's perspective, you want your skaters to place as high as possible and so you'd obviously want judges who will score those athletes favorably (and probably deflate the scores of top competitors).

  16. #16
    GS Supporter ladyjane's Avatar
    Join Date
    Jun 2012
    Posts
    1,120
    Country: Netherlands

    3 Not allowed!
    Sorry, I didn't check the maths. I'm quite good at it, that's not the point. My point is something different: you see I understand a bit of National Bias in the sense that a judge would give his/her own skaters too many points. The judge has seen them throughout the year, maybe he/she has given the skater advice, whatever. That's okay. Maybe not totally fair, but I get that. I probably would do it myself. It's not always intentional bias, it's just you've seen a skater/couple improve throughout the season (after test skates, champs camps, and all these things) and you'll just see them as being better than another might do. As someone stated earlier, the outliers are deleted anyway. However, the problems start when you give your own skaters high points and deliberately lower their competitors's points. Like what happened at the Olympics with the Chinese judge at the Pairs competition. I don't think the judge got penalised for giving the wonderful Chinese pair (too) high points, but for deliberately giving the German pair low scores. Personally (so, this is just my viewpoint, please bear that in mind when reading this) I can totally understand one might score one's own skaters too high, it's when one also gives too low scores to the competitors when it's really problematic. Such things might have been easier to detect under the 6.0 system. It's similar with reputation. Judges shouldn't take that into account, but of course they do. They don't live in a bubble and hear/see nothing.

    Don't get me wrong, I admire the effort, and I liked to read the AUS judges are mostly unbiased. But as long as Nationalism and Patriotism remain, so will National bias. It can only be checked as much as possible. But never totally eliminated for the reasons above. Rather human I think.

  17. #17
    Bona Fide Member
    Join Date
    Jul 2003
    Posts
    6,785

    1 Not allowed!
    Agreed. There is a difference between biased judging and dishonest judging.

    We all have biases, which may be based on nationality or familiarity with specific skaters' careers or personal lives or personal preferences about skating style or beliefs in what should count most highly in skating results or any number of other factors. Some of which we are aware of and might readily admit to or consciously try to overcome if we were trying to judge honestly, and others which are so ingrained in our beliefs that we are unaware of them.

    That's true for judges and other officials, fans, skaters and coaches, commentators, etc. (And TV commentators and producers may intentionally aim to present home country skaters in the best possible light, downplaying the positive qualities of their closest rivals.)

    Only the officials can have a direct effect on the scoring of the event. Are they trying to manipulate the results to get their federations' skaters to the highest possible place in the standings? That's cheating.

    Or are they judging as honestly as they know how, without conscious awareness that their personal preferences and personal histories are shaping how they perceive the various performances?

    That's bias, and we're all susceptible to it.

  18. #18
    Bona Fide Member
    Join Date
    Dec 2014
    Posts
    2,654

    1 Not allowed!
    Quote Originally Posted by gkelly View Post
    Agreed. There is a difference between biased judging and dishonest judging.
    Unfortunately, I find a lot of current judging downright dishonest rather than merely biased, especially on top. Which makes me slowly lose interest in the sport, to be honest.

  19. #19
    GS Supporter ladyjane's Avatar
    Join Date
    Jun 2012
    Posts
    1,120
    Country: Netherlands

    0 Not allowed!
    Quote Originally Posted by gkelly View Post
    Agreed. There is a difference between biased judging and dishonest judging.

    We all have biases, which may be based on nationality or familiarity with specific skaters' careers or personal lives or personal preferences about skating style or beliefs in what should count most highly in skating results or any number of other factors. Some of which we are aware of and might readily admit to or consciously try to overcome if we were trying to judge honestly, and others which are so ingrained in our beliefs that we are unaware of them.

    That's true for judges and other officials, fans, skaters and coaches, commentators, etc. (And TV commentators and producers may intentionally aim to present home country skaters in the best possible light, downplaying the positive qualities of their closest rivals.)

    Only the officials can have a direct effect on the scoring of the event. Are they trying to manipulate the results to get their federations' skaters to the highest possible place in the standings? That's cheating.

    Or are they judging as honestly as they know how, without conscious awareness that their personal preferences and personal histories are shaping how they perceive the various performances?

    That's bias, and we're all susceptible to it.
    Exactly! We're all biased. It's how we handle it that counts. I like your differential: biased judging (which everyone does) vs dishonest judging (which nobody has to do).

  20. #20
    Bona Fide Member
    Join Date
    Feb 2010
    Posts
    8,140

    1 Not allowed!
    Quote Originally Posted by Rissa View Post
    Unfortunately, I find a lot of current judging downright dishonest rather than merely biased, especially on top. Which makes me slowly lose interest in the sport, to be honest.
    Honestly, so much of judging is subjective that I rarely find any scores to be blatantly dishonest. Judges scoring the same element differently by 3 or 4 points in GOE is not terribly uncommon, but those differences aren't necessarily attributable to cheating because the scores for all the judges will usually span that entire range.

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •