Can we use machine learning to score elements | Page 3 | Golden Skate

Can we use machine learning to score elements

moriel

Record Breaker
Joined
Mar 18, 2015
When you're typing blindly, the keys stay in exactly the same place the whole time.

How are you imagining that the touchscreen or touchpad for GOE input would look (or feel) to accommodate all possible positive and negative criteria for each different kind of element?

Would it update each time the skater performs a new move and the technical panel calls that move?

Or would all the possible criteria for all different element types be on the screen at all times and the judge would have to type in the code or element number for the element they're scoring before finding the appropriate boxes by touch?

See moriel's post 14 in this thread for a proposed list of bullet points that could be assigned to the AI to determine, perhaps in conjunction with a human tech panel, and 11 remaining jump-related bullet points reserved for judges.

(And that's assuming that AI really could immediately determine everything in the first list in that post. At least at first, I'd expect many of them would usually need to be confirmed by the tech panel with video review after the program. Or else they would be left for judges to determine, making the list of judges bullet points longer.)

For example, one goe aspect / screen, where you press a key if it applies, and another key if it does not.

Also, as a side note, all or most of the bullet points listed in my post are currently evaluated by human judges. While one may argue that 11 is too much, it is still better than whatever they have to do now, which is evaluate around 30 bullet points for each jump.
Also, in my post, i point out that not all of those 11 apply to each jump, and several are mutually excluding, which means the number of actual bullet points presented to the judge will be even lower.
 

gkelly

Record Breaker
Joined
Jul 26, 2003
I don't understand how this would work in real life in real time.

Is a live tactile touchscreen with updated content on the screen even possible at this time?

Are judges supposed enter all the relevant bullet points in real time, before any tech panel reviews for rule vetting (even if the AI identifies most of the physical qualities of the jump)?

Are the GOEs supposed to update live, as much as possible, to allow for a live scoretracker?

What happens when a the time interval between the skater completing element A and beginning element B (and executing enough of it to be called) is smaller than the amount of time it takes for any or all judges to input each of the bullet points they want to apply to element A. What's to prevent them from entering element A bullets when the screen has already changed to element B.

Or if all elements executed remain plus all their data entry points remain on the screen, how does a judge make sure they're inputting the correct bullets for element A in the boxes or buttons for element A and not for element B? (This is already an issue when judges have to choose one score from a linear array that never changes from element to element. How much more will data entry be an issue if they have to choose multiple boxes from one of several arrays on the screen at the same time?)

Have you tried mocking up what you think the interface will look (or feel) like from moment to moment during an ongoing program and see how the data input will work?

Are you familiar with how the current system works, where and why data input errors are possible and how multiplying the number of inputs would likely affect accuracy?
 

katymay

Medalist
Joined
Mar 7, 2006
Before you call me crazy, I think machine learning can actually helping scoring quite a bit, which has been plagued with inconsistency and subjectivity. Taking a video of an element with pixels, and outputting a score is well within the realm of possibility of technologies right now. Plus with the added bonus of being able to get precise measurements on things like rotation, height, speed, etc. will help. Bias is certainly still possible and machines do have issues (for instance the color of your costume might actually affect the pixels of your video and thus the score) but on the whole it will most certainly have less bias than human judges.

Just training a model right now based on past scores and bullet point criteria etc., and using it on some videos of elements I reckon it will do quite well. Perhaps more "accurate" than real judges. Again crazy idea but I think it has its merits. I do think the biggest barrier is just humans getting over the fact that things like this machines can do quite well. (maybe not PCS yet, but it can get some metrics like speed and transitions too)

Japan is doing something like this for the Olympics, gymnastics. But the gymnastics community is skeptical.
 

lanceupper1114

On the Ice
Joined
Jun 3, 2018
If we want to see investment in new equipment and technology in figure skating to improve scoring consistency, it might be most beneficial to make the investment in lie detectors attached to electric-shock collars, which judges would have to wear. Zap!!
 

SarahSynchro

Record Breaker
Joined
Mar 7, 2014
Country
Canada
Perhaps this idea could piggyback off of Dartfish’s platform? It’s purpose is to analyze the athlete’s technique and execution of their sport’s required elements, figure skaters included. Essentially, Dartfish already has the basic technology needed for computerized judging.
 

tofuetoffee

Rinkside
Joined
Jun 12, 2017
Bumping up this thread because of something exciting from Japanese Junior Nats:

https://twitter.com/doubleflutz/status/1066676099101290496

fuji tv has developed new software named "Ice Scope" which tracks jump distance, height, and speed, and shows the data during the replays.
they debuted on the junior jnats broadcasts. really interested to see it applied to other comps!

https://youtu.be/E22bwsenntU?t=311



New technology used in a sport like figure skating is always a plus imo because it at least shows that there is an interest in research.

Perhaps the numbers are not relevant to scoring now, but is a step towards the right direction instead of just depending on the eye or how something looks.
 

Edwin

СделаноВХрустальном!
Record Breaker
Joined
Jan 5, 2019
Any updates on this?

Was this 3D laser tracker system also used in some recent JPN competitions where the height and flight of skaters was being measured in real time?

Given the both the controversy over and practical results (offsides, goals, penalties) achieved by the use of Video Assisted Referee in the last Football/Soccer World Championships and Hawkeye in tennis, can this kind of 'objective machine evaluation' be used to rule over challenges made by skaters over the technical content of their score?
I am new here, and don't even know if a contestant can protest/challenge his/her technical score under ISU rules. The ISU rule book is a hefty read and requires good knowledge of the sport.

Do we fans even want this? For it will kill a lot of discussion over <, * and ! markings, LOL
 

gkelly

Record Breaker
Joined
Jul 26, 2003
Under the current rules, skaters are not permitted to challenge rotation or edge calls (or level calls) made by the technical panel.

They are only allowed to challenge if the panel called a wrong element (e.g., toe loop instead of loop or double instead of triple), and there is a relatively narrow window of time in which they are allowed to do so.

Are you proposing that skaters should be allowed to challenge "field of play" calls by the panel? That would open several other cans of worms?

Or just that implementing technological assistance in making the calls would increase confidence that the calls were made correctly?
 

Edwin

СделаноВХрустальном!
Record Breaker
Joined
Jan 5, 2019
Under the current rules, skaters are not permitted to challenge rotation or edge calls (or level calls) made by the technical panel.

They are only allowed to challenge if the panel called a wrong element (e.g., toe loop instead of loop or double instead of triple), and there is a relatively narrow window of time in which they are allowed to do so.

Are you proposing that skaters should be allowed to challenge "field of play" calls by the panel? That would open several other cans of worms?

Or just that implementing technological assistance in making the calls would increase confidence that the calls were made correctly?

The second, we don't want more delays in scores and the next contestant having to wait too long for the protest to be dealt with by the master referee. Especially in this era where TV schedules and revenues rule so many a sports competition.
I am only acquainted with the procedure in gymnastics' highest level championships: a challenge of the technical content score costs a substantial sum and the money is forfeit when the protest is rejected.
Colour of medals have been decided by protests on more than one occasion. But please, no can of worms, only legitimate protests.

However, given the 1800 square meter ice surface of a competition rink, the technology of 3D scanners' full coverage needed would be quite challenging and probably expensive in order to be 100% fail safe without any intrusive additions to the boots of the skaters or TV or spectators, so no rotating laser beams for instance. It would be cool to have flight trajectories plotted for the replays though and perhaps those videos and data can even help trainers to understand the biomechanics of jumps.
 

gkelly

Record Breaker
Joined
Jul 26, 2003
It would be cool to have flight trajectories plotted for the replays though and perhaps those videos and data can even help trainers to understand the biomechanics of jumps.

Many coaches do use video-based tools such as Dartfish to analyze jump mechanics.

This would be something that would be used at home during training, not so much at a competition venue. So the technology would need to be available at home training rinks. More feasible for rinks/coaches with many serious skaters.

The coaches are looking for answers to the question of "What exactly is the skater doing?" and "What do they need to do differently to achieve a successful (or better) jump?" And they can position the cameras and the jump takeoffs as needed to get a good view.

Tech panels are looking just at "What did the skater actually achieve?"
And they need to be able to answer that question, as well as possible, for jumps at all possible locations and angles on the ice surface.
 

Edwin

СделаноВХрустальном!
Record Breaker
Joined
Jan 5, 2019
Tech panels are looking just at "What did the skater actually achieve?"
And they need to be able to answer that question, as well as possible, for jumps at all possible locations and angles on the ice surface.

Judges are all on one long side of the rink, right? And skaters tend to do throw their jumps and cascades in the far corners, hoping perhaps some small details might get missed.
Do judges and the tech panel have full access to the broadcast streams from all camera's and their own jog and shuttle knobs to replay a stream at will to accurately evaluate the execution of an element?
Or is it just following with the eye, memorising and comparing with previous performances, while scribbling quick notes in some kind of judge's shorthand and then calculate their score afterwards before keying it into the scoring system? Skaters are required to submit their program content beforehand, I have understood correctly?
 

moriel

Record Breaker
Joined
Mar 18, 2015
Judges are all on one long side of the rink, right? And skaters tend to do throw their jumps and cascades in the far corners, hoping perhaps some small details might get missed.
Do judges and the tech panel have full access to the broadcast streams from all camera's and their own jog and shuttle knobs to replay a stream at will to accurately evaluate the execution of an element?
Or is it just following with the eye, memorising and comparing with previous performances, while scribbling quick notes in some kind of judge's shorthand and then calculate their score afterwards before keying it into the scoring system? Skaters are required to submit their program content beforehand, I have understood correctly?

i think they have video from a single angle.
 

[email protected]

Medalist
Record Breaker
Joined
Mar 26, 2014
Figure skating is an expensive sport for athletes even in Russia where the State partially covers expenses. And there is little new money. Sambo was promised a new rink after the Olympics - I have not heard that they started to build one. All this fancy technology will be at whose cost? I have not heard of big sponsor money coming. Athletes and their parents? No, thank you.

My answer is quite simple - make tech calls more lenient. It's just a bad joke when Satoko can get for the same performance no calls at all vs. 3 URs depending on the panel. Who cares about lutz edge but a small group of KK like purists who are as biased as others? Even with the wrong edge lutz entry is different from flip. I can easily tell you if the skater did a lutz or a flip seating at the stands. Neither I nor 99% of viewers cannot tell without a slow mo if the edge was wrong especially not having the correct angle of view. Mao, Galustyan, Sotskova - the first names to come of those skaters who received scores the public did not understand. I remember Galustyan's "Anastasia" program. She was deeply involved in performance in her light yellow dress. And she was clean with 3Lz-3T. I remember her tears when she received what, 45? All jumps were named UR.

It is impossible to be universally strict with everyone for various reasons. Then they should be universally lenient not destroying skaters' tech scores for things that are invisible to a "naked eye". And if figure skating survives, may be in 50 years or so, there will be affordable techology that can make calls 100% impartial and correct.
 

gkelly

Record Breaker
Joined
Jul 26, 2003
Judges are all on one long side of the rink, right?

Yes.

And skaters tend to do throw their jumps and cascades in the far corners, hoping perhaps some small details might get missed.

That might be one reason.

Another is that for many of the jumps, especially toe jumps and especially quads, they need the full length of the rink to build up enough speed. (Toe jumps are usually entered from straighter approaches and edge jumps are more often approached on a circle.)

Also, skaters tend to practice most often on sessions with more than 5 and sometimes as many as 30 other skaters (and coaches) on the ice at the same time. So practice session traffic patterns tend to favor putting jumps toward the ends of the ice and spins toward the center. For edge jumps and lower-revolution jumps it would theoretically be possible to reverse that, but lots of spins filling the ends of the ice would make it difficult to practice triple and quad toe jumps effectively.

Do judges and the tech panel have full access to the broadcast streams from all camera's and their own jog and shuttle knobs to replay a stream at will to accurately evaluate the execution of an element?

No. They have access to one official camera angle usually from a camera that is placed near the technical panel.
The tech panel has access to slow motion, but I'm not sure if the judges do.

Or is it just following with the eye,

Yes.

memorising and comparing with previous performances,

They're not supposed to be comparing with previous performances. But knowing when to expect something unusual so they know when not to look away for a moment, based on having seen practices or previous performances, can be useful.

while scribbling quick notes in some kind of judge's shorthand

Yes.

and then calculate their score afterwards before keying it into the scoring system?

There can be some "calculation" of individual GOE scores if it's necessary to add up positive bullet points or negative reductions and to balance them out.

I'm not sure whether deciding on each program component score could be considered calculation. Maybe it depends on the kind of thought processes each judge uses to arrive at their scores.

But if by "calculating" you mean remembering/estimating the base values of the elements and adding up the final factored values of all the GOEs and PCS they award, that's not what they're supposed to be doing. Maybe remembering those kinds of values and doing the mental arithmetic comes naturally to some judges, but for most it would probably distract them from actually evaluating each element and program component on its own merits as they're supposed to.

Skaters are required to submit their program content beforehand, I have understood correctly?

They are, but that's just to make things easier for the technical panel to know what to expect. It's perfectly legal for skaters to deviate from the program plan they submitted.
 

gkelly

Record Breaker
Joined
Jul 26, 2003
I'm currently copyediting an article in which the authors are promoting their newly invented program/algorithm for a scientific measuring task completely unrelated to figure skating.

To explain why they believe their invention is needed, they review a few existing algorithms that could be used for that task but argue that they are unreliable or cumbersome and do not reach the level of trained human experts.

At the end, of course, they argue why their version can do what it was designed for and might be able to be useful for additional tasks.

I couldn't help but think of this thread.
 

Edwin

СделаноВХрустальном!
Record Breaker
Joined
Jan 5, 2019
So we’re probably not going to see any machine assisted scoring any time soon.
The supplier of the official timekeeping and scoring system affiliated to ISU will most likely provide the system for continental and world championships. And even for those like Longines, Rolex, Timex, Citizen, Seiko etc it will be a huge financial outlay.

Frame by frame time lapse comparison sequences of jumps etc are already possible with current broadcasting systems where everything is in the digital domain.
 
Top