How AI Will Help Keep Time at the Tokyo Olympics

Timing of Olympic races has not always been sophisticated. On April 10, 1896, some 17 runners competed in the very first Olympic marathon. The course ran approximately 40 km, and of the athletes representing five different nations, it was a Greek water carrier, Spyridon Louis, who eventually emerged victorious, winning in two hours, 58 minutes, and 50 seconds.

How do we know this impressive statistic? The very same stopwatch set running by the judge in Marathon at the start of that historic race was then delivered ahead of the runners—by bicycle, no less—to record the momentous time as Louis crossed the finish line in Athens just shy of three hours later.

Things have, as you would expect, improved much in the last 125 years. Omega, which has provided the official timekeeping services to the Olympics since 1932, has been tasked with developing many of these advances. Perhaps most notable was the first photoelectric cell photo-finish camera, “Magic Eye,” introduced in 1948 in London, created as a joint effort between Omega and the British Race Finish Recording Company. Prior to this, it was the human eye married with the push of a stopwatch that was used to determine which athlete had first crossed the finish line. 

The technology was, however, viewed with caution. Hand-stopping was still used for official times in 1948. It was another 20 years before the photo-finish camera's use was widespread. In Mexico 1968, however, the Games eventually switched to being exclusively timed by electronics, with photo-finish cameras capturing 10 new world records. Though despite this technological competency, some 45 manual timekeepers were sent to Mexico just in case the hardware failed.

Since then, the issue for Omega Timing has not only been further increasing accuracy (Omega introduced its Quantum Timer in London 2012 with a resolution of one millionth of a second, 100 times greater than previous devices), but also developing ways to monitor in real time the Olympics' ever-growing canon of sports represented. Skateboarding, sport climbing, and surfing, for example, are included in the new additions sanctioned by the International Olympic Committee for 2021 in Tokyo.

Indeed, sport climbing, as well as swimming, athletics, gymnastics, and equestrian, are among the events getting a new or completely revamped timing treatment this year. But perhaps the most interesting is how Omega has spent four years training its in-house artificial intelligence to learn beach volleyball.

“In volleyball, we're now using cameras with computer vision technologies to track not only athletes, but also the ball,” says Alain Zobrist, head of Omega Timing. “So it's a combination where we use camera technology and artificial intelligence to do this.”

Omega Timing's R&D department comprises 180 engineers, and the development process started with positioning systems and motion sensor systems in-house, according to Zobrist, in 2012. The goal was to get to a point where, for multiple sports at the 500-plus sports events it works on each year, Omega could provide detailed live data on athlete performance. That data would also have to take less than a tenth of a second to be measured, processed, and transmitted during events so that the information matches what viewers are seeing on screen. 

With beach volleyball, this meant taking this positioning and motion technology and training an AI to recognize myriad shot types—from smashes to blocks to spikes and variations thereof—and  pass types, as well as the ball's flight path, then combine this data with information gleaned from gyroscope sensors in the players clothing. These motion sensors let the system know the direction of movement of the athletes, as well as height of jumps, speed, etc. Once processed, this is all then fed live to broadcasters for use in commentary or on-screen graphics.

According to Zobrist, one of the hardest lessons for the AI to learn was accurately tracking the ball in play when the cameras could no longer see it. “Sometimes, it's covered by an athlete's body part. Sometimes it's out of the TV frame,” he says. “So, the challenge was to track the ball when you have lost it. To have the software predict where the ball goes, and then, when it appears again, recalculate the gap from when it lost the object and got it back, and fill in the [missing] data and then continue automatically. That was the one of the biggest issues.”

It's this tracking of the ball that is crucial for the AI to determine what is happening during play. “When you can track the ball, you will know where it was located and when it changed direction. And with the combination of the sensors on the athletes, the algorithm will then recognize the shot,” Zobrist says. “Whether it was a block or a smash. You will know which team and which player it was. So it's this combination of both technologies that allows us to be accurate in the measurement of the data.”

Omega Timing claims its beach volleyball system is 99 percent accurate, thanks to the sensors and multiple cameras running at 250 frames a second. Toby Breckon, professor in computer vision and image processing at Durham University, however, is interested to see if this stands up during the Games—and, crucially, if the system is fooled by differences in race and gender. 

“What has been done is reasonably impressive. And you would need a large data set to train an AI on all the different moves,” Breckon says. “But one of the things is accuracy. How often does it get it wrong in terms of those different moves? How often does it lose track of the ball? And also if it works uniformly over all races and genders. Is that 99 percent accuracy on, say, the USA women's team and 99 percent accuracy on the Ghanaian women's team?”

Zobrist is confident, and explains that while it may have been easier to call in Google or IBM to supply the AI expertise needed, this was not an option for Omega. “What is extremely important, whether it's for a scoring sport, or timing sport, is that we can't have discrepancies between the explanation of the performance and the ultimate result," he says. "So to protect the integrity of the result, we cannot rely on another company. We need to have the expertise to be able to explain the result and how the athletes got there.”

As for future timing and tracking upgrades, Zobrist is tight-lipped, but says the Paris Games in 2024 will be key. “You will see a whole new set of innovations. Of course, it will remain around timekeeping, scoring, and certainly also around motion sensors and positioning systems. And certainly also Los Angeles in 2028. We've got some really interesting projects for there that actually we've only just started.”