If you’ve watched any of the televised World Cup matches, you’ll have glimpsed the new technology being used by the VAR (video assistant referee) to help determine when a player is offside.

Though it usually appears on the TV screen for just a few seconds every match, it’s arguably one of the stars of Qatar 2022.

Semi-automated offside technology (SAOT) has taken decades to develop, requires sophisticated artificial intelligence (AI) systems, a battery of special cameras, as well as a ball packed with sensors, and about 10 years ago would have seemed impossible.

The computer vision technology it relies on is powering a new generation of data gleaning services, which are changing the way “the world game” and other sports are being coached and played.

Tracking every limb of every player

To refresh your memory, here’s how SAOT appears on TV: When a potential off-side is detected, play is stopped, and the view abruptly shifts to a virtual representation of the match, with the players appearing as mannequin-like figures in their player jerseys, frozen mid-stride.

A plane of translucent white running across the width of the field shows the exact position of the offside line.

If a single inch of an attacker (apart from their hands and arms) is beyond this line, they’re offside (pending the referee’s decision).

This may sound straightforward, but when you stop to think about what’s required to make this system both possible and reliable enough to use in a World Cup, it raises all kinds of questions.

Like, how does SAOT know the location of not only every player on the pitch, but the position of their limbs, down to the millimetre?

The story of how this technology was developed intersects with a motley assortment of other stories, from the real-life events behind the movie Moneyball, to England getting knocked out of the 2010 World Cup, to the origin of hugely popular Snapchat filters.

It begins in the summer of 1966.

A summer project that lasted decades

Computer vision, or teaching a computer to visually recognise objects, is something that may sound easy, but is really, really hard.

Take, for example, the task of recognising a bird. A computer can be shown a photo and taught that a particular pattern of pixels adds up to an object called “bird”. But birds flap around. They hop and move. The pattern of pixels changes with every photo.

To teach a computer to recognise a bird, you have to teach it to interpret what those pixels represent. You have to somehow teach it to recognise the common “bird” within millions of photos.

For some reason, at first it was thought this would be easy.

In 1966, MIT computer scientist Marvin Minsky assigned a first-year undergraduate student this problem to solve over the summer.

Needless to say, the student didn’t solve it, although their work laid the foundations for the field, says Simon Lucey, director of the Australian Institute for Machine Learning at the University of Adelaide.

“A lot of people thought that it would be really simple to get machines to sort of see like we as humans do,” he says.

“But it’s turned out to be obviously extremely difficult.”

The following decades saw slow progress. Robots could be taught to recognise boxes on assembly lines, for instance, or to read hand-written postcodes on envelopes, but that was about it.

Then, in 2012, there was a sudden advance.

“2012 was a big inflection point, and the inflection point was these things called deep neural networks,” Professor Lucey says.

“[They] basically allowed computer vision to jump from this sort of theoretical curiosity where governments were funding things … to actually companies realising that, ‘Hey, we can use this stuff.'”

The dawn of cat filters

From this 2012 inflection point flowed the computer vision applications that we use every day, like iPhone Face ID and Google reverse image search, as well as systems that are used on us, from facial recognition surveillance to police cars that scan number plates.

But first: What is a deep neural network (DNN)? Neural networks are designed to imitate how humans think and learn; they’re made up of layers of nodes, much like the human brain is made up of neurons. The network is said to be deeper based on the number of layers it has.