3D Ball Tracking with Unparalleled Precision

Maikel is one of BallJames’ most experienced team members. While finishing his master Biometrics and Computer Vision at Twente University, he was recruited by SciSports to start working on its most ambitious project to date: the first fully autonomous player and ball tracking system.

BallJames uses 14 4K cameras to track players, referees and the ball during a football game. Players are identified real-time by automatically recognizing their jersey numbers, and the ball is tracked in 3 dimensions, 25 times per second, accurate within a few centimeter, to ensure that every movement on the pitch is captured and transformed in data. All this data is of vital importance to coaches and trainers, to gauge the fitness of their players during the match, and when combined with the position of the ball for tactical analysis. This data can also be used for the fan at home who wants to see how fast the ball is played or what the speed of the player is during that fantastic sprint.

Maikel has worked on almost every component of the tracking solution. but his major contribution is the development of the ball tracker, the component that tracks the ball in three dimensions with unparalleled accuracy.

With the latest developments in camera technology, very fast hardware, and extensive machine learning techniques, autonomous deep learning networks can for the first time ever be used in complex, real-time scenarios.

Orange balls

Existing systems that do player and ball tracking for football using cameras often rely on human input to locate a certain object and then keep tracking while it remains in view. The drawback here is that humans can make mistakes, and are certainly not as accurate as machines can be. BallJames, in contrast, is built upon a completely different idea: let the machine learn autonomously, without any human input, to track the objects on the pitch. With the latest developments in camera technology, very fast hardware, and extensive machine learning techniques, autonomous deep learning networks can for the first time ever be used in complex, real-time scenarios.

‘We have created multiple components that build upon each other,’ Maikel explains. ‘First is detecting the ball in the single views of all 14 cameras independently. This is done using a deep learning network that we have trained to identify the ball from a camera image in all kinds of circumstances.’

Maikel has solved a lot of the challenges that came with that. He remembers when during a certain match the ball was not detected at all. ‘It turned out the home team had decided to play with an orange ball. The system had only learned to identify white balls so we had to retrain it to make sure it could handle balls of all colors.’ After that, he ran into the issue that when a goalkeeper put his white gloves together his hands were seen as the ball. ‘Solving all these issues was a real challenge but a fun one, and I’m proud we were able to solve all of them.’

Synchronize your watches

The multiview tracker is the next component. Once the ball is detected from a single camera, you still have no 3d positional data, as there is no depth perception in a single video from only one camera. If you can see the ball from multiple cameras then you can estimate the actual location in 3 dimensions. An important component is calibration of the video image to real word coordinates, using the lines of the pitch.

‘The issue we had to solve next had to do with timing,’ Maikel says. ‘The problem is: how do we know that the image from two different cameras is made on the exact same time? If there would be a timing difference of say 100ms, the ball could have traversed 2 meters. From that difference you cannot give a true estimate of the ball’s position.’

Why 14 cameras?

BallJames is a high-end system that uses 14 4K cameras placed strategically around the field of play. Maikel explains why this is necessary: ‘Often when players are dribbling the view, can get blocked for some cameras. Also when the ball is on the other side of the field and you only have cameras on one side, it’s very difficult to detect the ball, because it is quite small. More cameras are also necessary to accurately measure how high the ball is going. With 14 cameras you have the whole pitch covered and you will always be able to detect the ball.’

But it’s not only the ball that requires a lot of cameras. Jersey number recognition, needed for accurate player tracking is only possible if you are able to see a players back from every possible angle. BallJames is built to handle crowded situations, for instance with a corner kick in the penalty area, very well.

Creating ball tracks

The final component of the ball tracker is to create tracks that consistently follow the trajectory of the ball over time. Small mistakes that the detection has made can be corrected by interpolating the ball where it is missing for a few frames. In some cases Maikel needed to pull some more tricks out of his hat: ‘When a ball is shot very high, it can be very difficult for the camera to follow the ball, especially against a backdrop of the audience. By extrapolating the parabolic arc the ball makes I was able to recreate the track quite precisely. We are able to overlay the data tracks onto the actual footage to see if this works in practice.’

What’s next?

Maikel is excited about further developing the ball tracker to do more than just tracking, like detecting successful passes, or measuring the quality of shots on goal. ‘If we are then also able to, for instance, gauge the control a player has over the ball, we can really give insights into the game that our clients are waiting for.’

Get in touch if you want to know more about BallJames