How to track fast moving object using ARKit?

Question

I am trying to build an app on iOS in Swift, SwiftUi and using ARKit and RealityKit. I want the app to:

1. detect a soccer ball
2. detect a soccer goal
3. track the trajectory of the ball when shot at goal
4. detect when/if the ball hits/passes the goal

To detect the ball and goal using ARKit is working fine. However, to keep tracking the ball when in movement is very unreliable. And when its moving fast, ARKit fails to detect the ball at all.

I have tried to implement Apple Visions VNDetectTrajectoriesRequest using the following tutorial,https://developer.apple.com/documentation/vision/identifying_trajectories_in_video, which works as long as I am NOT using ARKit. DetectTrajectory needs a CMSampleBuffer, and ARKit only returns a PixelBuffer and I wasn't able to solve this issue and get it to work smoothly (I tried converting the pixelbuffer to CMSampleBuffer, but that resulted in the app being SO slow it was useless in realtime). And even if I get this to work it wouldn't solve my issue regarding detecting when the soccer hits the goal.. For that I believe that using depth/ARkit is necessary..

Does anyone have a solution for tracking A FAST MOVING OBJECT USING ARKIT?

I am close to giving up, but I have found this guy on Youtube that is detecting a balls trajectory using ARKit, https://www.youtube.com/watch?v=B4yfp1UmM5s. I have written to him but received no reply this far. Have been trying to solve this for over a week but I can't find a solution. Please internet, help me!

Answer 1

I think that such a mission is impossible for iOS device in 2022.

Firstly, let's assume the average speed of a soccer ball is 12 m/s, and ARKit and Vision track it at 60 fps. Any object moving at 12 m/s is difficult to qualitatively track at that frame rate, it's obvious. Even MoCap systems use at least 120 fps for tracking of much slower movements.

Secondly, in 5 seconds the ball approximately covers a distance of 60 meters. This is a fairly large distance (for detection/recognition) at which such a small object as a soccer ball will be unrecognizable, especially since it also spins during flight.