Apple Vision image recognition

Question

As many other developers, I have plunged myself into Apple's new ARKit technology. It's great. For a specific project however, I would like to be able to recognise (real-life) images in the scene, to either project something on it (just like Vuforia does with its target images), or to use it to trigger an event in my application.

In my research on how to accomplish this, I stumbled upon the Vision and CoreML frameworks by Apple. This seems promising, although I have not yet been able to wrap my head around it.

As I understand it, I should be able to do exactly what I want by finding rectangles using the Vision framework and feeding those into a CoreML model that simply compares it to the target images that I predefined within the model. It should then be able to spit out which target image it found.

Although this sounds good in my head, I have not yet found a way to do this. How would I go about creating a model like that, and is it even possible at all?

Answer 1

As of ARKit 1.5 (coming with IOS 11.3 in the spring of 2018), a feature seems to be implemented directly on top of ARKit that solves this problem.

ARKit will fully support image recognition. Upon recognition of an image, the 3d coordinates can be retrieved as an anchor, and therefore content can be placed onto them.

Answer 2

I found this project on Github some weeks ago: AR Kit Rectangle Detection

I think that is exactly what you are looking for...

Answer 3

Vision's ability to detect images was implemented in ARKit beginning from iOS 11.3+, so since then ARKit has ARImageAnchor subclass that extends ARAnchor parent class and conforms to ARTrackable protocol.

// Classes hierarchy and Protocol conformance...

ObjectiveC.NSObject: NSObjectProtocol
        ↳ ARKit.ARAnchor: ARAnchorCopying
                ↳ ARKit.ARImageAnchor: ARTrackable

ARWorldTrackingConfiguration class has a detectionImages instance property that is actually a set of images that ARKit attempts to detect in the user's environment.

open var detectionImages: Set<ARReferenceImage>!

And ARImageTrackingConfiguration class has a trackingImages instance property that is a set as well, it serves the same purpose – ARKit attempts to detect and track it in the user's environment.

open var trackingImages: Set<ARReferenceImage>

So, having a right configuration and an ability to automatically get ARImageAnchor in a ARSession, you can tether any geometry to that anchor.

P.S. If you want to know how to implement image detection feature in your ARKit app please look at this post.