ARKit and Vision frameworks for Object Recognition

Question

I would really like some guidance on combining Apple's new Vision API with ARKit in a way that enables object recognition. This would not need to track the moving object, just recognize it stable in 3d space for the AR experience to react accordingly.

I know this type of experience is available in services like Vuforia or Wikitude, but I would like to try it with only native Apple APIs.

Answer 1

You don't necessarily need to use Vision framework itself inside your project, because ARKit is already has this feature. All you need is to activate a detectionObjects instance property that you can use right from iOS 12:

var detectionObjects: Set<ARReferenceObject> { get set }

Let's see what Apple documentation says about it:

Use this property to choose known 3D objects for ARKit to find in the user's environment and present as ARObjectAnchor for use in your AR experience.

Here's a working code (as simple as that) :

import ARKit

class ViewController: UIViewController {

    @IBOutlet var sceneView: ARSCNView!
    let config = ARWorldTrackingConfiguration()

    override func viewDidLoad() {
        super.viewDidLoad()

        sceneView.delegate = self

        // add reference objects into Resources in your project
        guard let objects = ARReferenceObject.referenceObjects(inGroupNamed: "Resources", 
                                                                     bundle: nil) 
        else { 
            fatalError("No reference here!") 
            return
        }

        config.detectionObjects = objects
        sceneView.session.run(config)
    }
}

And, of course, insert an extension with renderer() method:

extension ViewController: ARSCNViewDelegate {

    func renderer(_ renderer: SCNSceneRenderer,
                 didAdd node: SCNNode,
                  for anchor: ARAnchor) {

        if let _ = anchor as? ARObjectAnchor {         // Special Type of anchor

            let sphereNode =  SCNNode(geometry: SCNSphere(radius: 0.05))
            sphereNode.geometry?.firstMaterial?.diffuse.contents = UIColor.green
            node.addChildNode(sphereNode)
        }
    }
}

Answer 2

• To recognize (multiple) objects including their coordinates (to use in ARKit), you will need to train an Object Detector (not Classifier!)

• To have it exported into a CoreML model smoothly, the easiest ways to build/train it are CreateML and TuriCreate (slightly less easy).

• To add a 3D object to a particular spot in AR experience, you will need to find 'worldTransform' for its ARAnchor (that you will need to subclass).

• To find the 'worldTransform' you will need to fire a 'hitTest()' on ARFrame on the spot where the object was recognized in 2D.

Well, this is probably a good place to stop because your question is rather high-level.

Answer 3

I'm relatively new to this site, so I can't yet comment.

ARKit 1.5 in iOS 11.3 includes Image Recognition.

There is also support for running Vision with ARKit.

I'm not entirely sure what you are asking for, but these might help.