Enabling gestures in RealityKit

Question

I have a custom usdz file (not create through code, but let's say a real chair!). I save this in a Entity.

Once I have it this is my code:

func updateUIView(_ uiView: ARView, context: Context) {
            
    if let modelEntity = model.modelEntity {
        
        print("\(model.modelName)")
        
        let anchorEntity = AnchorEntity(plane: .horizontal)
        
        anchorEntity.addChild(modelEntity.clone(recursive: true))
        
        uiView.scene.addAnchor(anchorEntity)
        
        // modelEntity.generateCollisionShapes(recursive: true) 
        // If we have multiple object, recursive true help to generate collision for all of them
        
        uiView.installGestures(.rotation, for: modelEntity as! Entity & HasCollision)
        
        uiView.debugOptions = .showPhysics
        
    } else {
        
        print("Unable to load modelEntity for \(model.modelName)")
        
    }
}

The problem here is that `"Argument type 'Entity' does not conform to expected type 'HasCollision'". So I cant add any gesture.

But I can't find any usefull resource to achieve my final goal. Is there any advice?

Answer 1

The problem is that you are trying to give the ModelEntity an ability it does not posses (It does not have a collision handler).

You need to create an Entity yourself, that will conform to HasCollision.

I would try something like this:

import RealityKit

class MyEntity: Entity, HasAnchoring, HasModel, HasCollision {

}

func updateUIView(_ uiView: ARView, context: Context) {
    // This is simply to create a dummy modelEntity
    let plane = MeshResource.generatePlane(width: 0.1, depth: 0.1)
    let texture = MaterialColorParameter.color(UIColor.red)
    var material = SimpleMaterial()
    material.baseColor = texture
    let modelEntity = ModelEntity(mesh: plane, materials: [material])
      
    // This is the class we have wrapping the model
    let myEntity = MyEntity()
    myEntity.addChild(modelEntity)
    
    // Now, we add the collision component
    let boxShape = ShapeResource.generateBox(width: 0.1, height: 0.1, depth: 0.1)
    let boxShapeCollisionComponent = CollisionComponent (
      shapes: [boxShape],
      mode: .trigger,
      filter: .default
    )
    myEntity.collision = boxShapeCollisionComponent
    // Or, you could of called myEntity.generateCollisionShapes(recursive: true)
    
    // Last thing, lets put this plane, with a box collision component,
    // right in front of us
    myEntity.transform = Transform.identity
    myEntity.transform.translation.z = myEntity.transform.translation.z - 0.3
    uiView.scene.addAnchor(myEntity)
    
    uiView.installGestures(.rotation, for: myEntity)
    
    uiView.debugOptions = .showPhysics
}

Answer 2

Use the forced form of downcasting (type casting) as! with Entity & HasCollision.

arView.installGestures([.rotation], for: modelEntity as! Entity & HasCollision)

or this way:

let entity = modelEntity as? Entity & HasCollision
arView.installGestures([.all], for: entity!)

A source instance method installGestures(_:for:) looks like this:

func installGestures(_ gestures: ARView.EntityGestures = .all,
                     for entity: HasCollision) -> [EntityGestureRecognizer]

Initial Settings in Reality Composer

Before compiling, in Reality Composer set physics = participates and motion type = fixed and accessibility = accessibility enabled for your model.

Full code version

import SwiftUI
import RealityKit

struct ARViewContainer: UIViewRepresentable {
    
    let boxAnchor = try! Experience.loadBox()
    
    func makeUIView(context: Context) -> ARView {
        
        let arView = ARView(frame: .zero)   
        arView.scene.anchors.append(boxAnchor)
        return arView
    }
    
    func updateUIView(_ uiView: ARView, context: Context) {
        
        if let modelEntity: Entity = boxAnchor.steelBox {    
            let anchorEntity = AnchorEntity(.plane(.vertical, 
                                             classification: .wall, 
                                             minimumBounds: [0.2, 0.2]))  
            anchorEntity.addChild(modelEntity.clone(recursive: true))    
            uiView.scene.addAnchor(anchorEntity)    
            modelEntity.generateCollisionShapes(recursive: true)    
            uiView.installGestures([.all], 
                               for: modelEntity as! Entity & HasCollision)    
            uiView.debugOptions = [.showPhysics]
        }
    }
}

P. S.

Also, this post will show you how raycasting works in conjunction with RealityKit gestures.

Answer 3

Enable Physics -> Participates on entity in Reality Composer and add anchor.entity?.generateCollisionShapes(recursive: true) before installGestures

My example:

Experience.loadPizzaAsync { result in
    guard case let .success(pizzaAnchor) = result
    else { return }
    
    arView.scene.addAnchor(pizzaAnchor)
    
    pizzaAnchor.pizza?.generateCollisionShapes(recursive: true)
    arView.installGestures([.rotation, .translation], for: pizzaAnchor.pizza as! HasCollision)
}

Answer 4

I got some other situation: I needed to load the model from .usdz file and it should have an animation. But also I needed to have gestures like translation and rotation. The research directed me to the thread with the right answer. The code from it I'll show underneath, the main idea is "to nest the loaded entity which has the animations inside of a ModelEntity, and to then give that ModelEntity an appropriate CollisionComponent based on the loaded model's bounds." (c)

loadRequest = Entity.loadAsync(contentsOf: url)
    .sink(receiveCompletion: { status in
        print(status)
    }) { entity in
           
        // Scaling entity to a reasonable size
        entity.setScale(SIMD3(repeating: 0.01), relativeTo: nil)
           
        // Creating parent ModelEntity
        let parentEntity = ModelEntity()
        parentEntity.addChild(entity)
           
        // Anchoring the entity and adding it to the scene
        let anchor = AnchorEntity(.plane(.horizontal, classification: .any, minimumBounds: .zero))
        anchor.addChild(parentEntity)
        self.arView.scene.addAnchor(anchor)
           
        // Playing availableAnimations on repeat
        entity.availableAnimations.forEach { entity.playAnimation($0.repeat()) }
           
        // Add a collision component to the parentEntity with a rough shape and appropriate offset for the model that it contains
        let entityBounds = entity.visualBounds(relativeTo: parentEntity)
        parentEntity.collision = CollisionComponent(shapes: [ShapeResource.generateBox(size: entityBounds.extents).offsetBy(translation: entityBounds.center)])
                       
        // installing gestures for the parentEntity
        self.arView.installGestures(for: parentEntity)
    }