HoloLens2 Spatial mapping extraction

Question

I'm currently stuck in a problem relating the HololLens2. I try to access the spatial mapping directly during runtime. So I want to implement a scanning procedure, where the user has to scan the room and then I want to export this mesh. Basically I want to create the .obj of the room scan like in the device portal but during runtime.

Therefore I've implemented a solution, which I've found here: https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk2/features/spatial-awareness/usage-guide?view=mrtkunity-2022-05

The problem now is that I don't get any meshes.

I currently use MRTK 2.8.3.0 and Unity 2022.3.10f1

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Microsoft.MixedReality.Toolkit;
using Microsoft.MixedReality.Toolkit.SpatialAwareness;
using TMPro;
using Unity.VisualScripting;

public class Scanner_new : MonoBehaviour
{
    public GameObject txt;
    static int count = 0;

    // Start is called before the first frame update
    void Start()
    {
        TextMeshPro textMesh = txt.GetComponent<TextMeshPro>();

        // Use CoreServices to quickly get access to the IMixedRealitySpatialAwarenessSystem
        var spatialAwarenessService = CoreServices.SpatialAwarenessSystem;

        // Cast to the IMixedRealityDataProviderAccess to get access to the data providers
        var dataProviderAccess = spatialAwarenessService as IMixedRealityDataProviderAccess;

        var meshObserver = dataProviderAccess.GetDataProvider<IMixedRealitySpatialAwarenessMeshObserver>();
        if(meshObserver == null){
                count++;
                textMesh.text = "No Observer! \n";
                Debug.Log("No Observer! \n");
                return;
            }
        textMesh.text += "Accessing all the meshes: \n";
        // Loop through all known Meshes
        foreach (SpatialAwarenessMeshObject meshObject in meshObserver.Meshes.Values)
        {   textMesh.text = "here1 \n";
            Mesh mesh = meshObject.Filter.mesh;
            var triangles = mesh.GetTriangles(0);
            foreach (var triangle in triangles){
                textMesh.text = "here2 \n";
                textMesh.text += triangle + "\n";
                count++;
            }
        }
    }

    // Update is called once per frame
    public void Updater()
    {
        TextMeshPro textMesh = txt.GetComponent<TextMeshPro>();
        if(count == 0){
            // Use CoreServices to quickly get access to the IMixedRealitySpatialAwarenessSystem
            var spatialAwarenessService = CoreServices.SpatialAwarenessSystem;

            // Cast to the IMixedRealityDataProviderAccess to get access to the data providers
            var dataProviderAccess = spatialAwarenessService as IMixedRealityDataProviderAccess;

            var meshObserver = dataProviderAccess.GetDataProvider<IMixedRealitySpatialAwarenessMeshObserver>();
            if(meshObserver == null){
                count++;
                textMesh.text = "No Observer! \n";
                Debug.Log("No Observer! \n");
                return;
            }
            textMesh.text += "Accessing all the meshes: \n";
            // Loop through all known Meshes
            foreach (SpatialAwarenessMeshObject meshObject in meshObserver.Meshes.Values)
            {
                textMesh.text = "here1 \n";
                Mesh mesh = meshObject.Filter.mesh;
                var triangles = mesh.GetTriangles(0);
                foreach (var triangle in triangles){
                    textMesh.text = "here2 \n";
                    textMesh.text += triangle + "\n";
                    count++;
                }
            }
        }
    }
}

Thanks for your help :)

Answer 1

Because I was busy for quite a long time to find a solution for this problem and now I have also found one I would like to share it here.

In Unity it is important that the Spatial Observer "OpenXR Spatial Mesh Observer" exists in the MixedReality Toollkit Gameobject in the Inspector under the Spatial Awareness tab.

If this is the case, the following code should work:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Microsoft.MixedReality.Toolkit;
using Microsoft.MixedReality.Toolkit.SpatialAwareness;
using TMPro;
using Unity.VisualScripting;
using System.Globalization;
using System.IO;
using System;

public class Scanner_new : MonoBehaviour
{
    public GameObject txt;
    

    // Start is called before the first frame update
    void Start()
    {
        
    }

    // Update is called once per frame
    public void Updater()
    {
        TextMeshPro textMesh = txt.GetComponent<TextMeshPro>();
        
        // Use CoreServices to quickly get access to the IMixedRealitySpatialAwarenessSystem
        var spatialAwarenessService = CoreServices.SpatialAwarenessSystem;

        // Cast to the IMixedRealityDataProviderAccess to get access to the data providers
        var dataProviderAccess = spatialAwarenessService as IMixedRealityDataProviderAccess;

        // Get the SpatialObjectMeshObserver specifically
        var meshObserverName = "OpenXR Spatial Mesh Observer";
        var spatialObjectMeshObserver = dataProviderAccess.GetDataProvider<IMixedRealitySpatialAwarenessMeshObserver>(meshObserverName);
        if(spatialObjectMeshObserver == null){
                textMesh.text += "No Observer! \n";
                return;
            }
        textMesh.text += "Accessing all the meshes: \n";
        List<MeshFilter> meshes = new List<MeshFilter>();
        // Loop through all known Meshes
        foreach (SpatialAwarenessMeshObject meshObject in spatialObjectMeshObserver.Meshes.Values)
        {   
            meshes.Add(meshObject.Filter);
        }
        create_obj(meshes, "\\Hello_World.obj");
        
        
    }
    private void create_obj(List<MeshFilter> meshes, string targetFileName){
        TextMeshPro textMesh = txt.GetComponent<TextMeshPro>();
        int adjust = 1;
        MeshFilter[] mf = meshes.ToArray();
        int countVertex = 0;

        System.Text.StringBuilder objFileContent = new System.Text.StringBuilder(4096);
        objFileContent.Append("# Automatic export of Hololens 2 scanned mesh scan. \r\n\r\n");

        // for each object gets the vertexes normals and faces
        for (int i = 0; i < mf.Length; i++)
        {
            Mesh m = mf[i].sharedMesh;
            objFileContent.Append("o Object." + (i + 1) + "\r\n");
            
            // inverts x as the coordinates are different from the ones of regular obj
            foreach (Vector3 v in m.vertices)
            {
                float x = -v.x;
                float z = v.z;
                objFileContent.Append("v " + x.ToString("0.000000", CultureInfo.InvariantCulture) + " " + v.y.ToString("0.000000", CultureInfo.InvariantCulture) + " " + z.ToString("0.000000", CultureInfo.InvariantCulture) + "\r\n");
            }

            objFileContent.Append("\r\n\r\n");

            foreach (Vector3 n in m.normals)
            {
                float x = n.x;
                float z = n.z;
                objFileContent.Append("vn " + x.ToString("0.000000", CultureInfo.InvariantCulture) + " " + n.y.ToString("0.000000", CultureInfo.InvariantCulture) + " " + z.ToString("0.000000", CultureInfo.InvariantCulture) + "\r\n");
            }
            objFileContent.Append("\r\n\r\n");

            // the count of the faces starts with 1 and is cumulative for 
            // all objects on the scene this is why it is add 
            // adjust + countVertex as the id of the vertexes
            for (int ti = 0; ti < m.triangles.Length; ti += 3)
            {
                objFileContent.Append("f " + (m.triangles[ti] + adjust + countVertex ) + "//" + (m.triangles[ti] + adjust + countVertex) + " " + (m.triangles[ti + 1] + adjust + countVertex) + "//" + (m.triangles[ti + 1] + adjust + countVertex) + " " + (m.triangles[ti + 2] + adjust + countVertex) + "//" + (m.triangles[ti + 2] + adjust + countVertex) + "\r\n");
            }
            objFileContent.Append("\r\n\r\n");
            countVertex += m.vertexCount;
      }
      try{
        // string objPath = Path.Combine(sceneFolderPath, sceneName + ".obj"); 
        using (StreamWriter sw = File.CreateText(Application.persistentDataPath + targetFileName))
        {
            sw.WriteLine(objFileContent.ToString());
        }
        textMesh.text += "SUCCESS: OBJ SAVED";
      }
      catch(Exception e){
        textMesh.text += e;
      }
       
    }
}

As you can see, the code is producing directly an .obj-file from the saved data. This code is inspired by the answer to the following question: MRTK 2.4 - Save Spatial Mesh on Runtime

The .obj-file can then be accessed over the device-portal->File Explorer and there in the localState-folder of your application :)