I'm working on a project where I track an object using a Model Target and display its 3D model in a virtual environment for interaction. The user wears a VR headset and views the environment with the 3D model. The camera is placed in front of the user, so I aligned the camera and headset reference frames to align the real object with its 3D model.
However, the alignment is affected by the camera lens distortion. When the object is in the center of the image (principal point), the alignment is correct, but as it moves toward the edges, the distortion causes a misalignment.
To resolve this, I gathered the camera's intrinsic parameters and tried to build a Custom Driver (as suggested here: https://library.vuforia.com/platform-support/external-camera-calibration).
To build the Driver, I edited the RefDriverImplData.h and RefDriverImpl.cpp files in the Driver Template sample (10-13-3) adding my camera's intrinsic parameters. I also edited the Driver.h file in the UWP SDK (10-13-3) to add the camera parameters and built the Driver Template sample, as described for the File Driver sample (https://library.vuforia.com/platform-support/building-and-using-file-driver-sample). Then, since I'm using Unity (2021.3.15f1), I added the compiled binaries to Assets/Plugins/WindowsStoreApps/x64 and a script for delayed initialization with the custom driver.
When I run the project in Play Mode, I get two error messages (attached to this post) and one on screen. I also tried building the project, but it crashes without any error message.
Is it enough to provide the intrinsic parameters in the Driver Template sample files and Driver.h to make it work with the camera? And is this the correct way to provide intrinsic parameters to Vuforia to get undistorted images before tracking?
Error Log:
1.
Failed to create Vuforia engine: Vuforia driver library load error
UnityEngine.Debug:LogError (object)
Vuforia.Internal.Utility.UnityLogger:LogError (string)
Vuforia.Internal.Utility.Log:Error (string)
Vuforia.Internal.Core.Engine:InitOnCameraReady ()
Vuforia.WebCam:HandleFirstWebCamFrame ()
Vuforia.WebCam:<Init>b__33_0 (bool)
Vuforia.Internal.Utility.VuforiaCoroutineUtility/<RunCoroutineWithTimeout>d__1:MoveNext ()
UnityEngine.SetupCoroutine:InvokeMoveNext (System.Collections.IEnumerator,intptr)
2.
Vuforia Engine initialization failed: INIT_VUFORIA_DRIVER_FAILED
INIT VUFORIA DRIVER FAILED
Failed to initialize Vuforia Engine.
UnityEngine.Debug:LogError (object)
DefaultInitializationErrorHandler:SetErrorCode (Vuforia.VuforiaInitError) (at Library/PackageCache/com.ptc.vuforia.engine@ad7ad60b4246/Vuforia/Scripts/DefaultInitializationErrorHandler.cs:157)
DefaultInitializationErrorHandler:OnVuforiaInitializationError (Vuforia.VuforiaInitError) (at Library/PackageCache/com.ptc.vuforia.engine@ad7ad60b4246/Vuforia/Scripts/DefaultInitializationErrorHandler.cs:24)
System.Delegate:DynamicInvoke (object[])
Vuforia.Utility.ExtensionMethods.DelegateHelper:InvokeDelegate (System.Delegate,object[])
Vuforia.Utility.ExtensionMethods.DelegateHelper:InvokeWithExceptionHandling<Vuforia.VuforiaInitError> (System.Action`1<Vuforia.VuforiaInitError>,Vuforia.VuforiaInitError)
Vuforia.VuforiaApplication:VuforiaInitialized (Vuforia.VuforiaInitError)
System.Delegate:DynamicInvoke (object[])
Vuforia.Utility.ExtensionMethods.DelegateHelper:InvokeDelegate (System.Delegate,object[])
Vuforia.Utility.ExtensionMethods.DelegateHelper:InvokeWithExceptionHandling<Vuforia.VuforiaInitError> (System.Action`1<Vuforia.VuforiaInitError>,Vuforia.VuforiaInitError)
Vuforia.Internal.Core.Engine:InitOnCameraReady ()
Vuforia.WebCam:HandleFirstWebCamFrame ()
Vuforia.WebCam:<Init>b__33_0 (bool)
Vuforia.Internal.Utility.VuforiaCoroutineUtility/<RunCoroutineWithTimeout>d__1:MoveNext ()
UnityEngine.SetupCoroutine:InvokeMoveNext (System.Collections.IEnumerator,intptr)
Note that from the description here https://library.vuforia.com/platform-support/creating-custom-driver#cameracallback-class your Driver implementation has to provide both image buffer and camera intrinsics. When enabling Driver Vuforia does not access the system camera anymore so it is not possible to just provide intrinsics via Driver.