Overlaying a .obj file on an ArUco marker

Question

I have some boilerplate code to detect ArUco markers from a frame:

import cv2

# Load the camera
cap = cv2.VideoCapture(0)

# Set the dictionary to use
dictionary = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_6X6_250)

while(True):
    # Capture frame-by-frame
    ret, frame = cap.read()
    if not ret: continue
    detector = cv2.aruco.ArucoDetector(dictionary)

    # Detect markers
    corners, ids, _ = detector.detectMarkers(frame)

    # Draw markers
    frame = cv2.aruco.drawDetectedMarkers(frame, corners, ids)

    # Display the resulting frame
    cv2.imshow('frame', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()

I would like to be able to import an .obj file and overlay it over the ArUco marker. I don't want to open any extra windows, and would prefer for it to be able to run real time.

Is there a way to do this?

Answer 1

Maybe you could specify what is inside your .obj file? If you could load your .obj file as a kind of frame or picture, you could show it. You could simply write everything (like : ) on the frame, while knowing the corner positions. See the following example:

# !/bin/python3
import cv2
import numpy as np

def draw_smiley(image, center):
    # Calculate the positions of the eyes and the smile based on the center
    eye1 = (center[0] - 100, center[1] - 100)
    eye2 = (center[0] + 100, center[1] - 100)
    smile_center = (center[0], center[1] + 50)

    # Draw two filled circles for the eyes
    cv2.circle(image, eye1, 50, (255, 255, 255), -1)
    cv2.circle(image, eye2, 50, (255, 255, 255), -1)

    # Draw an arc for the smile
    cv2.ellipse(image, smile_center, (200, 100), 0, 0, 180, (255, 255, 255), 10)

    return image

# Load the camera
cap = cv2.VideoCapture(0)

# Set the dictionary to use
dictionary = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_6X6_250)

while(True):
    # Capture frame-by-frame
    ret, frame = cap.read()
    if not ret: continue
    detector = cv2.aruco.ArucoDetector(dictionary)

    # Detect markers
    corners, ids, _ = detector.detectMarkers(frame)

    # Iterate over the corners
    for i, corner in enumerate(corners):
        # Each corner is a list of points. We can calculate the center of the marker.
        center = np.mean(corner, axis=0).astype(int)

        # Draw a smiley face at the center of each marker
        frame = draw_smiley(frame, tuple(center[0]))

    # draw raw markers
    #frame = cv2.aruco.drawDetectedMarkers(frame, corners, ids)

    # Display the resulting frame
    cv2.imshow('frame', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()

Answer 2

Yes, it is possible to overlay a 3D model onto an ArUco marker in real-time using OpenCV and additional libraries like OpenGL for rendering.

import cv2
import numpy as np
# Import necessary libraries for OpenGL and 3D model loading

# Function to initialize OpenGL
def init_gl(width, height):
    # Set up OpenGL context
    pass

# Function to draw the model
def draw_model(model, modelview_matrix, projection_matrix):
    # Render the model using OpenGL 
    pass

camera_matrix = np.array([[fx, 0, cx],
                          [0, fy, cy],
                          [0,  0,  1]])
dist_coeffs = np.array([k1, k2, p1, p2, k3])

model = load_3d_model('model.obj')

cap = cv2.VideoCapture(0)
dictionary = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_6X6_250)

init_gl(cap.get(3), cap.get(4))

while True:
    ret, frame = cap.read()
    if not ret: continue

    detector = cv2.aruco.ArucoDetector(dictionary)
    corners, ids, _ = detector.detectMarkers(frame)

    if ids is not None:
        rvecs, tvecs, _ = cv2.aruco.estimatePoseSingleMarkers(corners, marker_size, camera_matrix, dist_coeffs)
        for rvec, tvec in zip(rvecs, tvecs):
            # Prepare OpenGL modelview and projection matrices
            modelview_matrix = prepare_modelview_matrix(rvec, tvec)
            projection_matrix = prepare_projection_matrix(camera_matrix)

            # Draw the model
            draw_model(model, modelview_matrix, projection_matrix)

    cv2.imshow('frame', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

Answer 3

In broad strokes:

• get a library that can draw your model
• set its projection matrix to be equivalent to the intrinsics matrix of your camera
• derive a model-view matrix from the marker's pose (rvec and tvec)
  • "model-view" as a single matrix because we don't have a world space where the camera can have its own pose
• draw the model on top of your camera picture

OpenCV has a module called ovis that integrates Ogre3D. It'lll draw the model for you. It's a "contrib" module, so it's community-maintained. All the mathy details for using it are in this article:

https://www.ogre3d.org/2020/12/24/augmented-reality-made-simple-with-ogre-and-opencv

The challenge here is getting a build of (or building) OpenCV that is built against the Ogre3D dependency. The official opencv-python-contrib package doesn't come with that dependency.

If you know OpenGL, you should be able to set it up equivalently. Or pick any other 3D graphics library that'll let you draw your model using a specifically set projection matrix.

You can use OpenCV 2D drawing primitives too (lines, filled polygons) but that'll result in either a wireframe or in flat-shaded polygons and you having to do your own calculations for the shading (Phong is fairly easy) and Z-buffer stuff. You'd have to transform the model vertices into camera space, then project.

It's doable. I'm not sure if I'm inclined to elaborate that into a working piece of code just now. I don't have much practice with OpenGL. I'd most likely present the "wireframe" option. Also, the question doesn't contain the model data. I'd have to scare up some model data from the internet or mock up my own (it wouldn't be much more than a cube or pyramid).