I am trying to implement a Pose Normalisation methodology as mentioned in the paper LipSync3D, where I need to project 3D human face mesh vertices on a cylindrical coordinate system with a vertical axis such that most face vertices are equidistant from axis.
To implement this, the paper refers to another paper called OPTIMAL UV SPACES FOR FACIAL MORPHABLE MODEL CONSTRUCTION, which describes the methodology in the following section.
Cylindrical Coordinate projection
However, it is difficult to infer on how do 3D to 2D translation of mesh vertices to the z-x plane before applying the Coope's linear least squares method to fit the points optimally.
I tried the following, but its the wrong approach:
import mediapipe.python.solutions.face_mesh as mp_face_mesh
import mediapipe.python.solutions.drawing_utils as mp_drawing
import mediapipe.python.solutions.drawing_styles as mp_drawing_styles
import cv2
# Landmarks to dict
def landmark_to_dict(landmark_list):
landmark_dict = {}
for idx, landmark in enumerate(landmark_list):
landmark_dict[idx] = [landmark.x, landmark.y, landmark.z]
return landmark_dict
# Path to a human face
image = cv2.imread('img0002.png')
image_rows, image_cols, _ = image.shape
with mp_face_mesh.FaceMesh(
static_image_mode=True,
max_num_faces=1,
refine_landmarks=True,
min_detection_confidence=0.5) as face_mesh:
results = face_mesh.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
reference_dict = landmark_to_dict(results.multi_face_landmarks[0].landmark)
# Convert keypoints to np array
import numpy as np
arr = np.empty((0,3), int)
for i in range(len(reference_dict)):
row = np.array(reference_dict[i])
arr = np.vstack([arr,row])
import numpy as np
from ai import cs
# converting spherical spiral from spherical to cartesian coordinates
r, phi, z = cs.cart2cyl(
x = arr[:,0],
y = arr[:,1],
z = arr[:,2]
)
# Calculate new projections (unclear on this step / just a guess)
r_new = (r - (y - x))**2
# cylindrical to cartesian
x_new, y_new, z_new = cs.cyl2cart(r=r_new, phi=phi, z=z)
If anyone could provide a pseudo code to implement this, that’d be very helpful.