Projecting the vertices of a tennis court from 3D to 2D is not working

Question

Background

I've defined a tennis court with 3D points, and I'm trying to use OpenCV to project the 3D points as a 2D image.

Below are the 3D points I'm using to define the tennis court. For reference:

• Left on the court is -X and Right is +X,
• Up is the -Y and Down is +Y, and
• Far side of net is +Z and Close side is -Z.
• Point (0,0,0) is the intersection of the left side line with the net line (basically the left net post).

def DefineCourtPoints():
    objp = {}
    objp[1] = [0,0,39] # 1.  Intersection of the upper base line with the left side line
    objp[2] = [0,0,-39] # 2.  Intersection of the lower base line with the left side line
    objp[3] = [36,0,-39] # 3.  Intersection of the lower base line with the right side line
    objp[4] = [36,0,39] # 4.  Intersection of the upper base line with the right side line  
    objp[5] = [4.5,0,39] # 5.  Intersection of the upper base line with the left singles line
    objp[6] = [4.5,0,-39] # 6.  Intersection of the lower base line with the left singles line
    objp[7] = [31.5,0,-39] # 7.  Intersection of the lower base line with the right singles line
    objp[8] = [31.5,0,39] # 8.  Intersection of the upper base line with the right singles line
    objp[9] = [4.5,0,21] # 9.  Intersection of the left singles line with the upper service line
    objp[10] = [31.5,0,21] # 10.  Intersection of the right singles line with the upper service line
    objp[11] = [4.5,0,-21] # 11.  Intersection of the left singles line with the lower service line
    objp[12] = [31.5,0,-21] # 12.  Intersection of the right singles line with the lower service line
    objp[13] = [18,0,21] # 13.  Intersection of the upper service line with the center service line
    objp[14] = [18,0,-21] # 14.  Intersection of the lower service line with the center service line
    objp[15] = [0,0,0] # 15.  Intersection of the left side line with the net line (this will be considered (0,0) )
    objp[16] = [36,0,0] # 16.  Intersection of the right side line with the net line

    return objp

For additional reference, here are the intrinsic matrix and distortion parameters for my camera (which were obtained using OpenCV and have been validated):

intrinsic_mtx = np.array([
    [1882.77177, 0.0, 973.572122],
    [0.0, 1880.83035, 537.299982],
    [0.0, 0.0, 1.0]
])

distortion = np.array([0.232714433, -1.35119878, -0.00188551612, 0.00166434182, 2.50351701])

Here is the code I use to project each of the 3D points to 2D, and to graph the points/lines using Matplotlib.

def ProjectPoints(intrinsic_mtx, distortion, R, T, objp)
    x_points = []
    y_points = []

    # Loop through each of the 3D points and project them to 2D.
    for index in objp: 
        2d_point = cv2.projectPoints(
                        np.array(objp[index], dtype=np.float64),
                        cv2.Rodrigues(R)[0],
                        T,
                        intrinsic_mtx, 
                        distortion
                     )[0][0][0]

        print(2d_point)
        x_points.append(2d_point[0])
        y_points.append(2d_point[1])

    # Graph the court boundary.
    lines = [(1,2), # A line exists between point 1 and 2...
            (2,3), # A line exists between point 2 and 3...
            (3,4), # A line exists between point 3 and 4...
            (4,1)] # A line exists between point 4 and 1...

    x_lines = list()
    y_lines = list()
    for pair in lines:
        for i in range(2):
            x_lines.append(x_points[pair[i]-1])
            y_lines.append(y_points[pair[i]-1])
        # Append None to separate the lines.
        x_lines.append(None)
        y_lines.append(None)

    fig, ax = plt.subplots()
    ax.scatter(x_points, y_points)
    ax.axis('equal')
    ax.plot(x_lines, y_lines, c = 'r')
    plt.show()

Problem

When I project the points and graph them, I expect to get something looking like a tennis court, but I just get a blob of points!

To troubleshoot, I decided to graph just outer edges of the court using the four corners of the court (which are the first four points in DefineCourtPoints). I expected to see a rectangle or parallelogram as these are valid projections of a rectangle. However, I get stuff like the following, which doesn't really make any sense since it has crossed lines:

The above result was obtained using R = [0, 0, 0] and a T = [0, -10, 0]

Here are the coordinates of the four points for the above result, showing that I didn't cause the weird shape by somehow graphing the lines in the wrong order:

# Intersection of the upper base line with the left side line
[973.572122 , 1019.56417431]

# Intersection of the lower base line with the left side 
line
[973.572122 , 55.03578969]

# Intersection of the lower base line with the right side 
line
[-764.37105031, 55.03578969]

# Intersection of the upper base line with the right side line
[2711.51529431 , 1019.56417431]

Question

Why am I getting such weird projections for such a simple 3D object as a rectangle/tennis court boundary?

Is anyone getting the same results that I'm getting?

Thanks!

Try it yourself

import numpy as np
import cv2
import matplotlib.pyplot as plt

def DefineCourtPoints(): # Just the corners of the court.
    objp = {}
    objp[1] = [0,0,39] # 1.  Intersection of the upper base line with the left side line
    objp[2] = [0,0,-39] # 2.  Intersection of the lower base line with the left side line
    objp[3] = [36,0,-39] # 3.  Intersection of the lower base line with the right side line
    objp[4] = [36,0,39] # 4.  Intersection of the upper base line with the right side line  

objp = DefineCourtPoints()
intrinsic_mtx = np.array([
    [1882.77177, 0.0, 973.572122],
    [0.0, 1880.83035, 537.299982],
    [0.0, 0.0, 1.0]
])

distortion = np.array([0.232714433, -1.35119878, -0.00188551612, 0.00166434182, 2.50351701])
R = np.array([0,0,0])
T = np.array([0,-10,0])
ProjectPoints(intrinsic_mtx, distortion, R, T, objp)