Optimizing Manhattan distance calculation for frames of video

Question

I'm trying to generate Manhattan distances for a specific cropped section of each frame of a video, as compared to the first frame of the video. These are high framerate videos with ~5000+ frames per video. It currently takes ~120 seconds per video to do this analysis and generate a list of frames and their associated Manhattan Distances. I've a couple of things to optimize the speed - using the built in scipy.spacial.distance cdist function, trying out the euclidian distance using np.linalg.norm, and doing the grayscale conversion outside of the loop as a video pre-processing step. None of these changes made a significant change in the computational time. Is there any way to significantly speed up this process (the while loop in the function below)?

def compare_images_master():
    current_frame = 0
    numFrames = count_frames_automatic(resized_videocap)
    GrayOriginalFrame = cv2.cvtColor(DetectionPlate, cv2.COLOR_BGR2GRAY)
    originalFrameMap = GrayOriginalFrame.astype(float)
    m_norm_list = [0] * numFrames
    frame_num_list = list(range(numFrames))
    start_compare_images = time.time()
    print("calculating Manhattan distances...")
    for current_frame in frame_num_list:
        resized_videocap.set(1, current_frame) #set current frame to the frame to be computed
        ret, currentFrameImage = resized_videocap.read() #read video file and open current
        currentFrameImageCropped = currentFrameImage[ref_pts[0][1]:ref_pts[1][1], ref_pts[0][0]: ref_pts[1][0]] #crop image to detection plate
        GrayCurrentFrame = cv2.cvtColor(currentFrameImageCropped, cv2.COLOR_BGR2GRAY) #convert to grayscale
        currentFrameMap = GrayCurrentFrame.astype(float) #convert from image to pixel frame map
        diff = originalFrameMap - currentFrameMap
        m_norm = np.sum(np.abs(diff))  # Manhattan norm
        m_norm_list[current_frame] = m_norm
    end_compare_images = time.time()
    print("Completed calculating Manhattan distances...")
    print ('time taken to generate manhattan distances', end_compare_images - start_compare_images)
    # time taken - 120seconds
    return m_norm_list, frame_num_list