How to properly use OpenCV VideoWriter to write monochrome video with float32 source data type in Python?

Question

I am trying to store video file from multiple sources (color, depth, and infrared) from Kinect sensors.

This is the image that I visualized using cv2.imshow command using the following code:

cv2.imshow("ir", ir / 65535.)
cv2.imshow("depth", depth / 4500.)
cv2.imshow("color", color)

IR and depth both are array with size of (height, width), float32. Color is an array with size of (height, width, 3), where 3 is the RGB channel and uint8 type from 0-255. Since IR and depth's values are large, we need to normalize them using the code above. This code gave the above figures.

Now I want to store a series of image array as a video file. I use the following code:

ir_video = cv2.VideoWriter('ir.mp4', cv2.VideoWriter_fourcc(*'MP42'), fps, (height, width), False)
depth_video = cv2.VideoWriter('depth.mp4', cv2.VideoWriter_fourcc(*'MP42'), fps, (height, width), False)
color_video = cv2.VideoWriter('color.mp4', cv2.VideoWriter_fourcc(*'MP42'), fps, (height, width), True)

for ir, depth, color in zip(ir_frames, depth_frames, color_frames):
    ir_video.write(ir / 65535.)
    depth_video.write(depth / 4500.)
    color_video.write(color)

ir_video.release()
depth_video.release()
color_video.release()

Color video works very well, looks very similar to the cv2.imshow command. However, IR and depth video are corrupted. All 0kb. I tried to change the fourcc code to cv2.VideoWriter_fourcc(*'mp4v'). This time the IR one saved a video that I can play. But it is very different from the cv2.imshow result. It is shown here.

I'm wondering how I can correctly save the result as with cv2.imshow command. What fourcc code should be used? Thanks a lot!

Answer 1

I worked on a similar project using other depth cameras (Orbbec, Asus Xtion) and afaik videowriter class of OpenCV does not support 16-bit depth images, that's why as suggested in the comments you should convert to 8 bit. You can take a look here for what I was using to save such a video (It's about using OpenNI2 but the main concept is there).

Answer 2

Starting with OpenCV 4.7.0 it is possible to write 16-bit depth videos, see the pull request which added support for it.

For VideoWriter you have to:

• specify CAP_FFMPEG because this only seems to be supported for FFmpeg at the moment
• use the FFV1 codec
• specify {VIDEOWRITER_PROP_DEPTH, CV_16U, VIDEOWRITER_PROP_IS_COLOR, false} as params

For VideoCapture (reading) you have to:

• specify CAP_FFMPEG
• specify {CAP_PROP_CONVERT_RGB, false} as params
  Note that this will print a warning like "VIDEOIO/FFMPEG: BGR conversion turned OFF ..." to the console.

There appear to be some limitations with this though, see the description of the pull request.

That pull request also added a unit test which contains a VideoWriter & VideoCapture round trip.

Answer 3

Solution provided in the question comments converts the image data from float32 to uint8 which is sufficient to save the video properly. However, a lot of information is lost due to uint8 only being able to represent 256 values. That is also the reason why the source data (IR and depth) are float32 and not uint8 as with the color images - a lot of information would be lost when saved as uint8. Therefore, I propose a solution that saves the video in uint16 format with the VideoWriter used by the author of the question.

At first, it is necessary to convert the values from float32 to uint16 (range 0-65,535). According to author's code, IR image seems to already fall in that range, so only casting to uint16 is necessary. However, depth image has to be normalized from its original range 0-4,500 to the uint16 range. This code should be placed in the for loop provided by author before the write method.

ir = ir.astype(np.uint16)
depth = (depth * (65_535.0 / 4500.0)).astype(np.uint16)

@Marcono1234 provided an idea how to save video with OpenCV VideoWriter (since version 4.7.0) but didn't provide the Python code. I find it not trivial to program it correctly, so I provide a fully working example of reading an image from your webcam, converting it to monochrome image 16 bits of depth and saving it as such. Run to record the video and use keyboard letter q to stop recording. The most important part is obviously the VideoWriter definition.

import cv2
import numpy as np

video_capture = cv2.VideoCapture(0)

if not video_capture.isOpened():
    print("Error reading video file")

video_width = int(video_capture.get(cv2.CAP_PROP_FRAME_WIDTH))
video_height = int(video_capture.get(cv2.CAP_PROP_FRAME_HEIGHT))

video_writer = cv2.VideoWriter(
    filename="video.mkv",
    apiPreference=cv2.CAP_FFMPEG,
    fourcc=cv2.VideoWriter_fourcc(*"FFV1"),
    fps=10.0,
    frameSize=(video_width, video_height),
    params=[
        cv2.VIDEOWRITER_PROP_DEPTH,
        cv2.CV_16U,
        cv2.VIDEOWRITER_PROP_IS_COLOR,
        0,  # false
    ],
)

while True:
    ret, frame = video_capture.read()
    if ret:
        # Convert the webcam image to mono 16-bit.
        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        frame = frame.astype(np.uint16)
        frame *= 2 ** 8

        video_writer.write(frame)

        cv2.imshow('Frame', frame)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    else:
        break

video_capture.release()
video_writer.release()
cv2.destroyAllWindows()

To make the example complete, you'd also like to read the video somehow. Here's again a fully working example of OpenCV VideoCapture with monochrome 16-bit depth video.

import cv2

video_capture = cv2.VideoCapture(
    filename="video.mkv",
    apiPreference=cv2.CAP_FFMPEG,
    params=[
        cv2.CAP_PROP_CONVERT_RGB,
        0,  # false
    ],
)

if not video_capture.isOpened():
    print("Error reading video file")

while True:
    ret, frame = video_capture.read()
    if ret:
        cv2.imshow('Frame', frame)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    else:
        break

video_capture.release()
cv2.destroyAllWindows()

Be aware that produced video might not be playable with every video player since FFV1 encoding of 16-bit depth mono video is not so common yet. VLC media player had this discussed and supports it since version 3.0.18 and some improvements might also come in version 4.0 when it is released.