Capture full-range/lossless rgb frame from capture card that supports NV12 and YUYV output

Question

I am trying to make a program which captures an image, then i need to compare captured image and the input data which i displayed, both should matc pixel by pixel

Here are the details of my capture card

$ v4l2-ctl --list-formats-ext -d /dev/video0

ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'NV12' (Y/CbCr 4:2:0)
                Size: Discrete 3840x2160
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 2560x1440
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 640x480
                        Interval: Discrete 0.017s (60.000 fps)
        [1]: 'YUYV' (YUYV 4:2:2)
                Size: Discrete 2560x1440
                        Interval: Discrete 0.020s (50.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 640x480
                        Interval: Discrete 0.017s (60.000 fps)
        [2]: '' (30313050-0000-0010-8000-00aa003)
        [3]: '' (e436eb7e-524f-11ce-9f53-0020af0)

$ v4l2-ctl --all

Driver Info:
        Driver name      : uvcvideo
        Card type        : ITE HDMI 4K+ Bridge: ITE HDMI 4
        Bus info         : usb-0000:00:14.0-6
        Driver version   : 5.18.0
        Capabilities     : 0x84a00001
                Video Capture
                Metadata Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
Priority: 2
Video input : 0 (Camera 1: ok)
Format Video Capture:
        Width/Height      : 1920/1080
        Pixel Format      : 'YUYV' (YUYV 4:2:2)
        Field             : None
        Bytes per Line    : 3840
        Size Image        : 4147200
        Colorspace        : sRGB
        Transfer Function : Rec. 709
        YCbCr/HSV Encoding: Rec. 709
        Quantization      : Default (maps to Limited Range)
        Flags             :
Crop Capability Video Capture:
        Bounds      : Left 0, Top 0, Width 1920, Height 1080
        Default     : Left 0, Top 0, Width 1920, Height 1080
        Pixel Aspect: 1/1
Selection Video Capture: crop_default, Left 0, Top 0, Width 1920, Height 1080, Flags:
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 1920, Height 1080, Flags:
Streaming Parameters Video Capture:
        Capabilities     : timeperframe
        Frames per second: 60.000 (60/1)
        Read buffers     : 0

I have tried using various methods opencv but ffmpeg came the closest

With below command i am able to get good results but not what i want

ffmpeg -y -f v4l2 -pix_fmt NV12 -video_size 1920x1080 -i /dev/video0 -pix_fmt bgra -frames:v 10 webcam%03d.bmp

Reference Image

RGB of Reference image

RGB of captured image

Note :- I am able to capture fine with Aforge on windows, but not with ffmpeg on linux. Would like to know if anyone has already got solution to this.

Thanks in advance.

Answer 1

We have to mark the input as BT.709 with "TV Range":

ffmpeg -y -f v4l2 -pix_fmt nv12 -video_size 1920x1080 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -color_range tv -i /dev/video0 -pix_fmt bgra -frames:v 10 webcam%03d.bmp

---

Marking the input color format as BT.709:

By default FFmpeg assumes BT.601 color format, and the input video applies BT.709 color format, so we have to mark the video as BT.709 using -color_primaries bt709 -color_trc bt709 -colorspace bt709 arguments.

When the input color format is BT.709, and FFmpeg convert it as it were BT.601, the result is wrong output colors.

---

Marking the range as "TV Range":
By default FFmpeg assumes assumes "Limited Range" (TV range), but we may add -color_range tv to be sure.

Note:
"TV Range" applies "Limited range" - the range of Y color channel is [16, 235] (U and V rane is [16, 240]).
(opposed to "Full range" or "PC Range" where YUV range is [0, 255]).

---

For reproducing the issue, we may use synthetic video (instead of camera).

Create a reference image:
ffmpeg -y -f lavfi -i testsrc=size=192x108:rate=1:duration=1 -pix_fmt bgra ref%03d.bmp

Create NV12 raw frame in BT.709, "Limited Range" (TV Range):
ffmpeg -y -f lavfi -i testsrc=size=192x108:rate=1:duration=1 -vf scale=out_color_matrix=bt709:out_range=tv -pix_fmt nv12 -f rawvideo in.nv12

Convert the raw frame to BMP without marking the color format and range (getting wrong colors):
ffmpeg -y -f rawvideo -pix_fmt nv12 -video_size 192x108 -i in.nv12 -pix_fmt bgra -frames:v 1 wrong_colors_out%03d.bmp

Convert the raw frame to BMP with marking the color format and range (getting correct colors):
ffmpeg -y -f rawvideo -pix_fmt nv12 -video_size 192x108 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -color_range tv -i in.nv12 -pix_fmt bgra -frames:v 1 out%03d.bmp

---

Ordered left to right:
Reference image, "wrong colors" and "correct colors":