I seek to ascertain the end-product quality of various cq ("constant quality") values for the h264_nvenc video encoder.
To use the h264_nvenc encoder through ffmpeg and with a target quality, I follow the instructions in Nvidia's documentation, last paragraph of the linked anchor.
I specify a Variable Bit Rate rate control mode (rc=vbr_hq) and a target quality (cq=<some int>). This should be enough to test various values of cq. The story should end there.
HOWEVER, that only outputs the same perfect quality video of extremely large size for any cq value. (Using cq=1, cq=20, and cq=51 all output a file with the same hash.)
Per documentation, I can also specify a maxBitRate (maxrate=<some size>). If I don't specify a maxrate, the documentation indicates:
If maxBitRate is not specified, the encoder will use as many bits as needed to achieve the target quality.
But that's not the behavior I'm observing. I'm seeing the encoder use as many bits as possible, not as many bits as needed. The resultant video has an overall bitrate of ~100Mb/s. Double the source video.
If I do set a maxrate, it is observed by the encoder. But if I have to set the bitrate correctly for every cq value, what is the point of the cq value? I would just set cq=1 and test various bitrates, which would then only be applicable to a specific video.
Is there some warning or output that notes that the bitrate was insufficient to reach the target quality that I'm not seeing? That would at least allow me to brute force this.
Or am I misunderstanding the relationship between these settings?
Constant Quality does exactly what it says, it maintains a constant level of perceived quality. It will vary the compression amount to maintain that desired quality. The resulting file size is not relevant for this use case.
Adding a max rate is used to avoid bitrate spikes, for example when you send the video over a network, and you typically know what that limit is.
If you want to asses video compression quality there are algorithms like VMAF to name one.
More reading:
CRF Guide
Understanding Rate Control Modes