C# 16-bit grayscale Bitmap array to video

Question

I have some scientific data that comes in 16-bit grayscale and is unsigned. I need the wide range of grayscale values and C# already supports the format for Bitmap structure using PixelFormat.Format16bppGrayScale, I am conducting some operations on the data and saving it into a List and I would like to write it into a video file for viewing.

I have looked for many FFMPEG libraries out there and I saw Accord.Video.FFMPEG which does support 32-bit ARGB, when I converted the Bitmaps into ARGB it worked and I could write the video file like this:

  int height = images[0].Height;
  int width = images[0].Width;
  var frameRate = 8;

  // create instance of video writer
  var vFWriter = new VideoFileWriter();
  // create new video file
  vFWriter.Open(outputVid + ".mp4", width, height, frameRate, VideoCodec.MPEG4);
  
  
  for(int i=0; i<images.Count; i++)
  {
    vFWriter.WriteVideoFrame(images[i]);  // MAIN LINE OF FOCUS
  }
  vFWriter.Close();
  vFWriter.Dispose();

As can be seen in the code there is no Stride nor PixelFormat provided to the VideoFileWriter and as per the documentation there are no such properties in the class.

I saw a similar issue people had with C++ and OpenCV2 where the only codec that supports 16-bit grayscale was FFV1. I knew it was a long shot but I tried it:

  vFWriter.Open(outputVid + ".mkv", width, height, frameRate, VideoCodec.FFV1);

However, as expected, it yielded no results.

The issue is an exception that is thrown stating the image being written into the VideoFileWriter about the argument being invalid:

Unhandled Exception: System.ArgumentException: Parameter is not valid.
at System.Drawing.Bitmap.LockBits(Rectangle rect, ImageLockMode flags, PixelFormat format, BitmapData bitmapData)
at System.Drawing.Bitmap.LockBits(Rectangle rect, ImageLockMode flags, PixelFormat format)
at Accord.Video.FFMPEG.VideoFileWriter.WriteVideoFrame(Bitmap frame, UInt32 frameIndex)

This is a common error when the PixelFormat is mismatched (the video writer wants 32-bit ARGB but it is receiving 16-bit grayscale).

I confirmed this was the case by updating the write code to this:

  for(int i=0; i<images.Count; i++)
  {
    var data = images[i].LockBits(new Rectangle(System.Drawing.Point.Empty, images[i].Size), ImageLockMode.ReadOnly, PixelFormat.Format16bppGrayScale);
    vFWriter.WriteVideoFrame(data);
    images[i].UnlockBits(data);
  }

And now the exception thrown says:

Unhandled Exception: System.ArgumentException: The provided bitmap must be 24 or 32 bpp color image or 8 bpp grayscale image.

I cannot afford to convert to 32-bit ARGB because that only allows 256 of grayscale and I need the whole 16-bit range.

I looked at other libraries that handle FFMPEG with C# and found Xabe.FFMPEG but it does not support constructing a video from images.

Are there libraries other than Accord.Video.FFMPEG which are capable of constructing a video from 16-bit grayscale frames?

Answer 1

There are as far as I know no good video codecs with 16 bit grayscale support, and I have looked.

One problem is that you typically need to do some kind of "window leveling" to select what portion of the data to show, instead of just throwing away the lower 8 bits. So even if you could encode and decode the data you would have difficulty making the playback look good. I have so far not seen any public image viewer that handled 16bit grayscale data well, let alone any video viewer.

A few possible options:

Custom format

We use a zip-archive as a container, with individual jpegls encoded frames, (png or jpeg2000) will also work. Compression ratio is not great, and you need custom tools for both recording and playback. But it will provide the best control over the data and playback.

Use 12 bit video

While full 16 bit video seem to be just about impossible, h265 and some other support 12 bit video. If that is enough for you it should give the best compression ratio by far, and should be compatible with most players. A potential problem is that I have not found any free encoding library that allow commercial use.

Use different color channels

You could split the 16 bit grayscale value into two 8 bit values and store them in different color channels in an RGB video. This will require a lossless video codec, but that should be possible with h265 or other state of the art codecs. It should give better compression, but would look horrible when viewed in a regular video player.

Separate the viewing file from the scientific file

Produce a regular 8 bit grayscale video file for people to view, while keeping another file with the full data to be used by other programs.