How to load image from memory with bitmap or byte array for image processing in ML.net

Question

I want to load images from a bitmap or byte[] in memory. Most of the samples are using files.

I want to run predictions on a video stream. We can get frames out in bitmaps or byte arrays.

public class MagsData2
{
    public Image ImageData;
}

public class MagsData
{
    public byte[] ImageData;
}   

Image btmap = Bitmap.FromFile("assets/images/img.jpg");
var images = new List<MagsData2>() { new MagsData2() { ImageData = btmap } };
var images = new List<MagsData>() { new MagsData() { ImageData = ImageToByte(btmap) } };

Using first one gives exception:

System.ArgumentOutOfRangeException: 'Could not determine an IDataView type and registered custom types for member ImageData (Parameter 'rawType')'

on line

IDataView imageDataView = mlContext.Data.LoadFromEnumerable(images);

Second one gives exception:

System.ArgumentOutOfRangeException: 'Schema mismatch for input column 'ImageData': 
expected String, got VarVector<Byte> (Parameter 'inputSchema')'

on line

var model = pipeline.Fit(data);

relevant code

var data = mlContext.Data.LoadFromEnumerable(new List<MagsData>());
            var pipeline = mlContext.Transforms.LoadImages(outputColumnName: "image", imageFolder: "", inputColumnName: nameof(MagsData.ImageData))
                .Append(mlContext.Transforms.ResizeImages(outputColumnName: "image", imageWidth: ImageNetSettings.imageWidth, imageHeight: ImageNetSettings.imageHeight, inputColumnName: "image"))
                .Append(mlContext.Transforms.ExtractPixels(outputColumnName: "image"))
                .Append(mlContext.Transforms.ApplyOnnxModel(modelFile: modelLocation, outputColumnNames: new[] { TinyYoloModelSettings.ModelOutput }, inputColumnNames: new[] { TinyYoloModelSettings.ModelInput }));
            var model = pipeline.Fit(data);

Answer 1

There are a few moving pieces but I will do my best to cover them off (and hope I get them all!).

I am assuming that you are using a pretrained model? If you are you don't need to pass your ml context any images when loading your context. I know it feels a bit funky but passing LoadFromEnumeable an empty list is ok in the case of a pretrained model. Also note you don't pass your pipeline the location of any images to process as processing an image is no longer the job of the pipeline.

Instead of loading images in the pipeline, we want to use a prediction engine which can be used to get our predictions (see https://learn.microsoft.com/en-us/dotnet/api/microsoft.ml.predictionengine-2?view=ml-dotnet). This is a generic which needs the input and the output types, so in our case I think you will want to do.

var PredictionEngine<MagsData, MagsPrediction> predictionEngine;

MagsPrediction is a class you define which matches the output of the model. In the case of TinyYolo2 it would look like

public class MagsPreidction
    {
        [ColumnName("grid")]
        public float[] PredictedLabels { get; set; }
    }

Putting it all together the prediction code looks like the below.

var emptyData = new List<MagsData2>();
var imageDataView = context.Data.LoadFromEnumerable(emptyData);

var pipeline = context.Transforms.ResizeImages(resizing: ImageResizingEstimator.ResizingKind.Fill, outputColumnName: "image", imageWidth: ImageSettings.imageWidth,imageHeight: ImageSettings.imageHeight, inputColumnName: nameof(MagsData2.Image))
Append(context.Transforms.ExtractPixels(outputColumnName: "image")).Append(context.Transforms.ApplyOnnxModel(modelFile: "LOCATION OF YOUR MODEL", outputColumnName: "grid", inputColumnName: "image"));

var model = pipeline.Fit(data);
var predictionEngine = context.Model.CreatePredictionEngine<MagsInput, MagsPrediction>(model);

var prediction = predictionEngine.Predict(new MagsData2{ Image = YOURIMAGE});

You can then use the prediction engine over and over again for each image and more of less follow the rest of the Microsoft docs to get the bounding boxes etc.

One of the best resources I have found on ML.Net is a Jon Wood's YouTube channel https://www.youtube.com/channel/UCrDke-1ToEZOAPDfrPGNdQw - some seriously good stuff and I can't recommend his channel enough.

Answer 2

You need to use the ImageType flag as follow, and use Bitmap instead of Image:

public class MagsData2
{
    [ColumnName("image")]
    [ImageType(512, 512)]
    public Bitmap ImageData;
}

You need to replace (512, 512) by the actual size of the resized image (ImageNetSettings.imageWidth, ImageNetSettings.imageHeight).

Then load the image with:

Image btmap = Bitmap.FromFile("assets/images/img.jpg");
var images = new List<MagsData2>() { new MagsData2() { ImageData = new Bitmap(btmap) } };

Then your pipeline becomes:

var data = mlContext.Data.LoadFromEnumerable(new List<MagsData2>());

var pipeline = mlContext.Transforms.ResizeImages(outputColumnName: "image", imageWidth: ImageNetSettings.imageWidth, imageHeight: ImageNetSettings.imageHeight, inputColumnName: "image")
    .Append(mlContext.Transforms.ExtractPixels(outputColumnName: "image"))
    .Append(mlContext.Transforms.ApplyOnnxModel(modelFile: modelLocation, outputColumnNames: new[] { TinyYoloModelSettings.ModelOutput }, inputColumnNames: new[] { TinyYoloModelSettings.ModelInput }));

var model = pipeline.Fit(data);

Answer 3

The only thing that solved in my case is to use lower version of Microsoft.ML (1.7.1).

Answer 4

To use ImageType

you must add

using Microsoft.ML.Transforms.Image;

and thanks for above comments, I soved my problem!