How to use two MPSImages at the input of MPSNNGraph

Question

I am trying to enter two images on MPSNNGraph.

However, even if I input an array like [input1, input2] on "withSourceImages", I can only input "input1" as input image. Ideally, when creating a graph as below, I want to make "inputImage1" as "input1" and "inputImage2" as "input2".

Actually, when I ran it like this and looked at the result of "concat", I was able to see what "input1" was concatenated, not "input2".

The graph look like:

let inputImage1 = MPSNNImageNode(handle: nil)
let inputImage2 = MPSNNImageNode(handle: nil)
let scale = MPSNNBilinearScaleNode(source: inputImage1,
                                   outputSize: MTLSize(width:256,
                                                       height: 256,
                                                       depth: 3))
let scale2 = MPSNNBilinearScaleNode(source: inputImage1,
                                   outputSize: MTLSize(width:64,
                                                       height: 64,
                                                       depth: 3))
...

let concat = MPSNNConcatenationNode(sources: [conv3.resultImage, scale2.resultImage])

...

if let graph = MPSNNGraph(device: commandQueue.device,
                          resultImage: tanh.resultImage,
                          resultImageIsNeeded: true){
        self.graph = graph
}

and part of encoding graph look like:

let input1 = MPSImage(texture: texture, ...)
let input2 = MPSImage(texture: texture2, ...)
graph.executeAsync(withSourceImages: [input1, input2]) { outputImage, error in
    ...
}

How do I enter the second input and the graph receive it?

Could you give me some advice?

Answer 1

The code you provide actually looks correct. Referencing the MPSNNGraph.h header here:

*  @param  sourceImages    A list of MPSImages to use as the source images for the graph.
*                          These should be in the same order as the list returned from
*                          MPSNNGraph.sourceImageHandles. They should be allocated against
*                          the same MTLDevice. There must be at least one source image.
*                          Note: this array is intended to handle the case where multiple
*                          input images are required to generate a single graph result.
*                          That is, the graph itself has multiple inputs.  If you need to
*                          execute the graph multiple times, then call this API multiple
*                          times, or better yet use [encodeToCommandBuffer:sourceImages:]
*                          multiple times.

I want to point out though, that MPSNNConcatenationNode behaves in a pretty unique way. It always concat on the depth(channel) dimension. When concating images with different spacial dimensions, it will respect the smaller one (i.e. 2x2x10 concat 4x4x15 -> 2x2x25). Maybe that is where your issue came from.

Answer 2

Scale2 uses the same input image as Scale1 in this example. When the MPSNNGraph analyzes this set of nodes, it never encounters inputImage2 because it is not used by the graph. Because it is not in the graph, the graph interface will not accept a second input. It wouldn't know what to do with it.

It seems likely to me that if you change the scale2 initialization to use inputImage2 then it will work as you probably intended.