Metal Core Image Sampler pixel format

Question

I have this Metal code that I want to translate to Metal CoreImage kernel.

    fragment half4 fragmentShaderThreshold ( MappedVertex in [[ stage_in ]],
                                 texture2d<half, access::sample> inputTexture [[ texture(0) ]]
                                 )
  {
  constexpr sampler s(s_address::clamp_to_edge, t_address::clamp_to_edge, min_filter::linear, mag_filter::linear);

  half3 rgb = inputTexture.sample(s, in.textureCoordinate).rgb; 
  half3 weights = half3(0.2126, 0.7152, 0.0722);

  float intensity = dot(rgb, weights);

  if (intensity >= 0.95) {
     rgb = half3(1.0, 0.0, 0.0);
  } else if (intensity <= 0.05) {
     rgb = half3(0.0, 0.0, 1.0);
  }

    return half4(rgb, 1.h);
 }

To do the same in Metal Core Image kernel, I think this is something that can be done with builtin filters. Because builtin filters can work in variety of color spaces such as BT.2020. Please comment on how this can be achieved using builtin filters.

If it is not possible using builtin filters, I was thinking of converting the image to grayscale and read the intensity (gray value) in Metal Core Image kernel. It looks like Metal Core Image samplers support only RGB based pixel formats (unlike Metal where textures can have pixel format such as r8). Is that correct?

Answer 1

Unfortunately, there is no simple built-in filter for converting an image to grayscale. CIPhotoEffectNoir does it, but it uses a color cube / look up table under the hood, and it's not documented how it is defined.

You could probably use CIColorMatrix to do the first part of the kernel:

let colorMatrixFilter = CIFilter.colorMatrix()
let lumaCoefficients = CIVector(x: 0.2126, y: 0.7152, z: 0.0722, w: 0.0)
colorMatrixFilter.rVector = lumaCoefficients
colorMatrixFilter.bVector = lumaCoefficients
colorMatrixFilter.gVector = lumaCoefficients

You could further adjust the luma coefficients to match the color space you are working in, though Core Image converts everything to extended linear sRGB by default, so these should be correct.

For the intensity thresholding you could probably use some clever combination of CIColorThreshold and CIBlendWithMask, but honestly, I would write a simple custom color kernel for that. It should also be faster at runtime.

And yes, Core Image always works with 4-channel textures. In a custom kernel that only produces a single-channel output, you usually write the same value in all 3 color channels. Alternatively, you can also specify a single-channel outputPixelFormat such as .Rh when initializing your kernel with init(functionName: String, fromMetalLibraryData: Data, outputPixelFormat: CIFormat), but CI will swizzle it back to a 4-channel image anyway before passing it as input to the next filter (unless that's a processor that explicitly works with single-channel inputs, but that's rare).