Smoothen Edges of Chroma Key -CoreImage

Question

I'm using the following code to remove green background from an image.But the edges of the image has green tint to it and some pixels are damaged.How can i smoothen this and make the cut out perfect.

 func chromaKeyFilter(fromHue: CGFloat, toHue: CGFloat) -> CIFilter?
    {
        // 1
        let size = 64
        var cubeRGB = [Float]()

        // 2
        for z in 0 ..< size {
            let blue = CGFloat(z) / CGFloat(size-1)
            for y in 0 ..< size {
                let green = CGFloat(y) / CGFloat(size-1)
                for x in 0 ..< size {
                    let red = CGFloat(x) / CGFloat(size-1)

                    // 3
                    let hue = getHue(red: red, green: green, blue: blue)
                    let alpha: CGFloat = (hue >= fromHue && hue <= toHue) ? 0: 1

                    // 4
                    cubeRGB.append(Float(red * alpha))
                    cubeRGB.append(Float(green * alpha))
                    cubeRGB.append(Float(blue * alpha))
                    cubeRGB.append(Float(alpha))
                }
            }
        }



  @IBAction func clicked(_ sender: Any) {
          let a = URL(fileURLWithPath:"green.png")
          let b = URL(fileURLWithPath:"back.jpg")

        let image1 = CIImage(contentsOf: a)
        let image2 = CIImage(contentsOf: b)





        let chromaCIFilter = self.chromaKeyFilter(fromHue: 0.3, toHue: 0.4)
        chromaCIFilter?.setValue(image1, forKey: kCIInputImageKey)
        let sourceCIImageWithoutBackground = chromaCIFilter?.outputImage

        /*let compositor = CIFilter(name:"CISourceOverCompositing")
        compositor?.setValue(sourceCIImageWithoutBackground, forKey: kCIInputImageKey)
        compositor?.setValue(image2, forKey: kCIInputBackgroundImageKey)
        let compositedCIImage = compositor?.outputImage*/

        var rep: NSCIImageRep = NSCIImageRep(ciImage: sourceCIImageWithoutBackground!)
        var nsImage: NSImage = NSImage(size: rep.size)
        nsImage.addRepresentation(rep)

        let url = URL(fileURLWithPath:"file.png")

        nsImage.pngWrite(to: url)
        super.viewDidLoad()
    }

Input:

Output:

Update:

Update 2:

Update 3:

Answer 1

Professional tools for chroma keying usually include what's called a spill suppressor. A spill suppressor finds pixels that contain small amounts of the chroma key color and shifts the color in the opposite direction. So green pixels will move towards magenta. This reduces the green fringing you often see around keyed footage.

The pixels you call damaged are just pixels that had some level of the chroma color in them and are being picked up by your keyer function. Rather than choosing a hard 0 or 1, you might consider a function that returns a value between 0 and 1 based on the color of the pixel. For example, you could find the angular distance of the current pixel's hue to the fromHue and toHue and maybe do something like this:

// Get the distance from the edges of the range, and convert to be between 0 and 1
var distance: CGFloat
if (fromHue <= hue) && (hue <= toHue) {
    distance = min(abs(hue - fromHue), abs(hue - toHue)) / ((toHue - fromHue) / 2.0)
} else {
    distance = 0.0
}
distance = 1.0 - distance
let alpha = sin(.pi * distance - .pi / 2.0) * 0.5 + 0.5

That will give you a smooth variation from the edges of the range to the center of the range. (Note that I've left off dealing with the fact that hue wraps around at 360°. That's something you'll have to handle.) The graph of the falloff looks like this:

Another thing you can do is limit the keying to only affect pixels where the saturation is above some threshold and the value is above some threshold. For very dark and/or unsaturated colors, you probably don't want to key it out. I think that would help with the issues you're seeing with the model's jacket, for example.

Answer 2

My (live) keyer works like this (with the enhancements user1118321 describes) and using its analyser I quickly noticed this is most likely not a true green screen image. It's one of many fake ones where the green screen seems to have been replaced with a saturated monochrome green. Though this may look nice, it introduces artefacts where the keyed subject (with fringes of the originally used green) meets the monochrome green. You can see a single green was used by looking at the histogram. Real green screen always have (in)visible shades of green. I was able to get a decent key but had to manually tweak some settings. With a NLE you can probably get much better results, but they will also be a lot more complex. So to get back to your issue, your code probably works as it is now (update #3), you just have to use a proper real life green screen image.