Color brightness/lightness acceptable range

Question

I'm creating a new color from two given colors; a source color and a target color. The new color is composed of the hue and saturation components of the target color but uses the brightness (aka lightness or luminance) of the source color.

var hue: CGFloat = 0.0
var saturation: CGFloat = 0.0
var newBrightness: CGFloat = 0.0
var oldBrightness: CGFloat = 0.0
var alpha: CGFloat = 0.0

oldColor.getHue(
  &hue,
  saturation: &saturation,
  brightness: &oldBrightness,
  alpha: &alpha
)

newColor.getHue(
  &hue,
  saturation: &saturation,
  brightness: &newBrightness,
  alpha: &alpha
)

let color = UIColor(
  hue: hue,
  saturation: saturation,
  brightness: oldBrightness,
  alpha: alpha
)

I use the new color as part of a scanline flood fill implementation where each pixel in a given image provides the source color and the target color is selected by the user.

The problem with this implementation reveals itself when the user selects a dark color (i.e. black). Using the source color's brightness with the target color may yield a completely different color.

I suspect that the brightness given to the new color is outside the acceptable range of values for that color. If just use the brightness for the target color, the painted area of the image will be completely flat. How can I keep the brightness relative to the original pixel so that the image as a whole maintains depth?