Emulating Photoshop Opacity Mask to HLSL ShaderEffect

1.5k views Asked by At

I am trying to write a HLSL to emulate the Photoshop's opacity mask: White translate to Alpha=1(opaque), Black translate to Alpha=0(transparent) and shades of gray will translate to the range of transparency. The following HLSL code i tried has a problem, the resultant image is that the black portion is not completely transparent. Not sure why. do anyone know what might be the problem?

sampler2D input : register(s0);
sampler2D Mask : register(s1);

float4 main(float2 uv : TEXCOORD) : COLOR 
{ 


float4 color;
color = tex2D(input,uv);

float4 mask = tex2D(Mask, uv);    

// reverse the premultiply rgb values
mask.rgb = clamp(mask.rgb / mask.a, 0, 1);

float grayscale = dot(mask, float3(0.3, 0.59, 0.11));       
color.a = grayscale;    

return color;
}

opacity mask problem

1

There are 1 answers

2
icube On

Opps managed to solve my own question. I leave the answer here for those might want to know the answer. The solution is to return the multiplication of the grayscale and R,G,B channel and not just set the value in A channel.

sampler2D input : register(s0);
sampler2D Mask : register(s1);

float4 main(float2 uv : TEXCOORD) : COLOR 
{ 


float4 color;
color = tex2D(input,uv);    

float4 mask = tex2D(Mask, uv);
float grayscale = (mask.r + mask.g + mask.b) /3;

color.r = (color.r / color.a) * grayscale;
color.g = (color.g / color.a) * grayscale;
color.b = (color.b / color.a) * grayscale;
color.a = grayscale;

return color;
}