Point light Dual-paraboloid VSM in deferred rendering

Question

I've been following this tutorial to implement my variance shadow mapping feature for point light in deferred rendering.

I'm using GLSL 3.3, left-handed coordinate system. Here is what I've been doing:

I render the scene to dual-paraboloid maps, storing depth and depth * depth.

Result:

Above image contains front and back maps. The point light is at the center of scene, you can see where it glows yellow the most.

Then I set up a full-screen shader pass. I do this by transforming the tutorial code from FX to GLSL.

Author's .fx code:

    float4 TexturePS(float3 normalW : TEXCOORD0, float2 tex0 : TEXCOORD1, float3 pos : TEXCOORD2) : COLOR
    {
        float4 texColor = tex2D(TexS, tex0 * TexScale);

        pos = mul(float4(pos, 1.0f), LightView);

        float L = length(pos);
        float3 P0 = pos / L;

        float alpha = .5f + pos.z / LightAttenuation;

        P0.z = P0.z + 1;
        P0.x = P0.x / P0.z;
        P0.y = P0.y / P0.z;
        P0.z = L / LightAttenuation;

        P0.x = .5f * P0.x + .5f;
        P0.y = -.5f * P0.y + .5f;

        float3 P1 = pos / L;

        P1.z = 1 - P1.z;
        P1.x = P1.x / P1.z;
        P1.y = P1.y / P1.z;
        P1.z = L / LightAttenuation;

        P1.x = .5f * P1.x + .5f;
        P1.y = -.5f * P1.y + .5f;

        float depth;
        float mydepth;
        float2 moments;
        if(alpha >= 0.5f)
        {
            moments = tex2D(ShadowFrontS, P0.xy).xy;
            depth = moments.x;
            mydepth = P0.z;
        }
        else
        {
            moments = tex2D(ShadowBackS, P1.xy).xy;
            depth = moments.x;
            mydepth = P1.z;
        }

        float lit_factor = (mydepth <= moments[0]);

        float E_x2 = moments.y;
        float Ex_2 = moments.x * moments.x;
        float variance = min(max(E_x2 - Ex_2, 0.0) + SHADOW_EPSILON, 1.0);
        float m_d = (moments.x - mydepth);
        float p = variance / (variance + m_d * m_d); //Chebychev's inequality

        texColor.xyz *= max(lit_factor, p + .2f);

        return texColor;
    }

My translated GLSL code:

    void main() {
        vec3 in_vertex = texture(scenePosTexture, texCoord).xyz; // get 3D vertex from 2D screen coordinate
        vec4 vert = lightViewMat * vec4(in_vertex, 1); // project vertex to point light space (view from light position, look target is -Z)

        float L = length(vert.xyz);

        float distance = length(lightPos - in_vertex);
        float denom = distance / lightRad + 1;
        float attenuation = 1.0 / (denom * denom);

        // to determine which paraboloid map to use
        float alpha = vert.z / attenuation + 0.5f;

        vec3 P0 = vert.xyz / L;

        P0.z = P0.z + 1;
        P0.x = P0.x / P0.z;
        P0.y = P0.y / P0.z;
        P0.z = L / attenuation;

        P0.x = .5f * P0.x + .5f;
        P0.y = -.5f * P0.y + .5f;

        vec3 P1 = vert.xyz / L;

        P1.z = 1 - P1.z;
        P1.x = P1.x / P1.z;
        P1.y = P1.y / P1.z;
        P1.z = L / attenuation;

        P1.x = .5f * P1.x + .5f;
        P1.y = -.5f * P1.y + .5f;

        // Variance shadow mapping
        float depth;
        float mydepth;
        vec2 moments;

        if(alpha >= 0.5f)
        {
            moments = texture(shadowMapFrontTexture, P0.xy).xy;
            depth = moments.x;
            mydepth = P0.z;
        }
        else
        {
            moments = texture(shadowMapBackTexture, P1.xy).xy;
            depth = moments.x;
            mydepth = P1.z;
        }

        // Original .fx code is: float lit_factor = (mydepth <= moments[0]);
        // I'm not sure my translated code belew is correct

        float lit_factor = 0;
        if (mydepth <= moments.x)
            lit_factor = mydepth;
        else
            lit_factor = moments.x;

        float E_x2 = moments.y;
        float Ex_2 = moments.x * moments.x;
        float variance = min(max(E_x2 - Ex_2, 0.0) + SHADOW_EPSILON, 1.0);
        float m_d = (moments.x - mydepth);
        float p = variance / (variance + m_d * m_d); //Chebychev's inequality

        fragColor = texture(sceneTexture, texCoord).rgb; // sample original color
        fragColor.rgb *= max(lit_factor, p + .2f);
    }

Render result

Right now I'm clueless about where I'm gonna touch to render the shadow correctly. Could someone point it out for me?

Answer 1

Some friend of mine pointed out that the Y component is flipped, that's why shadow looked like up-side down. After adding minus to P0 and P1's Y, it starts to show quite reasonable shadow:

But another problem is the location of shadow is wrong.

Answer 2

Why do you duplicate the paraboloid projection computation ?
You compute it on 'vert', then 'P0' and 'P1', shouldn't you do it only on 'P0' and 'P1' ? (The original code doesn't do this thing on 'pos').

EDIT:

Your lit_factor is wrong, it should be either 0.0 or 1.0.
You could use the step() GLSL intrinsic, in this way :
float lit_factor = step(mydepth, moments[0]);