Calculating tight ortho projection matrix for shadow mapping

Question

I'm trying to calculate tight ortho projection around the camera for better shadow mapping. I'm first calculating the camera frustum 8 points in world space using basic trigonometry using fov, position, right, forward, near, and far parameters of the camera as follows:

PerspectiveFrustum::PerspectiveFrustum(const Camera* camera)
{
    float height = tanf(camera->GetFov() / 2.0f) * camera->GetNear();
    float width = height * Screen::GetWidth() / Screen::GetHeight();
    glm::vec3 nearTop = camera->GetUp() * camera->GetNear() * height;
    glm::vec3 nearRight = camera->GetRight() * camera->GetNear() * width;
    glm::vec3 nearCenter = camera->GetEye() + camera->GetForward() * camera->GetNear();
    glm::vec3 farTop = camera->GetUp() * camera->GetFar() * height;
    glm::vec3 farRight = camera->GetRight() * camera->GetFar() * width;
    glm::vec3 farCenter = camera->GetEye() + camera->GetForward() * camera->GetFar();
    m_RightNearBottom = nearCenter + nearRight - nearTop;
    m_RightNearTop = nearCenter + nearRight + nearTop;
    m_LeftNearBottom = nearCenter - nearRight - nearTop;
    m_LeftNearTop = nearCenter - nearRight + nearTop;
    m_RightFarBottom = farCenter + nearRight - nearTop;
    m_RightFarTop = farCenter + nearRight + nearTop;
    m_LeftFarBottom = farCenter - nearRight - nearTop;
    m_LeftFarTop = farCenter - nearRight + nearTop;
}

Then I calculate the frustum in light view and calculating the min and max point in each axis to calculate the bounding box of the ortho projection as follows:

inline glm::mat4 GetView() const
{
    return glm::lookAt(m_Position, glm::vec3(0.0f, 0.0f, 0.0f), glm::vec3(0.0f, 1.0f, 0.0f));
}

glm::mat4 DirectionalLight::GetProjection(const Camera& camera) const
{
    PerspectiveFrustum frustum = camera.GetFrustum();
    glm::mat4 lightView = GetView();
    std::array<glm::vec3, 8> frustumToLightView
    {
        lightView * glm::vec4(frustum.m_RightNearBottom, 1.0f),
        lightView * glm::vec4(frustum.m_RightNearTop, 1.0f),
        lightView * glm::vec4(frustum.m_LeftNearBottom, 1.0f),
        lightView * glm::vec4(frustum.m_LeftNearTop, 1.0f),
        lightView * glm::vec4(frustum.m_RightFarBottom, 1.0f),
        lightView * glm::vec4(frustum.m_RightFarTop, 1.0f),
        lightView * glm::vec4(frustum.m_LeftFarBottom, 1.0f),
        lightView * glm::vec4(frustum.m_LeftFarTop, 1.0f)
    };

    glm::vec3 min{ INFINITY, INFINITY, INFINITY };
    glm::vec3 max{ -INFINITY, -INFINITY, -INFINITY };
    for (unsigned int i = 0; i < frustumToLightView.size(); i++)
    {
        if (frustumToLightView[i].x < min.x)
            min.x = frustumToLightView[i].x;
        if (frustumToLightView[i].y < min.y)
            min.y = frustumToLightView[i].y;
        if (frustumToLightView[i].z < min.z)
            min.z = frustumToLightView[i].z;

        if (frustumToLightView[i].x > max.x)
            max.x = frustumToLightView[i].x;
        if (frustumToLightView[i].y > max.y)
            max.y = frustumToLightView[i].y;
        if (frustumToLightView[i].z > max.z)
            max.z = frustumToLightView[i].z;
    }
    return glm::ortho(min.x, max.x, min.y, max.y, min.z, max.z);
}

Doing this gives me empty shadow map, so something clearly wrong and I haven't being doing this right. Can someone help me by telling me what I'm doing wrong and why?

EDIT: As said my calculations of the frustum were wrong and I've changed them to the following:

PerspectiveFrustum::PerspectiveFrustum(const Camera* camera)
{
    float nearHalfHeight = tanf(camera->GetFov() / 2.0f) * camera->GetNear();
    float nearHalfWidth = nearHalfHeight * Screen::GetWidth() / Screen::GetHeight();
    float farHalfHeight = tanf(camera->GetFov() / 2.0f) * camera->GetFar();
    float farHalfWidth = farHalfHeight * Screen::GetWidth() / Screen::GetHeight();

    glm::vec3 nearCenter = camera->GetEye() + camera->GetForward() * camera->GetNear();
    glm::vec3 nearTop = camera->GetUp() * nearHalfHeight;
    glm::vec3 nearRight = camera->GetRight() * nearHalfWidth;

    glm::vec3 farCenter = camera->GetEye() + camera->GetForward() * camera->GetFar();
    glm::vec3 farTop = camera->GetUp() * farHalfHeight;
    glm::vec3 farRight = camera->GetRight() * farHalfWidth;

    m_RightNearBottom = nearCenter + nearRight - nearTop;
    m_RightNearTop = nearCenter + nearRight + nearTop;
    m_LeftNearBottom = nearCenter - nearRight - nearTop;
    m_LeftNearTop = nearCenter - nearRight + nearTop;
    m_RightFarBottom = farCenter + farRight - farTop;
    m_RightFarTop = farCenter + farRight + farTop;
    m_LeftFarBottom = farCenter - farRight - farTop;
    m_LeftFarTop = farCenter - farRight + farTop;
}

Also flipped the z coordinates when creating the ortho projection as follows:

return glm::ortho(min.x, max.x, min.y, max.y, -min.z, -max.z);

Yet still nothing renders to the depth map. Any ideas?
Here's captured results as you can see top left corner quad shows the shadow map which is completely wrong even drawing shadows on the objects themselves as a result as can be seen: https://gfycat.com/brightwealthybass

(The smearing of the shadow map values is just an artifact of the gif compresser I used it doesn't really happen so there's no problem of me not clearing the z-buffer of the FBO)

EDIT2:: Ok few things GetFov() returned degrees and not radians.. changed it. I Also try the transformation from NDC to world space with the following code:

    glm::mat4 inverseProjectViewMatrix = glm::inverse(camera.GetProjection() * camera.GetView());

    std::array<glm::vec4, 8> NDC =
    {
        glm::vec4{-1.0f, -1.0f, -1.0f, 1.0f},
        glm::vec4{1.0f, -1.0f, -1.0f, 1.0f},
        glm::vec4{-1.0f, 1.0f, -1.0f, 1.0f},
        glm::vec4{1.0f, 1.0f, -1.0f, 1.0f},
        glm::vec4{-1.0f, -1.0f, 1.0f, 1.0f},
        glm::vec4{1.0f, -1.0f, 1.0f, 1.0f},
        glm::vec4{-1.0f, 1.0f, 1.0f, 1.0f},
        glm::vec4{1.0f, 1.0f, 1.0f, 1.0f},
    };

    for (size_t i = 0; i < NDC.size(); i++)
    {
        NDC[i] = inverseProjectViewMatrix * NDC[i];
        NDC[i] /= NDC[i].w;
    }

For the far coordinates of the frustum they're equal to my calculation of the frustum, but for the near corners they're off as if my calculation of the near corners is halved by 2 (for x and y only). For example: RIGHT TOP NEAR CORNER: my calculation yields - {0.055, 0.041, 2.9} inverse NDC yields - {0.11, 0.082, 2.8}

So I'm not sure where my calculation got wrong, maybe you could point out? Even with the inversed NDC coordinates I tried to use them as following:

glm::mat4 DirectionalLight::GetProjection(const Camera& camera) const
{
    glm::mat4 lightView = GetView();

    glm::mat4 inverseProjectViewMatrix = glm::inverse(camera.GetProjection() * camera.GetView());

    std::array<glm::vec4, 8> NDC =
    {
        glm::vec4{-1.0f, -1.0f, 0.0f, 1.0f},
        glm::vec4{1.0f, -1.0f, 0.0f, 1.0f},
        glm::vec4{-1.0f, 1.0f, 0.0f, 1.0f},
        glm::vec4{1.0f, 1.0f, 0.0f, 1.0f},
        glm::vec4{-1.0f, -1.0f, 1.0f, 1.0f},
        glm::vec4{1.0f, -1.0f, 1.0f, 1.0f},
        glm::vec4{-1.0f, 1.0f, 1.0f, 1.0f},
        glm::vec4{1.0f, 1.0f, 1.0f, 1.0f},
    };

    for (size_t i = 0; i < NDC.size(); i++)
    {
        NDC[i] = lightView * inverseProjectViewMatrix * NDC[i];
        NDC[i] /= NDC[i].w;
    }

    glm::vec3 min{ INFINITY, INFINITY, INFINITY };
    glm::vec3 max{ -INFINITY, -INFINITY, -INFINITY };
    for (unsigned int i = 0; i < NDC.size(); i++)
    {
        if (NDC[i].x < min.x)
            min.x = NDC[i].x;
        if (NDC[i].y < min.y)
            min.y = NDC[i].y;
        if (NDC[i].z < min.z)
            min.z = NDC[i].z;

        if (NDC[i].x > max.x)
            max.x = NDC[i].x;
        if (NDC[i].y > max.y)
            max.y = NDC[i].y;
        if (NDC[i].z > max.z)
            max.z = NDC[i].z;
    }
    return glm::ortho(min.x, max.x, min.y, max.y, min.z, max.z);
}

And still got bad result: https://gfycat.com/negativemalealtiplanochinchillamouse

Answer 1

Let's start with your frustum calculation here:

float height = tanf(camera->GetFov() / 2.0f) * camera->GetNear();
[...]
glm::vec3 nearTop = camera->GetUp() * camera->GetNear() * height;
[...]
glm::vec3 farTop = camera->GetUp() * camera->GetFar() * height;

That's one to many GetNear in your multiplications. Conceptually, you could height represent half of the frustum height at unit distance (I still would name it differently) without projecting it to the near plane, then the rest of your formulas make more sense.

However, the whole approach is doubtful to begin with. To get the frustum corners in world space, you can simply unproject all 8 vertices of the [-1,1]^3 NDC cube. Since you want to transform that into your light space, you can even combine it to a single matrix m = lightView * inverse(projection * view), just don't forget the perspective divide after the multiplying the NDC cube vertices.

return glm::ortho(min.x, max.x, min.y, max.y, min.z, max.z);

Standard GL conventions use a view space where the camera is looking into negative z direction, but the zNear and zFar parameters are interpreted as distances along the viewing directions, so the actual viewing volume will range from -zFar, -zNear in view space. You'll have to flip the signs of your z dimension to get the actual bounding box you're looking for.