Shadow mapping in Opengl-es 2.0 with cubemap

Question

I'm quite new to OpenGL ES and I'm trying to add some shadows to my current scene. I decided to do that with the help of a cubemap. I'm using OpenGL es 2.0, so geometry shader or the gl_FragDepth variable is unavailable for me. I've googled some so I got some insight about the topic and reading this (http://www.cg.tuwien.ac.at/courses/Realtime/repetitorium/2010/OmnidirShadows.pdf) proved quite useful. Basically I rely on this linked doc.

But something is wrong with my code, because in the rendered scene every pixel is under shadow. I think the problem can be found in my shaders, but I paste here all my relevant code to see everything clearly.

Setup framebuffer and creating the cubemap:

GLuint FBO;
GLuint cubeTexture;

glGenFramebuffers(1, &FBO);
glBindFramebuffer(GL_FRAMEBUFFER, FBO);

// Depth texture
glGenTextures(1, &cubeTexture);
glBindTexture(GL_TEXTURE_CUBE_MAP, cubeTexture);

glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
//glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_WRAP_R, GL_CLAMP_TO_EDGE); // no GL_TEXTURE_WRAP_R

// right, left, top, bottom, front, back
for (int face = 0; face < 6; ++face) {
    // create space for the textures, content need not to be specified (last parameter is 0)
    glTexImage2D(GL_TEXTURE_CUBE_MAP_POSITIVE_X + face, 0, GL_DEPTH_COMPONENT16, 1024, 768, 0, GL_DEPTH_COMPONENT, GL_FLOAT, 0);
}

glBindTexture(GL_TEXTURE_CUBE_MAP, 0);

glBindFramebuffer(GL_FRAMEBUFFER, 0);

In the Display function I try to achieve 6 render passes to fill the shadow maps (the 6 sides of the cube).

Rendering:

glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glClearColor(0.0f, 0.0f, 0.0f, 0.0f);
glClearDepthf(1.0f);

glm::vec3 camPos = ... // position of the camera, it is in world space
glm::vec4 lightPos = ... // position of the light source, it is in world space

// 1. render into texture

float zNear = 0.1f;
float zFar = 100.0f;
// or should I use ortho instead of perspective?
glm::mat4 projMatrix = glm::perspective(90.0f, (float)esContext->width / esContext->height, zNear, zFar);

// The 6 cameras have to be placed to the light source and they need the proper view matrices 

glm::mat4 cubeMapMatrices[6]; // contains six basic and defined rotation matrices for the six directions 
cubeMapMatrices[0] = glm::make_mat4(rotPositiveX);
cubeMapMatrices[1] = glm::make_mat4(rotNegativeX);
cubeMapMatrices[2] = glm::make_mat4(rotPositiveY);
cubeMapMatrices[3] = glm::make_mat4(rotNegativeY);
cubeMapMatrices[4] = glm::make_mat4(rotPositiveZ);
cubeMapMatrices[5] = glm::make_mat4(rotNegativeZ);

glm::vec4 translation = lightPos;

glBindTexture(GL_TEXTURE_CUBE_MAP, cubeTexture);

glBindFramebuffer(GL_FRAMEBUFFER, FBO);

for (int face = 0; face < 6; ++face) {

    glClear(GL_DEPTH_BUFFER_BIT); // do I need this here?

    cubeMapMatrices[face][3] = translation; // the translation part is the same for all
    cubeMapMatrices[face] = projMatrix * cubeMapMatrices[face]; // now it's an mvp matrix

    glUniformMatrix4fv(cubeProjectionMatrixID, 1, GL_FALSE, glm::value_ptr(cubeMapMatrices[face]));

    // Attach depth cubemap texture to FBO's depth attachment point
    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_TEXTURE_CUBE_MAP_POSITIVE_X + face, cubeTexture, 0);

    int err = glCheckFramebufferStatus(GL_FRAMEBUFFER);
    if (err != GL_FRAMEBUFFER_COMPLETE) {
        std::cout << "Framebuffer error, status: " << err << std::endl;
    }

    RenderScene(); // do the drawing 
}

glBindFramebuffer(GL_FRAMEBUFFER, 0);

// 2. render into screen

glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

RenderScene(); // do the drawing

// swap the buffers

So here come the different shaders. I have one vertex shader and one fragment shader for the depth calculation and one vertex shader and one fragment shader for the screen rendering. My problem is that I'm not sure how to write into the cubemap, if the framebuffer is in use, then will the gl_Position determine the coordinates in the given cube face?

vertex shader for depth calculating:

in vec3 vPosition;

uniform mat4 mModel;
uniform mat4 mCubeProjection;
uniform vec4 vLight;

out vec4 vFrag Position; // world space
out vec4 vLightPosition; // mvp transformed

void main()
{
    vec4 position = vec4(vPosition, 1.0);
    vFragPosition  = mModel* position;
    vLightPosition = vLight;
    gl_Position = mCubeProjection * vFragPosition;
}

fragment shader for depth calculating:

in vec4 vFragPosition; // world space
in vec4 vLightPosition; // world space

out float depthValue;

void main()
{
    depthValue = distance(vFragPosition.xyz, vLightPosition.xyz); // need normalization?
}

vertex shader for rendering to the screen:

uniform mat4 mModel;
uniform mat4 mView;
uniform mat4 mProjection;
uniform vec4 vLight; // it is in world space

out vec3 lw; // light position in world space
out vec3 pw; // pixel position in world space

void main()
{
    vec4 position = vec4(vPosition, 1.0);
    lw = vLight.xyz;
    pw = (mModel* position).xyz;
    gl_Position  = mProjection* mView * mModel* position;
}

fragment shader for rendering to the screen:

in vec3 lw;
in vec3 pw;

uniform samplerCube cubeMap;

out vec4 outputColor;

void main()
{
    vec3 lookup = pw - lw;
    float smValue = texture(cubeMap, lookup).r; // retrieves texels from a texture (d, d, d, 1.0)
    float distance = length(lookup); // dist from the fragment to the light

    float eps = 0.1;
    float shadowVal = 1.0;

    if (smValue + eps < distance) {
        shadowVal = 0.1; // in shadow
    }

    // here comes the lighting stuff
    // ...

    outputColor =  outputColor * shadowVal;
}

So again, the issue is that every pixel falls under shadow. From the code I excluded some uniform passes to the shader, but they are ok. Can you give me an advice what should I fix in the code? Are my shaders (especially for the first pass) correct, do I set the transformations for the cube mapping properly? Thank you.

P.S: This is my first question here, I hope it is clear enough and fulfils the requirements of a correctly posted question.

Answer 1

Fundamentally your problem is quite simple: you are using a depth attachment for your cube map, and the depth buffer stores perspective depth.

What your shader expects to see in the shadowmap is the non-perspective distance from the light to the nearest fragment. You actually went ahead and calculated this in your fragment shader for the shadowmap creation, but you output it to the color buffer (which has nothing attached to it) instead of the depth buffer.

---

There are at least three possible solutions to this problem (in order of performance, worst first):

1. Write to gl_FragDepth instead of depthValue (which is actually a color buffer target).

2. Attach your cubemap to GL_COLOR_ATTACHMENT0 and use a color-renderable format instead of GL_DEPTH_COMPONENT.

3. Remove depthValue from your shader and ONLY write perspective depth.

Option 1 is absolutely awful, even though I have seen people reference tutorials that suggest doing this. If you write to gl_FragDepth you will disable hardware depth buffer optimizations on a lot of hardware and this will make the pass to generate your shadow maps perform worse.

Option 2 is better, because it benefits from hardware Z-buffer optimizations but it still requires a lot of memory bandwidth because you are effectively storing two different depth values (one in the color attachment and one in the depth attachment).

Option 3, while the most complicated, is also generally the best performing. That is because you can cut the memory bandwidth needed to compute your shadowmaps in half by only storing hardware depth.

If you are interested in learning more about Option 3, I would suggest you take a look at this related question. Instead of comparing the distance, you will actually compute a perspective depth to use for comparison. You trade a few extra computations in the fragment shader when applying shadows for a lot less memory bandwidth when creating the shadowmaps.

---

Now to solve your immediate problem with the least amount of work, you should pursue Option 2. Leave Option 3 on the table for a future optimization; it is best not to optimize things until you have at least something working.