Parralax mapping does not work correctly using OpenGL and GLSL

Question

I try to implement parralax mapping in my 3D engine using OpenGL and GLSL API but the display is not correct. To learn and apply the complexity of a such technique I was inspired by the following PDF tutorial (page 16, 17 and 18):

https://www.opengl.org/sdk/docs/tutorials/TyphoonLabs/Chapter_4.pdf

To produce a very basic parralax effect (without any lighting effect) I need to use 2 textures:

- 1 diffuse (color) texture (BPP: 24 -> RGB - format: JPEG)

- 1 displacement (height/grayscale) texture (BPP: 24 -> RGB - format: JPEG)

I used the famous and very usefull software 'CrazyBump' to generate my displacement map. Plus this software can display a 3D view of what the parralax mapping will look like in an external 3D application like mine.

In a first time, Here's the display from 'CrazyBump' (CrazyBump use lighting effect but it's not important here):

As you can see the parralax effect is correctly rendered.

And now here's the rendering in my scene (using the same displacement texture generated by 'CrazyBump' and without luminosity. All I want to see is the fake deformation of the surface like above).

As you can see, the display is not the same and of course not correct.

To try producing the same effect I apply the course in the PDF file I talked about at the beginning of my post.

For informations, I have already implemented 'normmal mapping' technique previously for my engine (so the tangent and bitangent vectors are correct!).

To execute my shader program I need the position of the camera in world space and matrices (ModelViewProj, ModelMatrix and NormalMatrix).

Here's the client C++ code I use:

glm::mat4 modelViewMatrix = pRenderBatch->GetModelViewMatrix();

glm::mat3 normalMatrix = glm::mat3(glm::vec3(modelViewMatrix[0]),
    glm::vec3(modelViewMatrix[1]), glm::vec3(modelViewMatrix[2]));

this->SetUniform("ModelViewProjMatrix", pRenderBatch->GetModelViewProjMatrix());
this->SetUniform("ModelViewMatrix", modelViewMatrix);
this->SetUniform("NormalMatrix", normalMatrix);

//Bound on channel 0
glActiveTexture(GL_TEXTURE0);
this->m_pTextureManager.PushAndBindTexture(
    pMaterial->GetDiffuseTexture());
{
    this->SetUniform("DiffuseSampler", 0);
}
//Bound on channel 1
glActiveTexture(GL_TEXTURE1);
    this->m_pTextureManager.PushAndBindTexture(
        pMaterial->GetDisplacementTexture());
{
    this->SetUniform("HeightSampler", 1);
}

The Vertex Shader:

#version 440

/*
** Vertex attributes.
*/
layout (location = 0) in vec4 VertexPosition;
layout (location = 1) in vec2 VertexTexture;
layout (location = 2) in vec3 VertexNormal;
layout (location = 3) in vec3 VertexTangent;
layout (location = 4) in vec3 VertexBitangent;

/*
** Uniform matrices.
*/
uniform mat4 ModelViewProjMatrix;
uniform mat4 ModelViewMatrix;
uniform mat3 NormalMatrix;

//Outputs
out vec2 TexCoords;
out vec3 viewDir_TS;

/*
** Vertex shader entry point.
*/
void main(void)
{
    //Texture coordinates
    TexCoords = VertexTexture;

    //Vertex position in world space
    vec3 Position_CS = vec3(ModelViewMatrix * VertexPosition);
    //Vertex normal in world space
    vec3 Normal_CS = NormalMatrix * VertexNormal;
    //Vertex tangent in world space
    vec3 Tangent_CS = NormalMatrix * VertexTangent;
    //Vertex bitangent in world space
    vec3 Bitangent_CS = NormalMatrix * VertexBitangent;

    //View vector in world space
    vec3 viewDir_CS = -Position_CS;

    //TBN matrix
    mat3 TBN =  mat3(
        Tangent_CS.x, Bitangent_CS.x, Normal_CS.x,
        Tangent_CS.y, Bitangent_CS.y, Normal_CS.y,
        Tangent_CS.z, Bitangent_CS.z, Normal_CS.z);

    //2 others ways to compute view vector in tangent space

        //mat3 TBN = transpose(mat3(Tangent_CS, Bitangent_CS, Normal_CS));

        /*viewDir_TS = vec3(
            dot(viewDir_CS, Tangent_CS),
            dot(viewDir_CS, Bitangent_CS),
            dot(viewDir_CS, Normal_CS)
        );*/

    //View vector converted in tangent space (not normalized)
    viewDir_TS = TBN * viewDir_CS;

    gl_Position = ModelViewProjMatrix * VertexPosition;
}

And finally the Fragment Shader:

#version 440

layout (location = 0) out vec4 FragColor;

//Texture coordinates
in vec2 TexCoords;

//View (camera) vector in tangent space
in vec3 viewDir_TS;

//Diffuse texture sampler
uniform sampler2D DiffuseSampler;
//Displacement texture sampler
//(height map/grayscale map)
uniform sampler2D HeightSampler;

/*
** Fragment shader entry point
*/
void main(void)
{
    //Parralax intensity {scale(s), bias(b)}
    vec2 ScaleBias = vec2(0.04f, 0.02f);

    //Height(h) range [0;1] (float) recovered from height map (HeightSampler)
    float Height = texture2D(HeightSampler, TexCoords.st).r; 

    //Height scaled and biased according to the formula: hsb = h · s + b
    float HSB = Height * ScaleBias.x + ScaleBias.y;

    //View vector in tangent space normalized
    vec3 viewDirNorm_TS = normalize(viewDir_TS);

    //Computes texture offset according to the formula: Tn = To + (hsb · V{x, y}) 
    vec2 textOffset = TexCoords + (viewDirNorm_TS.xy * HSB);

    //Computes final diffuse texture color using parralax offset
    FragColor = texture2D(DiffuseSampler, textOffset);
}

I tried to modify the scale and bias values without any success: the displays is still not correct.

I thought my displacement texture was not correctly loaded but it's not the case (for information I use the NVIDIA NSight degugger).

If I load my displacement map the following manner (GL_LUMINANCE):

glTexImage2D(this->m_Target, 0, GL_LUMINANCE,
    this->m_PixelData.GetWidth(), this->m_PixelData.GetHeight(),
            0, GL_BGR, GL_UNSIGNED_BYTE, OFFSET_BUFFER(0));

The pixel buffer begins by:

And if I load my displacement map the following manner (GL_RGB):

glTexImage2D(this->m_Target, 0, GL_RGB, this->m_PixelData.GetWidth(), this->m_PixelData.GetHeight(), 0, GL_BGR, GL_UNSIGNED_BYTE, OFFSET_BUFFER(0));

The pixel buffer begins by:

In these two cases we have grayscale pixels.

So my problem does not seem to comes from the texture loaded in memory. Maybe there is a problem with the matrices or a problem of space. I'm really lost.

Does anyone can help me, please ?

Thanks so much in adavance for your help!

Answer 1

The single textOffset method is an early parallax technique which makes a very large approximation: that the depth at the offset will be the same. That's why the effect might just look a bit strange. I think in your case one of the tangent vectors is facing the wrong way. As a test, try negating textOffset.x or textOffset.y. I'm more used to seeing shaders use eye-space before the jump to tangent space but can't see any immediate issues with your code.

For a better effect, trace (or "step") through the heightmap until you find an intersection. Use a binary or secant search to improve it. Then use the coordinate of your intersection for colour. This sometimes called relief mapping, steep parallax mapping and parallax occlusion mapping.

This is untested, but hopefully gives the idea:

const int steps = 20;
const float scale = 0.1;

vec3 pos = vec3(TexCoords, 1.0); //1.0 as ray starts at surface/max height
vec3 dir = -viewDir_TS / viewDir_TS.z; //I assume viewDir_TS.z is negative
dir.xy *= scale;
dir /= steps;

//linear steps
float height;
for (int i = 0; i < steps; ++i)
{
    pos += dir;
    height = texture2D(HeightSampler, pos.xy).r
    if (pos.z < height)
        break;
}

//binary search
for (int i = 0; i < 4; ++i)
{
    dir *= 0.5;
    if (pos.z < height)
        pos -= dir;
    else
        pos += dir;
    height = texture2D(HeightSampler, pos.xy).r
}

//write output
FragColor = texture2D(DiffuseSampler, pos.xy);

A couple of smaller things: I see the normal and tangent vectors multiplied by the normal matrix all the time. The tangents should really be multiplied by the modelview matrix, though this won't matter if the transform is orthonormal which is pretty much always is. Computing and storing bitangents is not necessary as it can be computed dynamically with a cross product, avoiding a bit of memory bandwidth when drawing.

Answer 2

The problem just came from the line:

float HSB = Height * ScaleBias.x + ScaleBias.y;

It's not an addition but a substraction:

float HSB = Height * ScaleBias.x - ScaleBias.y;

Screenshot 1:

Screenshot 2:

Of course I've added normal mapping for luminosity.

I hope this post will be usefull!