Tessellation Optimizations [OpenGL]

Question

Tessellation of the terrain: I'm rendering my terrain with clip-maps frustum culling and tessellation shaders. Now to improve performance more, I want to use some tessellation shaders in a more efficient way. My aim is to achieve a pretty high (~100 FPS) frame rate with decent visual quality, so I'm gonna combine 3 tessellation factors, a tessellation map, view dependent (pixels per edge) tessellation, and silhouette tessellation.

My idea is a weighted average of them:

tessfac = tesscoeff*(w1*tessmapfactor+w2*silhtessfactor+w3*ppetessfactor)/(w1+w2+w3);

with the 3 factors varying between 0...1, and the tesscoeff could be an uniform too to adjust the levels. I have the map and the edge radius factors but...

Silhouettes: How to find the silhouette patches on a mesh described with a simple y(x,z) function? I'm using vertex-texture-fetch so I know the neighboring vertices/normals. Is it a problem if I'm using quad patches, not triangles, or it doesn't make it harder to find them?

(EDITED TO BACK-PATCH) Backpatch Culling: Another problem I've encountered is the back-face culling. As I've said, I'm using quad patches, so one may not lie on a plane. With triangles I would compare the normal to the eye direction, but on the quads I can't, or do not know how to do it.

Occlusion Culling: I'm rendering the terrain with geometry clip-maps, with the algorithm:

foreach(LOD)
    foreach(quarter)
        foreach(quad in quarter) //I mean x+z+, x-z+, x-z- and x+z- by quarters
            quad.clipFrustum();
            if(inFrustum) quad.render();
        endloop
    endloop
endloop

I have one single vertex buffer for a simple n*n quad shaped mess. I have 4 index buffers for this mesh, one for each quadrant of this quad. For LOD=0 I render the center/bottom left quad, and for the other levels I only render the L-shaped mess with the other 3 (top,topleft,left) quadrants, scaled by pow(2,LOD). Then the whole mesh is scaled with a constant (gridscale).

Since this whole mesh is positioned every frame in front of the camera, the mesh is always rendered quadrant by quadrant from the view origin. Maybe it could be used for some culling like early-z or occlusion in the tessellation control shader?

I'm looking for some pointer on these because all I've found are some NVIDIA/AMD papers without much technical background. http://www.geeks3d.com/20101201/amd-graphics-blog-tessellation-for-all/ the link to the full article is broken too... (from this post: OpenGL Low-Level Performance Questions)

Answer 1

You can cull a triangle (3 points) using the following,

float3 vEdge0 = patch[1].PosW - patch[0].PosW;
float3 vEdge1 = patch[2].PosW - patch[0].PosW;

float3 vFaceNormal0 = normalize(cross(vEdge1, vEdge0));
float2 vView0 = normalize(patch[0].PosW - gEyePosW);

if (dot(vView0, vFaceNormal0) < -0.25f)
{
    pt.EdgeTess[0] = 0.0f;
    pt.EdgeTess[1] = 0.0f;
    pt.EdgeTess[2] = 0.0f;

    pt.InsideTess = 0.0f;

    return pt;
}

You should be able to cull a patch by forming 4 triangles like [\] (2 triangles from top left point to bottom right point) and [/] (2 triangles from top right to bottom left) from the 4 corner points of the patch and if all dot(vView, vFaceNormal) < -0.25f, it is pretty safe to say that the patch is facing away from the user's eye. But this is an approximation of averages (your 4 triangles from the corner points). Depending on your patch size, and how much crazy stuff (mountains and valleys) you have going on in the patch, this may not work in every situation, and that may be the reason there isn't much info on the net about it. You will just have to test and see.

I will say that you could even go a step further, if for some reason you see this not working in all your situations and absolutely want to cull in the hull shader that you could pass in the HeightMap as a texture and sample even more triangles (like get the mid points of the patch) to make [x] to get 4 more triangles even [*] to get 8 more triangles.

A safer way to accomplish the same thing without a huge amount of work is to frustum cull, pass in the planes of your bounding camera frustum and check the bounding box of the patch. Here is the shader code;

float3 vMin = float3(patch[0].PosW.x, minY, patch[0].PosW.z);
float3 vMax = float3(patch[3].PosW.x, maxY, patch[3].PosW.z);

if (!IntersectsOrContainsBoundingBox(vMin.x, vMin.y, vMin.z, vMax.x, vMax.y, vMax.z))
{
    pt.EdgeTess[0] = 0.0f;
    pt.EdgeTess[1] = 0.0f;
    pt.EdgeTess[2] = 0.0f;
    pt.EdgeTess[3] = 0.0f;

    pt.InsideTess[0] = 0.0f;
    pt.InsideTess[1] = 0.0f;

    return pt;
}

bool IntersectsOrContainsBoundingBox(float minWidth, float minHeight, float minDepth, float maxWidth, float maxHeight, float maxDepth)
{
// check box outside/inside of frustum
for (int i = 0; i < 6; i++)
{
    int output = 0;
    output += (((gWorldFrustumPlanes[i].x * minWidth) + (gWorldFrustumPlanes[i].y * minHeight) + (gWorldFrustumPlanes[i].z * minDepth) + (gWorldFrustumPlanes[i].w * 1.0f) < 0.0) ? 1 : 0);
    output += (((gWorldFrustumPlanes[i].x * maxWidth) + (gWorldFrustumPlanes[i].y * minHeight) + (gWorldFrustumPlanes[i].z * minDepth) + (gWorldFrustumPlanes[i].w * 1.0f) < 0.0) ? 1 : 0);
    output += (((gWorldFrustumPlanes[i].x * minWidth) + (gWorldFrustumPlanes[i].y * maxHeight) + (gWorldFrustumPlanes[i].z * minDepth) + (gWorldFrustumPlanes[i].w * 1.0f) < 0.0) ? 1 : 0);
    output += (((gWorldFrustumPlanes[i].x * maxWidth) + (gWorldFrustumPlanes[i].y * maxHeight) + (gWorldFrustumPlanes[i].z * minDepth) + (gWorldFrustumPlanes[i].w * 1.0f) < 0.0) ? 1 : 0);
    output += (((gWorldFrustumPlanes[i].x * minWidth) + (gWorldFrustumPlanes[i].y * minHeight) + (gWorldFrustumPlanes[i].z * maxDepth) + (gWorldFrustumPlanes[i].w * 1.0f) < 0.0) ? 1 : 0);
    output += (((gWorldFrustumPlanes[i].x * maxWidth) + (gWorldFrustumPlanes[i].y * minHeight) + (gWorldFrustumPlanes[i].z * maxDepth) + (gWorldFrustumPlanes[i].w * 1.0f) < 0.0) ? 1 : 0);
    output += (((gWorldFrustumPlanes[i].x * minWidth) + (gWorldFrustumPlanes[i].y * maxHeight) + (gWorldFrustumPlanes[i].z * maxDepth) + (gWorldFrustumPlanes[i].w * 1.0f) < 0.0) ? 1 : 0);
    output += (((gWorldFrustumPlanes[i].x * maxWidth) + (gWorldFrustumPlanes[i].y * maxHeight) + (gWorldFrustumPlanes[i].z * maxDepth) + (gWorldFrustumPlanes[i].w * 1.0f) < 0.0) ? 1 : 0);
    if (output == 8)
    {
        return false;
    }
}

return true;
}