Computing the surface normal at hit point in OptiX 7

Question

I'm studying the OptixTriangle example of OptiX 7.3. However, the meaning of the parameters of optixTrace are not clear:

optixTrace(
        params.handle,
        ray_origin,
        ray_direction,
        0.0f,                       // --- Min intersection distance (tmin)
        1e16f,                      // --- Max intersection distance (tmax)
        0.0f,                       // --- rayTime -- used for motion blur
        OptixVisibilityMask( 255 ), // --- Specify always visible
        OPTIX_RAY_FLAG_NONE,        // --- OPTIX_RAY_FLAG_DISABLE_ANYHIT
        0,                          // --- SBT offset   -- See SBT discussion
        1,                          // --- SBT stride   -- See SBT discussion
        0,                          // --- missSBTIndex -- See SBT discussion
        p0, p1, p2 );
float3 result;
result.x = int_as_float( p0 );
result.y = int_as_float( p1 );
result.z = int_as_float( p2 );

// Record results in our output raster
params.image[idx.y * params.image_width + idx.x] = make_color( result );

In particular:

1. What do the parameters p0, p1 and p2 stand for?
2. How is it possible to compute the surface normal at the hit point so that I could colour the triangle with that information?

Answer 1

1. In OptixTriangle the variables p0, p1 and p2 carry the color calculated by the closest hit program back to the ray generation program. OptiX calls the concept ray payload. It's a mechanism to attach up to 8 values to a ray and pass them along the program pipeline where each program being called can read and write the payload values. More on this is in the OptiX Programming Guide.

2. In case of a triangle primitive being hit (as in OptixTriangle) you have to obtain the triangle coordinates from the acceleration structure and apply some vector algebra on them to calculate the normal of one of the vertices. The normal at the hit point is the same as for any triangle vertex: they all share the same plane. To get hit point coordinates anyway the OptiX API provides the barycentric coordinates of the intersection via primitive attributes to the hit programs.

To translate this into code you won't get around a deeper understanding of OptiX 7. A good point to start is How to Get Started with OptiX 7 as it actually walks you step by step through the OptixTriangle example. To follow-up visit the GitHub repo Siggraph 2019/2020 OptiX 7/7.3 Course Tutorial Code. Walking through the examples makes a steep learning curve. The 5th example therein shows normal calculation.