Simulating older 3D games per-polygon clipping in Unity?

Question

I'm trying to emulate older 3D games that did per-polygon clipping.

Using degenerate triangles, it works except when looking away with the camera.

Here's an animated GIF, a sphere with that material applied to it:

The green faces are what I'm expecting, triangle is completely clipped.

The red faces are obviously wrong, they're all glued to screen center.

Here's the shader in question:

Shader"PerPolygonClipping"
{
    Properties
    {
        _MainTex ("Texture", 2D) = "white" {}
        _Distance ("Distance", Float) = 1.0
    }
    SubShader
    {
        Tags
        {
            "RenderType"="Opaque"
        }
        LOD 100

        Pass
        {
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"

            struct appdata
            {
                float4 vertex : POSITION;
                float2 uv : TEXCOORD0;
                fixed4 color : COLOR;
            };

            struct v2f
            {
                float2 uv : TEXCOORD0;
                float4 vertex : SV_POSITION;
                fixed4 color : COLOR;
            };

            sampler2D _MainTex;
            float4 _MainTex_ST;
            float _Distance;

            v2f vert(appdata v)
            {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
                o.uv = TRANSFORM_TEX(v.uv, _MainTex);

                const float3 world_position = mul(unity_ObjectToWorld, v.vertex).xyz;
                const float distance = length(world_position - _WorldSpaceCameraPos);
                float difference = _Distance - distance;
                
                if (difference < 0)
                {
                    o.color = fixed4(1, 0, 0, 1);
                }
                else if (difference > 0)
                {
                    o.color = fixed4(0, 1, 0, 1);
                }
                else
                {
                    o.color = fixed4(0, 0, 1, 1);
                }

                // https://gamedev.net/forums/topic/421211-clip-in-vertex-shader-in-hlsl/3805467/
                o.vertex *= step(0, difference);
                return o;
            }

            fixed4 frag(v2f i) : SV_Target
            {
                fixed4 col = tex2D(_MainTex, i.uv);
                col *= i.color;
                return col;
            }
            ENDCG
        }
    }
}

Suggestions are welcome on how to fix this problem.

Answer 1

The issue is that you're attempting to clip triangles at a per-vertex level. Each vertex is being checked independently, and so it's possible that only one of the three vertices that make up a triangle is moved to the center of the screen, resulting in the stretching you're seeing with the red triangles. In order to solve this problem, you'll need to clip entire triangles rather than just the vertices.

The easiest way to solve this problem in a shader is by performing the clipping in a geometry shader, since the geometry shader can operate on the entire triangle at once, though it should be noted that geometry shaders are not supported in Metal, and are generally not recommended for performance reasons.

// Culling triangles in the geometry shader
// This is a pass-through geometry shader which will only output triangles
// which have a center point nearer to the camera than _Distance.
[maxvertexcount(3)]
void geom(triangle v2f v[3], inout TriangleStream<v2f> triStream)
{    
    // Find the center of the triangle by averaging together the 3 vertices.
    const float3 center = (v[0].worldPos + v[1].worldPos + v[2].worldPos) / 3.0;

    // Calculate the distance from the triangle to the camera.
    const float distance = length(world_position - _WorldSpaceCameraPos);

    // If the distance is below the cutoff, output all three vertices.
    // Otherwise, no vertices will be output, and the triangle will not be drawn.
    if (distance < _Distance)
    {
        triStream.Append(v[0]);
        triStream.Append(v[1]);
        triStream.Append(v[2]);
    }

    triStream.RestartStrip();
}

---

A slightly more complex solution which does not rely on a geometry shader would be to create another set of texture coordinates for your mesh, where the UVW coordinate of each vertex is the center of its connected triangle. Your vertex shader can then check this triangle center coordinate instead of the vertex position. Since it will be the same for each vertex of the triangle, the whole triangle will be clipped at once.

This is a bit annoying, since it involves generating new meshes with additional data, but it would allow you to achieve the clipping effect you're after using only vertex and fragment shaders.

using System.Collections.Generic;
using UnityEditor;
using UnityEngine;

// This is an editor wizard to create new meshes with additional data.
// Put this in an "Editor" folder in your project.
public class CustomMeshWizard : ScriptableWizard
{
    public Mesh mesh;

    [MenuItem("Assets/Create/Custom Mesh")]
    static void CreateWizard()
    {
        DisplayWizard< CustomMeshWizard >("Create Custom Mesh", "Create");
    }

    void OnWizardCreate()
    {
        // Note that there are much more efficient ways of doing this,
        // but many of them require a lot more code.
        // For simplicity's sake, use the old clunky Mesh api.
        // This example script only works on meshes with one submesh.
        Mesh newMesh = new Mesh();

        // Vertices must be split so they don't share UVs.
        // To do this, make new vertices for each triangle.    
        var newVerts = new List<Vector3>();
        var newNorms = new List<Vector3>();
        var newUVs = new List<Vector2>();
        var newTriCenters = new List<Vector3>();
        var newTriIndices = new List<int>();

        // Iterate over existing triangles, and copy their vertices.
        var oldTriIndices = mesh.triangles;
        var oldVerts = mesh.vertices;
        var oldNorms = mesh.normals;
        var oldUVs = mesh.uv;
        
        for (int i = 0; i < oldTriIndices.Length; i += 3)
        {
             var i0 = oldTriIndices[i + 0];
             var i1 = oldTriIndices[i + 1];
             var i2 = oldTriIndices[i + 2];

             var v0 = oldVerts[i0];
             var v1 = oldVerts[i1];
             var v2 = oldVerts[i2];
             var triCenter = (v0 + v1 + v2) / 3.0f;

             newVerts.Add(v0);
             newNorms.Add(oldNorms[i0]);
             newUVs.Add(oldUVs[i0]);
             newTriCenters.Add(triCenter);

             newVerts.Add(v1);
             newNorms.Add(oldNorms[i1]);
             newUVs.Add(oldUVs[i1]);
             newTriCenters.Add(triCenter);

             newVerts.Add(v2);
             newNorms.Add(oldNorms[i2]);
             newUVs.Add(oldUVs[i2]);
             newTriCenters.Add(triCenter);

             newTriIndices.Add(i + 0);
             newTriIndices.Add(i + 1);
             newTriIndices.Add(i + 2);
        }

        newMesh.SetVertices(newVerts);
        newMesh.SetNormals(newNorms);
        newMesh.SetUVs(0, newUVs);
        newMesh.SetUVs(1, newTriCenters);
        newMesh.SetTriangles(newTriIndices, 0);

        AssetDatabase.CreateAsset(newMesh, "Assets/custom mesh.asset");
        Selection.activeObject = newMesh;
    }
}