Postprocessing Perspective Transformation in Three.js

Question

i want to create a scene in three.js, where the camera is moving. In every frame there are 4 points visible which form a quadriliteral. Subsequently, I want to process each frame by warping these 4 points to the canvas edges, so that they form a rectangle with the same size as the canvas. Ideally I would do this with the post-processing solution of three.js by creating a shader and passing it to the ShaderPass() class, but I am thankful for any other solution.

I have read about perspective transforms and came accross this post: Transform quadrilateral into a rectangle? . I understand how to create the transformation matrix but I am a bit confused on how to apply it in post-processing to a 2d image frame because I only read about applying transformation matrices to 3d objects.

I attached an image showing the canvas output before the post-processing step. I want to warp the 4 corners of the red quadriliteral to the canvas edges. I know their coordinates in 3d world space and 2d screen space by using the .project() method of three.js.

I tried the post-processing in a project with p5.js and opencv.js by using opencv's warpPerspective() function and it kind of worked, but I want to switch to three.js for different reasons.

Any help or pointers to relevant topics are highly appreciated!

Thanks, Luca

Answer 1

After several hours of confusion, I finally solved the problem and created a shader for three.js that does perspective warping. I used the 'dltjs' library (https://github.com/charlee/dltjs) to calculate the homography matrix. In the warpPoint() function in the fragment shader, the matrix is multiplied by a coordinate and then dehomogenized to compute the transformed coordinate.

Here is the WarpShader code. p0 contains the rectangle points, p1 the quadrilateral points. M is the homography matrix, which is then passed to the shader as a uniform.

import * as THREE from 'three';
import dlt from 'dltjs';

let p0 = [[0, 0], [0, 1], [1, 0], [1, 1]];
let p1 = [[0.3, 0.3], [0.1, 0.9], [0.9, 0.1], [0.9, 0.9]];
let M = dlt.dlt2d(p0, p1);

const WarpShader = {

    uniforms: {
        tDiffuse: null,
        transformMat: {
            value: new THREE.Matrix3(
                M[0][0], M[0][1], M[0][2],
                M[1][0], M[1][1], M[1][2],
                M[2][0], M[2][1], M[2][2]
            )
        }
    },

    vertexShader: /* glsl */`

        varying vec2 vUv;

        void main() {
            gl_Position = projectionMatrix * modelViewMatrix * vec4( position, 1.0 );
            vUv = uv;
        }
        `,

    fragmentShader: /* glsl */`

        uniform sampler2D tDiffuse;
        varying vec2 vUv;
        uniform mat3 transformMat;

        vec2 warpPoint(mat3 transformMat, vec2 p) {
            vec3 result = transformMat * vec3(p, 1.0); // transformation
            return vec2(result.x / result.z, result.y / result.z); // dehomogenization
        }

        void main() {
            vec2 coords = warpPoint(transformMat, vUv);
            gl_FragColor = texture2D(tDiffuse, coords);
        }`

};

export { WarpShader };