I have a set of 3d points that form a 2d polygon. They are ordered in either clockwise or counter clockwise order (one or the other, but I don't know which beforehand).
Now, I want to triangulate it, which I'm trying to do by using a ear cutting approach. The problem is that, in order for the algorithm to work, I need to check if a given angle is concave or not.
In 2d space, and given clockwise order, you can use the cross product and see the direction of the resulting arrow to know if the angle is concave (as you're effectively checking if it makes a left or right turn). However, without knowing whether my points are clockwise or counter-clockwise, and without knowing the normal of the plane, I have no idea on how to approach the problem.
There are some similar questions such as this one, but they assume 2d space, which makes things easier.
You need first to project all points on XY plane, do the triangulation, then connect triangles to original 3D points. To go from 3D to 2D you need a frame to frame transformation. The original frame will be computed from the most suitable three vertices (origin, x axis and y axis) of the original contour.