Calculate normal vector of a polygon - Newells Method

Question

I'm trying to calculate the surface normal of a 2D polygon. I am using Newell's method from the OpenGL wiki to calculate the surface normal. https://www.opengl.org/wiki/Calculating_a_Surface_Normal From my understanding the normal should be in the y direction but it always returns [0, 0, 0]. The y value gets changed to -1 on the second iteration and back to zero on the fourth iteration.

p = [[0, 0, 0]
    [1, 0, 0]
    [0, 0, 1]
    [1, 0, 1]]

function calcNormal(p) {
    var normal = [0, 0, 0];
    for(var i = 0; i < p.length; i++) {
        var j = (i + 1) % (p.length);       
        normal[0] += (p[i][1] - p[j][1]) * (p[i][2] + p[j][2]);
        normal[1] += (p[i][2] - p[j][2]) * (p[i][0] + p[j][0]);
        normal[2] += (p[i][0] - p[j][0]) * (p[i][1] + p[j][1]);
    }
    return normal;
}

Answer 1

You're using a degenerate polygon for testing. If you draw it in the xz-plane, with the vertices numbered from 0 to 3, it looks like this:

2 ---- 3
 \    /
  \  /
   \/
   /\
  /  \
 /    \
0 ---- 1

This polygon does not have a well defined normal, since it changes orientation in the middle, and folds over itself.

If you swap the last two vertices:

p = [[0, 0, 0]
     [1, 0, 0]
     [1, 0, 1]
     [0, 0, 1]]

It will look like this, and you should get much more meaningful results:

3 ---- 2
|      |
|      |
|      |
0 ---- 1

Answer 2

OpenGL's version is failing for some cases especially when Polygon is 2D and you providing more than 3 vertices for calculation (4 in your case). If you provide only 3 vertices it will calculate correctly (also consider to use vector product to get normal). Here is the link to Game Development Stack Exchange to the similar question with different approaches to this problem.