Numerical unstable normals calculation causes segfault crashes

[henkson]

We encountered a problem where a mesh caused xatlas to crash with a segfault. We were able to strip down the mesh to a very simple example (3 vertices, 2 faces), with this OBJ description:

v 0 695.61237 85.88235
v 0 994.86237 245.88235
v 0 699.2378 87.820755
f 1 2 3
f 2 1 3

The 2 faces share the same vertices but have opposite winding direction. They lie in the YZ plane (x == 0).

We were able to debug the xatlas code and found that the calculation of the normals of the faces goes wrong, because the cross product of two vertices is numerically instable if the angle between them is very small. This is the case for this example, it is a very "long and narrow" triangle.
The calculated normal for both faces is the same (-1, 0, 0), while in fact the first face should have (1, 0, 0).

The code causing the issue is Mesh.computeFaceNormal. It always takes "the first" corner of a face to calculate the normal, while it would be better to use "the corner whose angle is the closest to a right angle".