3DMM Face Sample

Part two of a two part project to help me understand Microsoft's Deep-3D-Face-Reconstruction paper. Click here for part one.

Warning: This project uses DirectX 9, which is outdated now and may cause difficulties when building on modern Windows installations. Also, what I am attempting to do in this project is outside my area of expertise, so any conclusions I made should be taken with caution.

Note: The purpose of this project was to help me understand what 3D Morphable Models (3DMM) are and how they are used with face reconstruction DNNs. While it can be used to create a 3DMM of a human face, it is unlikely that the approach used will result in a production quality 3DMM that would be suitable for research or commercial use. Consider what follows to be of educational value only.

Introduction

Deep Neural Networks (DNNs) that perform Face Reconstruction (e.g. Microsoft's Deep-3D-Face-Reconstruction), or that take a 2D image of a human face and output a 3D mesh, often include a 3D Morphable Face Model (3DMM) as part of their pipeline. The purpose of this project was to help gain a practical understanding of 3DMMs when used in this context. Specifically, when I downloaded and started to use Microsoft's Deep-3D-Face-Reconstruction the following questions came to mind:

1. What exactly is a 3DMM?
2. How does the DNN output the mesh? If the face mesh has 70,000+ vertices, does this mean the last layer of the DNN has this many outputs? (no, as it turns out).
3. 3DMMs are stored in files. What exactly are in these files?

What follows is an attempt to answer these questions by creating a DirectX/C++ application that demonstrates how to construct and render a 3D Morphable Model (3DMM) of a human Face. An option to export the model is provided along with sample code for loading the model into Matlab.

Theory

The first step in creating a 3DMM of a human face is to obtain a large number of 3D scans of faces. In the case of the 2009 Basel model, 100 male faces and 100 females faces were used. These scans are then converted into 3D meshes. Once acquired, each mesh is represented as a series of shape vectors:

Each shape is represented as a column vector with n vertices:

where the mean/average mesh is defined as:

If we subtract the average from each mesh

then new models can be expressed as the mean plus a linear combination of the m shape delta vectors:

The dimensionality of the system is m but we can use Principle Component Analysis to determine a new set of basis vectors for the system such that the most dominant components are included first and the rest can be ignored. i.e.

where k is the new dimension (hopefully significantly smaller than m) and E represents the Eigenvectors returned from the Singular Value Decomposition (SVD) method that we use to perform the Principle Component Analysis.

In the demo application, the Meshes folder includes 204 face meshes and so we have m = 204. After performing SVD on these meshes, only the first 20 are included and the rest are discarded, so we have k=20. In the image below, the 20 sliders are used to control the scalar multipliers (presented as alpha in the previous equation):

The image below shows theses scalars set to zero, so the image displayed is the average/mean face.

The second image shows the change in the mesh when the first slider (first principle component) is non-zero.

The following conclusions can be drawn from the above analysis:

  1) A DNN that produces a face mesh as its output just needs to return a vector of k scalars that are then applied to the 3DMM to produce the actual mesh.

  

  2) A 3DMM of a human face consists of the following:

• k eigenvectors, each of length n where n is the number of vertices of the face mesh.

• the average/mean face mesh (also of dimension n).

• indices that will apply to vertices to form the actual triangles.

Building the software

1. The software was built using Visual Studio 2019 community edition. Be sure to install the MFC module.
2. It requires the DirectX 9 June 2010 DirectX SDK
3. The software uses the Eigen library to perform the SVD. To install Eigen clone it from github:

    git clone https://gitlab.com/libeigen/eigen.git

and add an Environment variable named EIGEN_DIR containing the pathname to which Eigen was cloned.

Running the software

When starting the software, the following steps are performed:

1. loads the sample meshes from the 'meshes' folder
2. calculates the average mesh by adding all vertices and dividing by the total number of meshes
3. subtracts the average mesh from each of the meshes
4. calculates the eigenvectors using the JacobiSVD SVD function in the Eigen libraries
5. creates and renders a mesh on-the-fly created using the following equation where the scalar parameters are dynamically updated from the slider controls.

        

Additional Notes

1. The software won't display the main dialog box until the above steps are complete, so it may appear not to start up right away.
2. Normally the sample meshes (location in the 'meshes' folder) used to create the model are scanned from real faces. In this case it wasn't practical to do this, and the meshes were createed using Deep-3D-Face-Reconstruction. The images from which the meshes were created were aligned (with respect to eyes, nose...) to produce better results. See here why alignment of the meshes is important.
3. Production quality 3D Morphable Models of human faces typically separate shape from facial expressions and have a set of eigenvectors for both. Some also include a third set for texture of the face.
4. Finally, I have not attempted to optimize performance (via threading, vectorization, cache coherency etc).

Further Reading

1. For more information on the DirectX rendering method used in this application see "Introduction To 3D Game Programming With Directx 9.0C: A Shader Approach" by Frank Luna.
2. Steve Brunton's youtube channel includes several videos on how to use singular value decomposition (SVD) to efficiently represent human faces in 2D.