Merge pull request #1 from cvjena/develop

Develop - Merge newest changes
cvjena · Nov 3, 2023 · b0758db · b0758db
2 parents ccaa4a1 + fb525a6
commit b0758db
Show file tree

Hide file tree

Showing 24 changed files with 389 additions and 820 deletions.
diff --git a/.gitignore b/.gitignore
@@ -51,6 +51,4 @@ MANIFEST
 # Per-project virtualenvs
 .venv*/
 .conda*/
-*.ipynb
 *.ipynb_checkpoints
-temp_data
diff --git a/README.md b/README.md
@@ -1,64 +1,61 @@
 # face-projection
 
-This projects main goal is to have a simple library to project information into faces while retaining the facial structure.
+![Teaser](files/teaser.jpg)
 
+Are you looking for a powerful, user-friendly tool to project information onto faces while maintaining their natural structure?
+Look no further – `face-projection` is here to simplify how you visualize facial data.
+The tool is open-source, easy to use, and written in Python, allowing you to easily integrate it into your existing workflow.
+We try to keep the dependencies to a minimum so you can use them in your projects without worrying about compatibility issues.
+However, we do not guarantee perfect anatomical correctness of the results but try to keep the distortion to a minimum.
 
-## Description
-
-Using the Canonical Face Model provided by Google, we can draw any UV map inside the given boundaries to the face.
-This allows us to visualize certain properties of facial features.
-The user can provide `face_data` which then projected onto the `face_img`.
-We support automatic face detection with landmark extraction using `mediapipe`.
-However, the user can also provide the landmarks manually if they are already known and fit the Canonical Face Model.
+All people shown here in the repository examples were generated with `StableDiffusion` (Prompt: `A professional portrait photo of a person directly looking <emotion> at the camera, white background, photoshop, Instagram`)
 
 ## Installation
 
-### MacOS
-
-To use meshpy a C++ compiler is required.
-On MacOS you can use xcodebuild which is installed by default.
-To enable it please run the following command and agree to the license agreement.
-Then it should be able to compile the C++ code in your python environment.
+The tool is available on [PyPI](https://pypi.org/project/face-projection/).
 
 ```bash
-sudo xcodebuild -license
+pip install face-projection
 ```
 
 ## Usage
 
+The tool reduces the overhead of projecting information onto faces to a minimum.
+Load the data and face, project it, and you are done.
+You must only ensure the data is inside the canonical face model (see [electromyogram](https://github.com/cvjena/electromyogram) for an example).
+The head pose can be arbitrary, but the face must be visible.
+
 ```python
 import face_projection as fp
+from PIL import Image
+
+image_face = np.asarray(Image.open("face.jpg").convert("RGB"))
+image_data = np.asarray(Image.open("data.jpg").convert("RGB"))
 
 warper = fp.Warper()
+warped = warper.apply(image_face, image_data, beta=0.2)
+```
 
-face_img = load_image("face.jpg") # load image of face with your favorite library
-face_data = warper.create_canvas() # creates a empty image with the current size of the face model
+We automatically detect the face in the image, compute the landmarks based on the `Blaze` model, and warp the data onto it.
+You can decide how much of the underlying face should be visible by adjusting the `beta` parameter.
 
-# draw something on the face_data
-# your implementation here
+In `examples/` you can find a more detailed example, which generates the teaser image.
 
-# project the face_data onto the face_img, using the automatic face detection
-# beta is the blending factor between 0 and 1
-warped_face = warper.apply(face_img, face_data, beta=0.2)
+## Future Work
 
-# if you already know the landmarks, you can provide them manually
-warped_face = warper.apply(face_img, face_data, landmarks=landmarks, beta=0.2)
-```
+We have many ideas for future work, but we are also happy to hear your suggestions.
+The following todos are in no particular order but should give you an idea of what we are planning.
 
-## Future Work
-- [ ] The user can provide a mask to disable certain parts of the projection
-- [ ] Sample generators for different face properties
+- [ ] The user can provide a mask to turn off certain parts of the projection
+- [X] Different default warpings based on the six default Ekman emotions
 - [ ] More face models
 - [ ] Custom face models
-- [ ] Canvas fitter to fit to the general face model
-
-
-## References
+- [ ] Upgrade to the newest mediapipe version to avoid MacOS build issues
 
 ## Citation
 
 If you use our work, please cite our paper:
 
 ```bibtex
-TODO :^)
+under publication, so coming soon :^)
 ```
diff --git a/example/data/powermap_angry.png b/example/data/powermap_angry.png
diff --git a/example/data/powermap_disgusted.png b/example/data/powermap_disgusted.png
diff --git a/example/data/powermap_fearful.png b/example/data/powermap_fearful.png
diff --git a/example/data/powermap_happy.png b/example/data/powermap_happy.png
diff --git a/example/data/powermap_neutral.png b/example/data/powermap_neutral.png
diff --git a/example/data/powermap_sad.png b/example/data/powermap_sad.png
diff --git a/example/data/powermap_surprised.png b/example/data/powermap_surprised.png
diff --git a/example/data/stable_diffusion_angry.png b/example/data/stable_diffusion_angry.png
diff --git a/example/data/stable_diffusion_disgusted.png b/example/data/stable_diffusion_disgusted.png
diff --git a/example/data/stable_diffusion_fearful.png b/example/data/stable_diffusion_fearful.png
diff --git a/example/data/stable_diffusion_happy.png b/example/data/stable_diffusion_happy.png
diff --git a/example/data/stable_diffusion_neutral.png b/example/data/stable_diffusion_neutral.png
diff --git a/example/data/stable_diffusion_sad.png b/example/data/stable_diffusion_sad.png
diff --git a/example/data/stable_diffusion_surprised.png b/example/data/stable_diffusion_surprised.png
diff --git a/example/emotion.ipynb b/example/emotion.ipynb
diff --git a/files/teaser.jpg b/files/teaser.jpg
diff --git a/setup.cfg b/setup.cfg
@@ -48,6 +48,7 @@ install_requires =
     opencv-python-headless
     mediapipe==0.9.3.0; platform_system != "Darwin" or platform.machine != "arm64"
     mediapipe-silicon==0.9.2.1; platform_system == "Darwin" and platform.machine == "arm64"
+    h5py>=3.10
 
 [options.packages.find]
 where = src