Merge pull request #5 from cvjena/develop

Add support for Blendshapes

Author: Timozen

README.md

@@ -12,6 +12,7 @@ A small demo is hosted [here](https://semg.inf-cv.uni-jena.de/), together with t
 - **Visualize muscle activity**: The EMG Intensity plot allows you to visualize the intensity of muscle activity over the face, providing insights into patterns and variations.
 - **Designed explicitly for facial muscles**: The tool focuses on facial muscles, enabling you to study and understand muscle activity in the face, which can be particularly useful in fields like facial expression analysis, neuroscience, and rehabilitation.
 - **FACS**: Visualize the Facial Action Coding System at the correct anatomical locations for a more intuitive understanding of the data.
+- **Blend Shapes**: Many existing blend shape models base their movements on the FACS. We provide a location mapping for commonly used blend shapes models and where they are located on the face to the AU muscle.
 - **Potential for extension**: While the current focus is on facial muscles, this tool could potentially be extended to analyze other muscle groups.
 - **Beyond muscles**: The tool can also be used to plot additional facial information, such as oxygen saturation, but this is not officially supported yet.
 
@@ -64,9 +65,11 @@ For the colorization, the users can use any color map from [matplotlib](https://
 We currently support the two following schematics for acquiring the EMG data.
 If you want to have your own, please open an issue or create a pull request, and we will be happy to add it.
 
-| [Fridlund and Cappacio, 1986](https://pubmed.ncbi.nlm.nih.gov/3809364/) | [Kuramoto et al., 2019](https://onlinelibrary.wiley.com/doi/10.1002/npr2.12059) | [Ekman and Friesen - FACS](https://psycnet.apa.org/record/1971-07999-001)|
-|---|---|---|
-| ![Locations ](files/locations_fridlund.jpg)  |  ![Locations ](files/locations_kuramoto.jpg) | ![Locations ](files/locations_facs.jpg) |
+| [Fridlund and Cappacio, 1986](https://pubmed.ncbi.nlm.nih.gov/3809364/) | [Kuramoto et al., 2019](https://onlinelibrary.wiley.com/doi/10.1002/npr2.12059)|
+| :---: | :---: |
+| ![Locations ](files/locations_fridlund.jpg)  |  ![Locations ](files/locations_kuramoto.jpg) |
+| [Ekman and Friesen - FACS](https://psycnet.apa.org/record/1971-07999-001)| [Mediapipe Blendshapes](https://storage.googleapis.com/mediapipe-assets/Model%20Card%20Blendshape%20V2.pdf)|
+| ![Locations ](files/locations_facs.jpg) | ![Locations ](files/locations_blendshapes.jpg) |
 
 If you want to define your custom scheme, create a new class inherited from `emg.Schematic` and implement the `locations` member. If you support the mirroring of the face, implement the `pairs_L` and `pairs_R` members.
 Then, use it in the `interpolate` function, and you are good to go.

examples/schemes.ipynb

@@ -7,7 +7,8 @@
     "Scheme",
     "colorize",
     "get_colormap",
-    "postprocess"
+    "postprocess",
+    "Blendshapes"
 ]
 
 from electromyogram.plot import (
@@ -18,4 +19,4 @@
     postprocess,
 )
 
-from electromyogram.schemes import FACS, Fridlund, Kuramoto, Scheme
+from electromyogram.schemes import FACS, Fridlund, Kuramoto, Scheme, Blendshapes

files/locations_blendshapes.jpg

@@ -1,4 +1,4 @@
-__all__ = ["Scheme", "Kuramoto", "Fridlund", "FACS"]
+__all__ = ["Scheme", "Kuramoto", "Fridlund", "FACS", "Blendshapes"]
 
 import abc
 from typing import Optional
@@ -229,4 +229,93 @@ class FACS(Scheme):
 
         "AU26_R": (-70.0, -50.0),        
         "AU26_L": ( 70.0, -50.0),
+    }
+
+class Blendshapes(Scheme):
+    # X  1 - browDownLeft
+    # X  2 - browDownRight
+    # X  3 - browInnerUp
+    # X  4 - browOuterUpLeft
+    # X  5 - browOuterUpRight
+    # X  6 - cheekPuff
+    # X  7 - cheekSquintLeft
+    # X  8 - cheekSquintRight
+    # X  9 - eyeBlinkLeft
+    # X 10 - eyeBlinkRight
+    # X 11 - eyeLookDownLeft
+    # X 12 - eyeLookDownRight
+    # - 13 - eyeLookInLeft
+    # - 14 - eyeLookInRight
+    # - 15 - eyeLookOutLeft
+    # - 16 - eyeLookOutRight
+    # X 17 - eyeLookUpRight
+    # X 18 - eyeLookUpRight
+    # X 19 - eyeSquintLeft
+    # X 20 - eyeSquintRight
+    # X 21 - eyeWideLeft
+    # X 22 - eyeWideRight
+    # X 23 - jawForward
+    # - 24 - jawLeft
+    # X 25 - jawOpen
+    # - 26 - jawRight
+    # X 27 - mouthClose
+    # X 28 - mouthDimpleLeft
+    # X 29 - mouthDimpleRight
+    # X 30 - mouthFrownLeft
+    # X 31 - mouthFrownRight
+    # X 32 - mouthFunnel
+    # X 33 - mouthLeft
+    # - 34 - mouthLowerDownLeft
+    # - 35 - mouthLowerDownRight
+    # X 36 - mouthPressLeft
+    # X 37 - mouthPressRight
+    # X 38 - mouthPucker
+    # X 39 - mouthRight
+    # X 40 - mouthRollLower
+    # X 41 - mouthRollUpper
+    # X 42 - mouthShrugLower
+    # X 43 - mouthShrugUpper
+    # X 44 - mouthSmileLeft
+    # X 45 - mouthSmileRight
+    # X 46 - mouthStretchLeft
+    # X 47 - mouthStretchRight
+    # - 48 - mouthUpperUpLeft
+    # - 49 - mouthUpperUpRight
+    # X 50 - noseSneerLeft
+    # X 51 - noseSneerRight
+    # X 52 - tongueOut
+    locations = {
+        "browInnerUp": (0.0, 65.0),
+        
+        "browOuterUpRight": (-32.0, 60.0),
+        "browOuterUpLeft":  ( 32.0, 60.0),
+        
+        "browDownRight": ( -9.0, 49.0),
+        "browDownLeft":  (  9.0, 49.0),
+
+        "cheekSquintRight": (-42.0, 10.0),
+        "cheekSquintLeft":  ( 42.0, 10.0),
+        
+        "noseSneerRight": (-16.0, -0.0),
+        "noseSneerLeft":  ( 16.0, -0.0),
+        
+        "mouthShrugUpper": (0.0, -25.0),
+        
+        "mouthRight": (-27.0, -38.0),
+        "mouthLeft":  ( 27.0, -38.0),
+        
+        "mouthSmileRight": (-53.0, -8.0),
+        "mouthSmileLeft":  ( 53.0, -8.0),
+        
+        "mouthDimpleRight": (-48.0, -24.0),
+        "mouthDimpleLeft":  ( 48.0, -24.0),
+        
+        "mouthFrownRight": (-26.0, -69.0),
+        "mouthFrownLeft":  ( 26.0, -69.0),
+        
+        "jawForward": (-0.0, -65.0),
+
+        "mouthStretchRight": (-53.0, -42.0),
+        "mouthStretchLeft":  ( 53.0, -42.0),
+
     }