Skip to content

Commit

Permalink
Add support for Blendshapes
Browse files Browse the repository at this point in the history
- add locations of mediapipes blendshapes to mapped AU and then muscle
  • Loading branch information
Timozen committed Nov 10, 2023
1 parent 9aa41e8 commit 0aca0c7
Show file tree
Hide file tree
Showing 5 changed files with 178 additions and 11 deletions.
9 changes: 6 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ A small demo is hosted [here](https://semg.inf-cv.uni-jena.de/), together with t
- **Visualize muscle activity**: The EMG Intensity plot allows you to visualize the intensity of muscle activity over the face, providing insights into patterns and variations.
- **Designed explicitly for facial muscles**: The tool focuses on facial muscles, enabling you to study and understand muscle activity in the face, which can be particularly useful in fields like facial expression analysis, neuroscience, and rehabilitation.
- **FACS**: Visualize the Facial Action Coding System at the correct anatomical locations for a more intuitive understanding of the data.
- **Blend Shapes**: Many existing blend shape models base their movements on the FACS. We provide a location mapping for commonly used blend shapes models and where they are located on the face to the AU muscle.
- **Potential for extension**: While the current focus is on facial muscles, this tool could potentially be extended to analyze other muscle groups.
- **Beyond muscles**: The tool can also be used to plot additional facial information, such as oxygen saturation, but this is not officially supported yet.

Expand Down Expand Up @@ -64,9 +65,11 @@ For the colorization, the users can use any color map from [matplotlib](https://
We currently support the two following schematics for acquiring the EMG data.
If you want to have your own, please open an issue or create a pull request, and we will be happy to add it.

| [Fridlund and Cappacio, 1986](https://pubmed.ncbi.nlm.nih.gov/3809364/) | [Kuramoto et al., 2019](https://onlinelibrary.wiley.com/doi/10.1002/npr2.12059) | [Ekman and Friesen - FACS](https://psycnet.apa.org/record/1971-07999-001)|
|---|---|---|
| ![Locations ](files/locations_fridlund.jpg) | ![Locations ](files/locations_kuramoto.jpg) | ![Locations ](files/locations_facs.jpg) |
| [Fridlund and Cappacio, 1986](https://pubmed.ncbi.nlm.nih.gov/3809364/) | [Kuramoto et al., 2019](https://onlinelibrary.wiley.com/doi/10.1002/npr2.12059)|
| :---: | :---: |
| ![Locations ](files/locations_fridlund.jpg) | ![Locations ](files/locations_kuramoto.jpg) |
| [Ekman and Friesen - FACS](https://psycnet.apa.org/record/1971-07999-001)| [Mediapipe Blendshapes](https://storage.googleapis.com/mediapipe-assets/Model%20Card%20Blendshape%20V2.pdf)|
| ![Locations ](files/locations_facs.jpg) | ![Locations ](files/locations_blendshapes.jpg) |

If you want to define your custom scheme, create a new class inherited from `emg.Schematic` and implement the `locations` member. If you support the mirroring of the face, implement the `pairs_L` and `pairs_R` members.
Then, use it in the `interpolate` function, and you are good to go.
Expand Down
84 changes: 79 additions & 5 deletions examples/schemes.ipynb

Large diffs are not rendered by default.

Binary file added files/locations_blendshapes.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 3 additions & 2 deletions src/electromyogram/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
"Scheme",
"colorize",
"get_colormap",
"postprocess"
"postprocess",
"Blendshapes"
]

from electromyogram.plot import (
Expand All @@ -18,4 +19,4 @@
postprocess,
)

from electromyogram.schemes import FACS, Fridlund, Kuramoto, Scheme
from electromyogram.schemes import FACS, Fridlund, Kuramoto, Scheme, Blendshapes
91 changes: 90 additions & 1 deletion src/electromyogram/schemes.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
__all__ = ["Scheme", "Kuramoto", "Fridlund", "FACS"]
__all__ = ["Scheme", "Kuramoto", "Fridlund", "FACS", "Blendshapes"]

import abc
from typing import Optional
Expand Down Expand Up @@ -229,4 +229,93 @@ class FACS(Scheme):

"AU26_R": (-70.0, -50.0),
"AU26_L": ( 70.0, -50.0),
}

class Blendshapes(Scheme):
# X 1 - browDownLeft
# X 2 - browDownRight
# X 3 - browInnerUp
# X 4 - browOuterUpLeft
# X 5 - browOuterUpRight
# X 6 - cheekPuff
# X 7 - cheekSquintLeft
# X 8 - cheekSquintRight
# X 9 - eyeBlinkLeft
# X 10 - eyeBlinkRight
# X 11 - eyeLookDownLeft
# X 12 - eyeLookDownRight
# - 13 - eyeLookInLeft
# - 14 - eyeLookInRight
# - 15 - eyeLookOutLeft
# - 16 - eyeLookOutRight
# X 17 - eyeLookUpRight
# X 18 - eyeLookUpRight
# X 19 - eyeSquintLeft
# X 20 - eyeSquintRight
# X 21 - eyeWideLeft
# X 22 - eyeWideRight
# X 23 - jawForward
# - 24 - jawLeft
# X 25 - jawOpen
# - 26 - jawRight
# X 27 - mouthClose
# X 28 - mouthDimpleLeft
# X 29 - mouthDimpleRight
# X 30 - mouthFrownLeft
# X 31 - mouthFrownRight
# X 32 - mouthFunnel
# X 33 - mouthLeft
# - 34 - mouthLowerDownLeft
# - 35 - mouthLowerDownRight
# X 36 - mouthPressLeft
# X 37 - mouthPressRight
# X 38 - mouthPucker
# X 39 - mouthRight
# X 40 - mouthRollLower
# X 41 - mouthRollUpper
# X 42 - mouthShrugLower
# X 43 - mouthShrugUpper
# X 44 - mouthSmileLeft
# X 45 - mouthSmileRight
# X 46 - mouthStretchLeft
# X 47 - mouthStretchRight
# - 48 - mouthUpperUpLeft
# - 49 - mouthUpperUpRight
# X 50 - noseSneerLeft
# X 51 - noseSneerRight
# X 52 - tongueOut
locations = {
"browInnerUp": (0.0, 65.0),

"browOuterUpRight": (-32.0, 60.0),
"browOuterUpLeft": ( 32.0, 60.0),

"browDownRight": ( -9.0, 49.0),
"browDownLeft": ( 9.0, 49.0),

"cheekSquintRight": (-42.0, 10.0),
"cheekSquintLeft": ( 42.0, 10.0),

"noseSneerRight": (-16.0, -0.0),
"noseSneerLeft": ( 16.0, -0.0),

"mouthShrugUpper": (0.0, -25.0),

"mouthRight": (-27.0, -38.0),
"mouthLeft": ( 27.0, -38.0),

"mouthSmileRight": (-53.0, -8.0),
"mouthSmileLeft": ( 53.0, -8.0),

"mouthDimpleRight": (-48.0, -24.0),
"mouthDimpleLeft": ( 48.0, -24.0),

"mouthFrownRight": (-26.0, -69.0),
"mouthFrownLeft": ( 26.0, -69.0),

"jawForward": (-0.0, -65.0),

"mouthStretchRight": (-53.0, -42.0),
"mouthStretchLeft": ( 53.0, -42.0),

}

0 comments on commit 0aca0c7

Please sign in to comment.