Skip to content

Home

Jump to bottom
Tim Büchner edited this page Apr 18, 2024 · 15 revisions

Welcome to the JeFaPaTo wiki!

Tutorials

This page contains an overview of the tutorials on how to use JeFaPaTo. We focus on these tutorials to be short when using the GUI version of our tool. After that, you should be able to use the tool independently and adapt the settings or workflow to your needs.

For now, we explain how to extract the EAR score and, afterward, the actual blinks from the time series. All steps include annotated images with red boxes, arrows, or numbers to guide you through the UI and UX. If you have any problem with something being unclear, please open an issue so we can improve the wiki tutorials together for other users 😄

The current tutorials are:

Helpful Information

This section will discuss some of the details of the inner workings of JeFaPaTo. Especially information about expected input file and generated output formats, computer vision models, and the correctness of the calculation will be addressed. Also, we give more insights how and which facial features are computed and how to read the visual summary of the statistics.

Input File Formats

Facial Feature Extraction

For the Facial Feature and EAR score extraction only a video file is needed. As the actual extraction is independent of the frames per second, you can choose which ever you fell works rights in your experimental setting. Regarding the video files, we currently support the same ones as OpenCV (the internal encodings of the video containers is handeld by ffmpeg)

  • .mp4
  • .flv
  • .ts
  • .mts
  • .avi
  • .mov
  • .wmv

We tested JeFaPaTo's capabilites regarding different video resolution and all tests with 1280x720, 1920x1080, and 3840x2160 succeeded. In our custom loading threads in the background we limit the usage RAM to not crash a computer. Therefore, loading and processing speed are rely on CPU, GPU, and hard disk space of the computer. If a video was recorded in a different orientation, you can use the Rotation setting inside the extraction GUI to adjust that.

Eye Blinkind Extraction

This method only expects .csv file as input with a header naming the according columns. Each row describes the frame id and the columns shall the be the actual features. If you use JeFaPaTo for the EAR score extraction, we automatially detect the EAR2D_* or EAR3D_* correct columns. If you use your own or modified .csv file, you have to manually select the custom in the acording menu area.

Output File Formats

Facial Feature Extraction

The output file for the facial feature extraction is always a .csv file with ; as chosen separator. The file will be created automatcially next to video file with the current timestamp. The file only uses ascii symbols and contains a header row. Each column in the header descirbes the extracted facial feature and gives a state about the extraction validity. Each row represents a single frame with including the according frame id inside the video

In the image below you can see such a extraction result:

  • frame: the actual frame id inside the video
  • EAR2D6_l: the according EAR2D6 score for the left eye at the according frame
  • EAR2D6_r: the according EAR2D6 score for the right eye at the according frame
  • EAR2D6_valid: is True or False, if the computation of the EAR score was valid eg. all landmarks were in the image, values are withing a logical range, etc.
  • BS_Valid: is True or False if the extraction of the facial blendshapes was valid eg. the face was fully visible.

Eye Blinkind Extraction

Computer Vision Models and Further Methods

Facial Features

Visual Summary

Tips and Tricks

Bounding Box

Video Conversion

Clone this wiki locally