Home

Welcome to the JeFaPaTo Wiki! In this wiki, we give a short overview of how to use JeFaPaTo and try to support you with simple tutorials to help you create your interesting studies later. If you have any questions or issues or find errors, please let us know via the issue function of GitHub.

Tutorials

This page contains an overview of the tutorials on how to use JeFaPaTo. We focus on these tutorials to be short when using the GUI version of our tool. After that, you should be able to use the tool independently and adapt the settings or workflow to your needs.

For now, we explain how to extract the EAR score and, afterward, the actual blinks from the time series. All steps include annotated images with red boxes, arrows, or numbers to guide you through the UI and UX. If you have any problem with something being unclear, please open an issue so we can improve the wiki tutorials together for other users 😄

The current tutorials are:

• EAR Score Extraction
• Eye Blinking Extraction

Helpful Information

This section will discuss some details of the inner workings of JeFaPaTo. Mainly, information about expected input files and generated output formats, computer vision models, and the correctness of the calculation will be addressed. Also, we give more insights into how and which facial features are computed and how to read the visual summary of the statistics.

Input File Formats

Facial Feature Extraction

Only a video file is needed for the Facial Feature and EAR score extraction. As the extraction is independent of the frames per second, you can choose whichever rigfeelyou frightens your experimental setting. Regarding the video files, we currently support the same ones as OpenCV (the internal encodings of the video containers are handled by ffmpeg)

• .mp4
• .flv
• .ts
• .mts
• .avi
• .mov
• .wmv

We tested JeFaPaTo's capabilities regarding different video resolutions, and all tests with 1280x720, 1920x1080, and 3840x2160succeeded. In our custom loading threads in the background, we limit RAM usage so as not to crash a computer. Therefore, loading and processing speed rely on the computer's CPU, GPU, and hard disk space. If a video was recorded in a different orientation, you can use theRotation` setting inside the extraction GUI to adjust that.

Eye Blinkind Extraction

This method only expects a .csv file as input with a header naming the according columns. Each row describes the frame id; the columns shall be the actual features. If you use JeFaPaTo for the EAR score extraction, we automatically detect the EAR2D_* or EAR3D_* correct columns. If you use your own or modified .csv file, you must manually select the custom in the according menu area.

Output File Formats

Facial Feature Extraction

The output file for the facial feature extraction is always a .csv file with ; as a chosen separator. The file will be created automatically next to the video file with the current timestamp. The file only uses ASCII symbols and contains a header row. Each column in the header describes the extracted facial feature and gives a state of the extraction validity. Each row represents a single frame incl, including according to the frame id inside the video.

In the image below, you can see such an extraction result:

• frame: the actual frame id inside the video
• EAR2D6_l: the according EAR2D6 score for the left eye at the according frame
• EAR2D6_r: the according EAR2D6 score for the right eye at the according frame
• EAR2D6_valid: is True or False if the computation of the EAR score was valid, e.g., all landmarks were in the image, values were within a logical range, etc.
• BS_Valid: is True or False if the extraction of the facial blendshapes was valid, e.g., the face was unmistakable.

Eye Blinkind Extraction

Computer Vision Models and Further Methods

Facial Features

Visual Summary

Tips and Tricks

Bounding Box

Video Conversion