Tutorial ‐ Eye Blinking Extraction

This is the second tutorial on how to use JeFaPaTo. In the first tutorial, we create an EAR score times series using the facial feature extraction GUI. If you haven't done that yet, then please check it out!

This tutorial will take this time series, automatically extract the blinks, and create a summary. In the end, our goal is to recreate the following visual summary!

If you encounter any problem in understanding this tutorial, or JeFaPaTo behaves differently from what is expected, then do not hesitate and create a new issue, and we will try to help you.

During this tutorial, we focus on the 240 FPS examples ear_score_240.csv in the examples folder of the repository.

Blinking Extraction GUI - Selection

JeFaPaTo's GUI is built up using a vertical tab system to distinguish between the tasks (facial feature extraction and blinking extraction). To extract blinks, please select Blinking extraction in the tab menu in the upper left corner of the GUI (the red arrow). Please note that this view is not the default view, so ensure you are in the correct window.

0. GUI Overview

Before extracting the blink, we will first give an overview of how the GUI is set up. It follows the same general structure as the facial feature extraction. You can see the displayed information on the left side, and on the right side, all settings and interactions are shown. As the GUI consists of many elements, we grouped them by utility and will explain them now in more detail (each enumerate item belongs to one of the red boxes in the image below).

1. Blink Extraction Preview

   • This area will be the main area where the extracted blinks will be shown.
   • This area will automatically update once you have run the extraction with the appropriate settings.
   • Furthermore, you can see in the upper-left corner that it consists of two more tabs. This will be used for the table and visual summary once all blinks have been confirmed.

2. EAR Score plot

   • This graph shows the ear plot for the left eye and right eye. Also, here we will follow the same color scheme for the right being red and the left being blue.
   • The x-axis is the video time, and the y-axis is the EAR score value between 0 and 1.
   • You can fully interact with the graph by zooming, scrolling, and panning. However, this might be a bit laggy, depending on the size of your time series.
   • By default, JeFaPaTo tries to load the correct EAR score columns from the according to .csv file. If this fails or is incorrect, you can manually correct this using the selection menu in box box 4.
   • You can also load your .csv feature file by dragging and dropping it onto the graph so that you know that any prior analysis you have done without saving will be lost.

3. Face Preview plot

   • This quality-of-life feature displays the actual frame for each blink. No issues arise if you do not have the original video from which the EAR score was extracted.
   • After the blink extraction, you can click on the rows in the table in box 1, and the corresponding frame of the video will be displayed. This way, you can confirm whether the automatic labeling of the blinking state as partial or complete was correct
   • Currently, the videos can only loaded via dragging and dropping them into this area.

4. Feature Selection

   • Once you have loaded a .csv feature file, we try to load the correct colums from the data. However, if this fails, you can manually correct the fit to the left and right eye.
   • If you obtained or modified the column header of the .csv file, please double-check if we loaded them correctly.

5. Settings Menu

   • This menu pane contains the settings and interactions for the blinking extraction. If no file is loaded, this menu is turned off by default. All settings are restored from the last active JeFaPaTo session.
   • The actual usage will be explained later in the tutorial.
   • On top of the settings, a help button will display the recommended settings for the extraction. Additionally, each set has its dedicated help button to access some help information (either by hovering or pressing the ? button)
   • On the bottom, you can see the Extract button, which will trigger the extraction algorithm. Afterward, the Summary button becomes enabled so that you can compute statistics after checking the extraction (if needed).

6. Export

   • Once you have extracted the blinks, the export function will be available (even if you haven't computed the Summary yet)
   • The exported files will look slightly different Depending on whether you have computed the summary.
   • If you choose Excel as the format, all information will be saved in a single Excel file, and the results will be stored in separate sheets.
   • If you choose CSV as the format, all the information will be saved in separate .csv` files.

7. Graph Interaction Menu

   • These buttons simplify the interaction with the EAR score graph.

1. FileLoading

During this tutorial, we focus on the 240 FPS examples ear_score_240.csv in the examples folder of the repository. You can load this file by dragging and dropping it into the graph area (box 1 in the image below) or using the button in the upper-right corner. After loading the file successfully, the entire EAR score time series will be shown for the left (blue) and right (red) eye. However, errors could happen when finding the correct column headers in the feature .csv file. Therefore, if necessary, you can manually check and correct the selected columns using the drop-down chosen for the left and right eye (see box 2).

The extraction settings menu becomes enabled after the file is fully loaded (see box 3). In the next section, we will show how to interact with it and how to get some help for the features.

You can interact freely with the whole graph by scrolling, zooming, and panning. In the lower-left corner, an A might appear (default feature of the used plotting library); you can press it to auto-scale the graph to fill the complete x and y-axis ranges.

2. Help Settings

Before starting the extraction, you can access the help at any time inside JeFaPaTo by pressing the help button at the top of the settings menu. This will cause a new help window to appear, showing all the information, including recommended settings for 30 FPS and 240 FPS.

3. Extraction Settings

After you read the recommendations for the 240 FPS case, we will set the values accordingly. We build the extraction algorithm based on the scipy.signal.find_peaks algorithm. Hence, we follow a similar naming scheme. If something is unclear for a specific value, press the ? button to get more help inside JeFaPaTo.

Currently, most settings are used to measure frames. We opened a new issue to make them frame-invariant and give everything in ms instead; thus, only the FPS selection matters. Furthermore, we developed a parameter-free extraction algorithm, which will likely be added in a future version of JeFaPaTo. We review the existing settings and their recommended values for 240 FPS.

• Minimum Distance: describes how many frames between two blinks apexis should be required.

• Minimum Prominance: describes the minimum prominence of blink in EAR score. Given that a person with an open eye score of 0.3, values below 0.2 would be considered blinks.

• Minimum Internal Width: describes the minimum length a blink should have. This value captures outliers and small unwanted peaks due to the recording.

• Maximum Internal Width: describes the maximum length a blink should be. This value helps to distinguish between short blinks and fully extended eyelid closures.

• Maximum Matching Distance: describes how many frames are allowed between a left and right eye blink to be considered a match. We attempt to match the left and right eye blinks, and small asynchronicity might appear.

• Partial Threshold Left: is the value for labeling a blink either partial or complete. Using the value auto, the splitting is learned in a data-driven way using Otsu's algorithm.

• Partial Threshold Right: same as for the left eye.

• Video FPS: This value converts all measurements from frame-based into time-based for later statistical analysis. This depends on the original recording of the video (not the internally saved one! be sure to choose the correct one)

• Smoothing: We support optional smoothing using a Savgol Filter onto the EAR score time series. We recommend a windowsize of 7 and a polynomial degree of 3.

After you set these values, press the Extract Blinks button below the settings, and we will look at the extract blinks.

4. Extraction

Once you have pressed the extraction button and the processing is complete, a table will appear in the box 1 area. All extracted and matched blinks for the left and right eye will appear. Also, each blink will be labeled as either partial or complete.

In the graph below (box 2), you can see that blinks also have been annotated there with either a circle for complete or a triangle for partial.

At this point, the Compute Summary buttons are enabled (see box 3), and the current state can be exported (see box 4).

A pop-up window will inform the user if an error occurs during the extraction. Please note that if an issue with the settings configuration leads to invalid results, the user should try to solve these issues themselves. If an unexpected code error arises, the user should open up a new issue on Github so that we can investigate the underlying cause.

Before creating a summary of the blinks, we now elaborate on how to check if the extracted blinks are valid in case the user wants to confirm it.

4.1 Optional - Checking with a video

As mentioned in the GUI overview, the user might drag and drop a video into the according GUI area to match extracted blinks and video frames for manual confirmation. In our case, we have the original 20-minute video of the test person and give it as an example. Please note that for really long videos (1h+), there is an issue with the frame computation, which might introduce some offsets. However, the cause is the underlying library for accessing the video stream data, not JeFaPaTo itself, but we will still address this issue.

After loading the video, the first frame will be shown by default, and a simple face detection algorithm tries to center around the facial area. So that you know, we assume that only one person is visible in the frame.

If you now interact with the table by clicking on any of the rows (see box 1) below, the GUI will shift its focus to that exact frame. In that movement, the graph below will center its view around that blink's apex (see box 2), and the according frame (see box 3) will be loaded from the video. This way, you can confirm or change the annotation label for the left and right eye using the drop-down menu. The colors help to identify the according label, white for not a blink, yellow for a partial blink, and green for a complete blink. The label None might appear if a left or right eye blink happened with the other eye.

The user can now go through the table to check the labeling state. Once this optional step is done, the summary can be computed.

5. Summary

We assume that the annotations and blinking have been confirmed (if confirmed). Now, we will compute the blinks statistics to get an excellent overview of what happened during the video.

First, the Compute Summary button must be pressed.

After that, the GUI should automatically switch to a new tab (see box 1) image below. Here, the interesting statistics are listed as a simple table (which will also be saved during the Export step). But we will now switch to a more interesting and comprehensive visual summary.

Switch the according tab (see the red arrow image below). The visual summary should be visible and give insight into how the blinking behavior varies throughout the video. For more information on how to read this graph, please see the according wiki entry.

6. Exporting

In this last step, we will discuss how to export the results of JeFaPaTo. First, we export the extracted blinks, the matched blinks table, and (if you computed the summary) the summary table. This can be done by pressing the Save button in the export menu (see box 1). The files are automatically saved next to the original input .csv file and can now be opened by other programs for further analysis. The resulting files might vary depending on your chosen export format (see the GUI overview).

Here, we show how the resulting Excel file looks when opened in LibreOffice. You see the same tables in JeFaPaTo with the same column naming as in the GUI (see box 1). Each table is saved in the corresponding sheet (see box 2).

6.1 Exporting the visual summary

To export the visual summary, we rely on the saving interface of the pyqtgraph. You can open the saving menu by right-clicking the graph and selecting the Export option in the flowting window menu. (Please note, if you don't see the Export window, it might open either in the background or on a second screen.)

Once this window opens (see arrow 1), select Image File (arrow 2). Now, you can change the graph's resolution (arrow 3). Copy the graph to your clipboard (arrow 4) or export it to your local hard drive (arrow 5).

Last words

This would be the end of the tutorial on how to extract blinks! You can use the resulting files to do statistical tests in your favorite tool, such as scipy, R, or even SPSS. If you still have issues or other problems, please create a new issue on Github so we can quickly help you!