UX issues and misunderstandings!

[Timozen]

This issue was created to collect and categorize GUI user experience problems.

[draabe]

Hi @Timozen,

quick summary of my UX while reviewing the package. Please don't feel the need to solve each and every point as part of the review process. I'm certain some remarks exceed the scope of the review and some are most likely a matter of opinion or personal preference. Nevertheless, I hope these help in creating a wiki/restructuring the paper/refining the UI.

Top level remarks

1.) I generally felt quite comfortable with the interface. The overall layouting (tabs for tasks, stuff happening on the left, controls on the right) feels intuitive and sensible. The main difficulty I had while "understanding" the UI was to figure out what I had to do right after starting it. This involved realizing that both tabs aren't really equal in that they both allow to solve a certain task, but rather that the first one is a pre-requisite for the second, and while the first one seems like a routine data-processing task, the second one actually is an analysis task containing your novel algorithms (Well, at least that's what I concluded). I don't know if that makes sense, but I guess what I'm getting at is that maybe some sort of entry point to the UX (maybe in the Wiki, or a welcoming screen, or a small textbox on each tab view) could mitigate this initial confusion and ease the user into the process a bit smoother.

2.) Furthermore, throughout the process I was quite curious on the validity of the calculations you make and I could see users having similar questions when JeFaPaTo is being used in medical or scientific contexts, especially giving the importance of trustworthy AI in these settings. For example, I was pondering on a couple of things:

• are the CV components (face bounding box detection, landmark detection) adopted from (pre)-trained models or trained/fine-tuned by you?
• How well is their performance and are there any potential quirks with this automatic detection?
• Do any other components contain machine learning models?
• How well are landmarks suited to detect all those features? What's the performance on each single feature?
• How well does the scipy.signal.find_peaks function work here? Could there be false positive blinks detected due to irrelevant local optima in the time series?
• How "standard" is the entire pipeline from raw video to final results, and which components are adopted (and evaluated in previous work) and how much is novel and introduced by you?

I'm aware that face detection models perform well and stable nowadays, and your average user probably doesn't ask these questions. But, ideally, I'd be able to get a basic idea on potential error sources and especially error margins of the results I extract from your software.

Specifics

These are some specific questions I asked myself during usage.

• What are requirements of input videos: file format, temporal resolution, image resolution, obstructedness (eyewear), minimal size of face, angle of face, etc.
• If I can adopt the bounding box, what does the required/ideal bounding box look like? Right on the face contour or with some padding?
• What happens after I run (press analyze) the Facial Feature Extraction? It took me some time to figure it created a .csv and where it was stored. Similar to find the exported file from the blinking extraction.
• What does the little "A" mean on the top left corner of both plots?
• Facial Feature Extraction plot: Why's the x-axis negative? What's the y-axis labeling (I assume prediction confidences?)? What's the green line doing? What's blue and red?
• How do I interpret the outcome of the facial features? Especially, more documentation on the output format of the features would be nice. For example, the landmarks are all tuples with three entries. I guess these are coordinates? If so, what's the coordinate system? I know this is me nit-picking, but I believe it's quite essential to know if I want to use JeFaPaTo for feature extraction and use these in further analyses.
• As discussed above, what are my error margins on all the features and the extracted blinks?
• The visual summary of the eye blinking extraction is really great. Is there a way to (high-resolution) export it? I'd need a really big screen to create a screenshot of sufficient quality for further use.
• As you provide executables (which I think is great), be aware that you're software might be used completely detached from the repository or source code. Thus, it might make sense to add a little About or Help section where you can link to the repo/wiki and acknowledge the authors/source of the software (and potentially add software version, License, etc).

Hope that helps, and feel free to reach out if anything is unclear. Otherwise, I'll just wait until you let us know in the main review thread about your progress!

Best,
@draabe

[Timozen]

Thanks a lot @draabe!!!

I updated the wiki page accordingly and included much new information.
Many of your points are valid and should be handled/noted somewhere, at least. I hope I have added everything to the wiki page.