feat: video thumbnail preview

[he-patrick]

Description:

This change introduces a thumbnail preview for videos. It uses ffmpeg-next to extract the raw RGB data of a frame at 0%, 25%, 50%, and 75% of a video. It then scores the frames based on their brightness, variance, and sharpness. If the extraction fails (i.e. the video file is corrupted, empty, etc.), then a placeholder image is generated.

Before changes

After changes

Example 1

Frame at 0%

brightness=0.0000, variance=0.1439, sharpness=0.0084, total=0.0503

Frame at 25%

brightness=0.0000, variance=0.0114, sharpness=0.0037, total=0.0050

Frame at 50%, Selected as thumbnail

brightness=0.2459, variance=0.2335, sharpness=0.1187, total=0.1974

Frame at 75%

brightness=0.0000, variance=0.1440, sharpness=0.0341, total=0.0588

Example 2

Frame at 0.0%

brightness=1.0000, variance=0.6925, sharpness=0.0568, total=0.5773

Frame at 25.0%

brightness=1.0000, variance=0.4763, sharpness=0.0870, total=0.5159

Frame at 50.0%

brightness=1.0000, variance=0.2753, sharpness=0.0505, total=0.4375

Frame at 75.0%

brightness=1.0000, variance=0.2424, sharpness=0.0042, total=0.4114

[he-patrick]

@houqp I was trying kiorg out, and thought it would be nice if I could see a preview (thumbnail) of videos, since I store most of my personal videos on my USB-stick and they have names that aren't very descriptive.

If you have the time, could you share any thoughts on this change?

There were a couple of improvements I was thinking of:

• Something like infer could be used to determine the file types rather than parsing the filename.
• Retrieving a frame ~10% through a video as the thumbnail would yield a more accurate representation for the video (the first frame is sometimes black).

I'll work on adding tests in the meantime.

[houqp]

thanks for the contribution, video support is something that I have wanted for myself, it's great that you picked it up.

Something like infer could be used to determine the file types rather than parsing the filename.

We are already using the file_type crate for this use-case, see:

kiorg/src/ui/preview/text.rs

Line 114 in 30bc8d3

let file_type_info = match FileType::try_from_file(&path) {

.

However, I think parsing the file extension is good enough to keep it snappy and simple. If the user provided an incorrect file extension, it's better for them to fix it instead.

Retrieving a frame ~10% through a video as the thumbnail would yield a more accurate representation for the video (the first frame is sometimes black).

This is good idea, instead of going through the first 10%, which can be a lot depending on size of the video. It might be better to do a uniform sample of constant frames across the video to keep the overhead low. For example, extract 4 frames at 0, 25%, 50%, 75%.

You can also take a look into key frame extraction, which is supposed to more likely occur during a scene change.

[he-patrick]

@houqp I switched to a ffmpeg crate because video-rs doesn't support seeking to specific times within the video, and would therefore require a lot of processing to extract frames at 25%, 50%, 75% in large videos.

Looks like all the tests are failing for ffmpeg-sys-next v7.1.3, I'm thinking now that this crate is bindings for ffmpeg so we would still need ffmpeg installed on the local machine.

At least some good news is the calculate_frame_quality function I created to calculate a frame score works well, though could use more fine tuning and additional calculations. Right now, it takes an average score of the brightness, variance, and sharpness.

Any thoughts on what to do next? I can't find a crate that can handle seeking to specific parts of the video and doesn't require a local install of packages.

Edit: Looking back, video-rs also requires ffmpeg to be installed on the machine.

[houqp]

The result looks good, it's okay to have ffmpeg as a dependency, you can install it in the CI to make the pipelines pass.

One thing to consider is that the built binary should not crash if ffmpeg is not installed.

[he-patrick]

The thumbnails are loading in a reasonable time (mostly <1s). Here's an example:

FFmpeg initialization took: 8.25µs
FFmpeg initialization took: 8.542µs
File opening and stream detection took: 26.045709ms
Decoder creation took: 117.667µs
Scaler creation took: 39.667µs
Seek to 0.0% took: 21.5µs
Frame scaling took: 1.205709ms
Pixel extraction took: 19.381917ms
Quality calculation took: 87.438542ms
Total decode and process time for 0.0%: 129.468792ms
Seek to 25.0% took: 273.291µs
Frame scaling took: 974.125µs
Pixel extraction took: 15.945791ms
Quality calculation took: 87.644083ms
Total decode and process time for 25.0%: 118.784875ms
Seek to 50.0% took: 332.875µs
Frame scaling took: 922.375µs
Pixel extraction took: 16.2815ms
Quality calculation took: 86.080541ms
Total decode and process time for 50.0%: 118.806584ms
Seek to 75.0% took: 164.583µs
Frame scaling took: 925.25µs
Pixel extraction took: 16.25725ms
Quality calculation took: 87.076291ms
Total decode and process time for 75.0%: 119.553542ms
**Total frame extraction took: 488.539666ms**

There is a bug where sometimes vertical videos are rotated the wrong way, I'll work on fixing that.

Other than that, a lot of tuning can be done for the brightness, variance, and sharpness calculations to make the selection more accurate, but that will take a lot of time through manual testing. Would you want me to include that work in this PR?

[houqp]

we can leave fine tuning into subsequent PRs, it's already better than what we have right now, so let's ship what we have.

[houqp]

can probably do the score calculation in separate threads in parallel to reduce the load time as a future optimization

[he-patrick]

Image rotation works, but takes 100-200ms. At least it's an infrequent operation.

[he-patrick]

Tested with mp4, m4v, mkv, webm, mov, avi, wmv, mpg, flv sample videos.

[he-patrick]

@houqp Ready for merge, let me know if you'd like any changes.

[houqp]

@he-patrick one missing feature here is that the binary doesn't run on machines that don't have all the necessary libraries installed anymore. The video preview feature should be optional in the sense that when these libraries are present, it works out of the box, if not, it should still run, but display the current preview content or simply display a message to let the users know they need to install the system dependency to enable the preview.

[he-patrick]

@houqp I think it's nice generating the placeholder thumbnail if ffmpeg is not available. Another option is to not generate the placeholder, but at least make the metadata table UI consistent with that of the images preview. WDYT?

Also, thanks for your support on this PR 🙏

[houqp]

yes, that looks great to me. it would be good to still add a note at the end to let the user know that they need to install ffmpeg for full video preview 👍

[he-patrick]

Ok, this is a lot more complicated than I thought. I'll need to use something like the libloading crate to load ffmpeg at runtime. Including the ffmpeg crate in the dependencies breaks the application if ffmpeg isn't installed because the dynamic linker tries to resolve all the ffmpeg library symbols at startup time.

Looks like this might take me some time to figure out, as I also have final exams in a few days 😅

[houqp]

no worries, take your time, let me know if you need suggestions on how to workaround this. i have a few ideas, but it's more fun if you figure this out by yourself ;) loading the dynamic library at runtime is the right first step.

[he-patrick]

Just came home from vacation and getting back on this. Sorry for the wait 🙏