This is the official repository of XVFI (eXtreme Video Frame Interpolation), https://arxiv.org/abs/2103.16206
Last Update: 20210607
We provide the training and test code along with the trained weights and the dataset (train+test) used for XVFI. If you find this repository useful, please consider citing our paper.
The 4K@30fps input frames are interpolated to be 4K@240fps frames. All results are encoded at 30fps to be played as x8 slow motion and spatially down-scaled due to the limit of file sizes. All methods are trained on X-TRAIN.
Some examples of X4K1000FPS dataset, which are frames of 1000-fps and 4K-resolution. Our dataset contains the various scenes with extreme motions. (Displayed in spatiotemporally subsampled .gif files)
We provide our X4K1000FPS dataset which consists of X-TEST and X-TRAIN. Please refer to our main/suppl. paper for the details of the dataset. You can download the dataset from this dropbox link.
X-TEST consists of 15 video clips with 33-length of 4K-1000fps frames. It follows the below directory format:
├──── YOUR_DIR/
├──── test/
├──── Type1/
├──── TEST01/
├──── 0000.png
├──── ...
└──── 0032.png
├──── TEST02/
├──── 0000.png
├──── ...
└──── 0032.png
├──── ...
├──── ...
X-TRAIN consists of 4,408 clips from various types of 110 scenes. The clips are 65-length of 1000fps frames. Each frame is the size of 768x768 cropped from 4K frame. It follows the below directory format:
├──── YOUR_DIR/
├──── train/
├──── 002/
├──── occ008.320/
├──── 0000.png
├──── ...
└──── 0064.png
├──── occ008.322/
├──── 0000.png
├──── ...
└──── 0064.png
├──── ...
├──── ...
After downloading the files from the link, decompress the encoded_test.tar.gz and encoded_train.tar.gz. The resulting .mp4 files can be decoded into .png files via running mp4_decoding.py. Please follow the instruction written in mp4_decoding.py.
Our code is implemented using PyTorch1.7, and was tested under the following setting:
- Python 3.7
- PyTorch 1.7.1
- CUDA 10.2
- cuDNN 7.6.5
- NVIDIA TITAN RTX GPU
- Ubuntu 16.04 LTS
Caution: since there is "align_corners" option in "nn.functional.interpolate" and "nn.functional.grid_sample" in PyTorch1.7, we recommend you to follow our settings. Especially, if you use the other PyTorch versions, it may lead to yield a different performance.
- Download the source codes in a directory of your choice <source_path>.
- First download our X-TEST test dataset by following the above section 'X4K1000FPS'.
- Download the pre-trained weights, which was trained by X-TRAIN, from this link to place in <source_path>/checkpoint_dir/XVFInet_X4K1000FPS_exp1.
XVFI
└── checkpoint_dir
└── XVFInet_X4K1000FPS_exp1
├── XVFInet_X4K1000FPS_exp1_latest.pt
- Run main.py with the following options in parse_args:
python main.py --gpu 0 --phase 'test' --exp_num 1 --dataset 'X4K1000FPS' --module_scale_factor 4 --S_tst 5 --multiple 8 ==> It would yield (PSNR/SSIM/tOF) = (30.12/0.870/2.15).
python main.py --gpu 0 --phase 'test' --exp_num 1 --dataset 'X4K1000FPS' --module_scale_factor 4 --S_tst 3 --multiple 8 ==> It would yield (PSNR/SSIM/tOF) = (28.86/0.858/2.67).
- After running with the above test option, you can get the result images in <source_path>/test_img_dir/XVFInet_X4K1000FPS_exp1, then obtain the PSNR/SSIM/tOF results per each test clip as "total_metrics.csv" in the same folder.
- Our proposed XVFI-Net can start from any downscaled input upward by regulating '--S_tst', which is adjustable in terms of the number of scales for inference according to the input resolutions or the motion magnitudes.
- You can get any Multi-Frame Interpolation (x M) result by regulating '--multiple'.
- Download the source codes in a directory of your choice <source_path>.
- First download Vimeo90K dataset from this link (including 'tri_trainlist.txt') to place in <source_path>/vimeo_triplet.
XVFI
└── vimeo_triplet
├── sequences
readme.txt
tri_testlist.txt
tri_trainlist.txt
- Download the pre-trained weights (XVFI-Net_v), which was trained by Vimeo90K, from this link to place in <source_path>/checkpoint_dir/XVFInet_Vimeo_exp1.
XVFI
└── checkpoint_dir
└── XVFInet_Vimeo_exp1
├── XVFInet_Vimeo_exp1_latest.pt
- Run main.py with the following options in parse_args:
python main.py --gpu 0 --phase 'test' --exp_num 1 --dataset 'Vimeo' --module_scale_factor 2 --S_tst 1 --multiple 2==> It would yield PSNR = 35.07 on Vimeo90K.
- After running with the above test option, you can get the result images in <source_path>/test_img_dir/XVFInet_Vimeo_exp1.
- There are certain code lines in front of the 'def main()' for a convenience when running with the Vimeo option.
- The SSIM result of 0.9760 as in Fig. 8 was measured by matlab ssim function for a fair comparison after running the above guide because other SOTA methods did so. We also upload "compare_psnr_ssim.m" matlab file to obtain it.
- It should be noted that there is a typo "S_trn and S_tst are set to 2" in the current version of XVFI paper, which should be modified to 1 (not 2), sorry for inconvenience.
- Download the source codes in a directory of your choice <source_path>.
- First prepare your own video datasets in <source_path>/custom_path by following a hierarchy as belows:
XVFI
└── custom_path
├── scene1
├── 'xxx.png'
├── ...
└── 'xxx.png'
...
├── sceneN
├── 'xxxxx.png'
├── ...
└── 'xxxxx.png'
-
Download the pre-trained weights trained on X-TRAIN or Vimeo90K as decribed above.
-
Run main.py with the following options in parse_args (ex) x8 Multi-Frame Interpolation):
# For the model trained on X-TRAIN
python main.py --gpu 0 --phase 'test_custom' --exp_num 1 --dataset 'X4K1000FPS' --module_scale_factor 4 --S_tst 5 --multiple 8 --custom_path './custom_path'# For the model trained on Vimeo90K
python main.py --gpu 0 --phase 'test_custom' --exp_num 1 --dataset 'Vimeo' --module_scale_factor 2 --S_tst 1 --multiple 8 --custom_path './custom_path'- Our proposed XVFI-Net can start from any downscaled input upward by regulating '--S_tst', which is adjustable in terms of the number of scales for inference according to the input resolutions or the motion magnitudes.
- You can get any Multi-Frame Interpolation (x M) result by regulating '--multiple'.
- It only supports for '.png' format.
- Since we can not cover diverse possibilites of naming rule for custom frames, please sort your own frames properly.
- Download the source codes in a directory of your choice <source_path>.
- First download our X-TRAIN train/val/test datasets by following the above section 'X4K1000FPS' and place them as belows:
XVFI
└── X4K1000FPS
├── train
├── 002
├── ...
└── 172
├── val
├── Type1
├── Type2
├── Type3
├── test
├── Type1
├── Type2
├── Type3
- Run main.py with the following options in parse_args:
python main.py --phase 'train' --exp_num 1 --dataset 'X4K1000FPS' --module_scale_factor 4 --S_trn 3 --S_tst 5- Download the source codes in a directory of your choice <source_path>.
- First download Vimeo90K dataset from this link (including 'tri_trainlist.txt') to place in <source_path>/vimeo_triplet.
XVFI
└── vimeo_triplet
├── sequences
readme.txt
tri_testlist.txt
tri_trainlist.txt
- Run main.py with the following options in parse_args:
python main.py --phase 'train' --exp_num 1 --dataset 'Vimeo' --module_scale_factor 2 --S_trn 1 --S_tst 1- You can freely regulate other arguments in the parser of main.py, here
Hyeonjun Sim*, Jihyong Oh*, and Munchurl Kim "XVFI: eXtreme Video Frame Interpolation", https://arxiv.org/abs/2103.16206, 2021. (* equal contribution)
BibTeX
@article{sim2021xvfi,
title={XVFI: eXtreme Video Frame Interpolation},
author={Sim, Hyeonjun and Oh, Jihyong and Kim, Munchurl},
journal={arXiv preprint arXiv:2103.16206},
year={2021}
}If you have any question, please send an email to either [email protected] or [email protected].
The source codes and datasets can be freely used for research and education only. Any commercial use should get formal permission first.