Skip to content
/ PR2022 Public

Code for the paper Dual-frame Spatio-temporal Feature Modulation for Video Enhancement

Notifications You must be signed in to change notification settings

pwp1208/PR2022

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 

Repository files navigation

Dual-frame Spatio-temporal Feature Modulation for Video Enhancement (PR-2022)

Prashant W Patil, Sunil Gupta, Santu Rana, and Svetha Venkatesh

paper


Abstract: Current video enhancement approaches have achieved good performance in specific rainy, hazy, foggy, and snowy weather conditions. However, they currently suffer from two important limitations. First, they can only handle degradation caused by single weather. Second, they use large, complex models with 10–50 millions of parameters needing high computing resources. As video enhancement is a pre- processing step for applications like video surveillance, traffic monitoring, autonomous driving, etc., it is necessary to have a lightweight enhancement module. Therefore, we propose a dual-frame spatio- temporal feature modulation architecture to handle the degradation caused by diverse weather condi- tions. The proposed architecture combines the concept of spatio-temporal multi-resolution feature mod- ulation with a multi-receptive parallel encoders and domain-based feature filtering modules to learn domain-specific features. Further, the architecture provides temporal consistency with recurrent feature merging, achieved by providing feedback of the previous frame output. The indoor (REVIDE, NYUDepth), synthetically generated outdoor weather degraded video de-hazing, and de-raining with veiling effect databases are used for experimentation. Also, the performance of the proposed method is analyzed for night-time de-hazing and de-raining with veiling effect weather conditions. Experimental results show the superior performance of our framework compared to existing state-of-the-art methods used for video de-hazing (indoor/outdoor) and de-raining with veiling effect weather conditions..


Network Overview

Requirements:

Python >= 3.5

Tensorflow == 2.0

Numpy

PIL

Testing Videos:

Keep Testing Videos Frames in "video/{dataset}" folder.

Checkpoints:

The checkpoints are provided for:
1. Keep the checkpoints in "./checkpoint/{}/"
2. Use the checkpoints provided in "ckpt" for synthetically generated day and night-time video de-hazing and de-raining with veiling effect case.

Download the checkpoint: Checkpoint

Testing Procedure:

1. Run "test.py"
3. Results will be saved in "outputs" folder

Database:

Synthetically Generated Day and Night-time Weather Degraded Database is available at: Database

Please use DAVIS-2016 database for ground-truth purpose.


## Citation
If our method is useful for your research, please consider citing:
    
     @article{patil2022dual,
 	title={Dual-frame Spatio-temporal Feature Modulation for Video Enhancement},
  	author={Patil, Prashant W and Gupta, Sunil and Rana, Santu and Venkatesh, Svetha},
  	journal={Pattern Recognition},
  	pages={108822},
  	year={2022},
  	publisher={Elsevier}
        }


## Contact
Please contact [email protected], if you are facing any issue.


About

Code for the paper Dual-frame Spatio-temporal Feature Modulation for Video Enhancement

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages