You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#### 1. Overview [[Paper (in progress)]]()[[Project Webpage (in progress)]]()
10
7
11
-
This repository provides the implementation of the RAW-to-RGB mapping approach and PyNET CNN presented in [this paper](https://arxiv.org/). The model is trained to convert **RAW Bayer data** obtained directly from mobile camera sensor into photos captured with a professional Canon 5D DSLR camera, thus replacing the entire hand-crafted ISP camera pipeline. The provided pre-trained PyNET model can be used to generate full-resolution **12MP photos** from RAW (DNG) image files captured using the Sony Exmor IMX380 camera sensor. More visual results of this approach for the Huawei P20 and BlackBerry KeyOne smartphones can be found [here](http://people.ee.ethz.ch/~ihnatova/pynet.html#demo).
8
+
This repository provides the implementation of further improvement of the PyNet model originally presented in [this paper](https://arxiv.org/abs/2002.05509).
12
9
13
-
<br/>
14
10
15
11
#### 2. Prerequisites
16
12
17
13
- Python: scipy, numpy, imageio and pillow packages
@@ -29,91 +24,36 @@ This repository provides the implementation of the RAW-to-RGB mapping approach a
29
24
30
25
<sub>*Please note that Google Drive has a quota limiting the number of downloads per day. To avoid it, you can login to your Google account and press "Add to My Drive" button instead of a direct download. Please check [this issue](https://github.com/aiff22/PyNET/issues/4) for more information.* </sub>
PyNET architecture has an inverted pyramidal shape and is processing the images at **five different scales** (levels). The model is trained sequentially, starting from the lowest 5th layer, which allows to achieve good reconstruction results at smaller image resolutions. After the bottom layer is pre-trained, the same procedure is applied to the next level till the training is done on the original resolution. Since each higher level is getting **upscaled high-quality features** from the lower part of the model, it mainly learns to reconstruct the missing low-level details and refines the results. In this work, we are additionally using one transposed convolutional layer (Level 0) on top of the model that upsamples the image to its target size.
31
+
[WIP]
44
32
45
-
<br/>
46
33
47
34
#### 5. Training the model
48
35
49
-
The model is trained level by level, starting from the lowest (5th) one:
50
-
51
-
```bash
52
-
python train_model.py level=<level>
53
-
```
54
-
55
-
Obligatory parameters:
56
-
57
-
>```level```: **```5, 4, 3, 2, 1, 0```**
58
-
59
-
Optional parameters and their default values:
60
-
61
-
>```batch_size```: **```50```** - batch size [small values can lead to unstable training] <br/>
62
-
>```train_size```: **```30000```** - the number of training patches randomly loaded each 1000 iterations <br/>
63
-
>```eval_step```: **```1000```** - each ```eval_step``` iterations the accuracy is computed and the model is saved <br/>
>```restore_iter```: **```None```** - iteration to restore (when not specified, the last saved model for PyNET's ```level+1``` is loaded)<br/>
66
-
>```num_train_iters```: **```5K, 5K, 20K, 20K, 35K, 100K (for levels 5 - 0)```** - the number of training iterations <br/>
67
-
>```vgg_dir```: **```vgg_pretrained/imagenet-vgg-verydeep-19.mat```** - path to the pre-trained VGG-19 network <br/>
68
-
>```dataset_dir```: **```raw_images/```** - path to the folder with **Zurich RAW to RGB dataset** <br/>
69
-
70
-
</br>
71
-
72
-
Below we provide the commands used for training the model on the Nvidia Tesla V100 GPU with 16GB of RAM. When using GPUs with smaller amount of memory, the batch size and the number of training iterations should be adjusted accordingly:
36
+
The model is trained level by level, starting from the lowest. The script below incorporates all training steps:
#### 6. Test the provided pre-trained models on full-resolution RAW image files
86
44
87
45
```bash
88
-
python test_model.py level=0 orig=true
46
+
python test_model_keras.py
89
47
```
90
48
91
49
Optional parameters:
92
50
93
-
>```use_gpu```: **```true```**,**```false```** - run the model on GPU or CPU <br/>
94
-
>```dataset_dir```: **```raw_images/```** - path to the folder with **Zurich RAW to RGB dataset** <br/>
51
+
>```--model```: - path to the Keras model checkpoint <br/>
52
+
>```--inp_path```: **```raw_images/test/```** - path to the folder with **Zurich RAW to RGB dataset** <br/>
53
+
>```--out_path```: **```.```** - path to the output images <br/>
95
54
96
-
<br/>
97
55
98
-
#### 7. Test the obtained model on full-resolution RAW image files
99
-
100
-
```bash
101
-
python test_model.py level=<level>
102
-
```
103
-
104
-
Obligatory parameters:
105
-
106
-
>```level```: **```5, 4, 3, 2, 1, 0```**
107
-
108
-
Optional parameters:
109
-
110
-
>```restore_iter```: **```None```** - iteration to restore (when not specified, the last saved model for level=```<level>``` is loaded)<br/>
111
-
>```use_gpu```: **```true```**,**```false```** - run the model on GPU or CPU <br/>
112
-
>```dataset_dir```: **```raw_images/```** - path to the folder with **Zurich RAW to RGB dataset** <br/>
113
-
114
-
<br/>
115
-
116
-
#### 8. Folder structure
56
+
#### 7. Folder structure
117
57
118
58
>```models/``` - logs and models that are saved during the training process <br/>
119
59
>```models/original/``` - the folder with the provided pre-trained PyNET model <br/>
@@ -123,44 +63,34 @@ Optional parameters:
123
63
>```vgg-pretrained/``` - the folder with the pre-trained VGG-19 network <br/>
124
64
125
65
>```load_dataset.py``` - python script that loads training data <br/>
>```vgg.py``` - loading the pre-trained vgg-19 network <br/>
131
71
132
-
<br/>
133
72
134
73
#### 9. Bonus files
135
74
136
75
These files can be useful for further experiments with the model / dataset:
137
76
138
77
>```dng_to_png.py``` - convert raw DNG camera files to PyNET's input format <br/>
139
-
>```evaluate_accuracy.py``` - compute PSNR and MS-SSIM scores on Zurich RAW-to-RGB dataset for your own model <br/>
78
+
>```ckpt2pb_keras.py``` - converts Keras checkpoint to TFLite format <br/>
79
+
>```evaluate_accuracy_tflite.py``` - compute PSNR and MS-SSIM scores on Zurich RAW-to-RGB dataset for TFLite model <br/>
140
80
141
-
<br/>
142
81
143
82
#### 10. License
144
83
145
-
Copyright (C) 2020 Andrey Ignatov. All rights reserved.
84
+
Copyright (C) 2022 Andrey Ignatov. All rights reserved.
146
85
147
86
Licensed under the [CC BY-NC-SA 4.0 (Attribution-NonCommercial-ShareAlike 4.0 International)](https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode).
148
87
149
88
The code is released for academic research use only.
150
89
151
-
<br/>
152
90
153
91
#### 11. Citation
92
+
[WIP]
154
93
155
-
```
156
-
@article{ignatov2020replacing,
157
-
title={Replacing Mobile Camera ISP with a Single Deep Learning Model},
158
-
author={Ignatov, Andrey and Van Gool, Luc and Timofte, Radu},
0 commit comments