Bi-Directional ConvLSTM U-Net with Densely Connected Convolutions

Deep auto-encoder-decoder network for medical image segmentation with state of the art results on skin lesion segmentation, lung segmentation, and retinal blood vessel segmentation. This method applies bidirectional convolutional LSTM layers in U-net structure to non-linearly encode both semantic and high-resolution information with non-linearly technique. Furthermore, it applies densely connected convolution layers to include collective knowledge in representation and boost convergence rate with batch normalization layers. If this code helps with your research please consider citing the following papers:

R. Azad, M. Asadi, Mahmood Fathy and Sergio Escalera "Bi-Directional ConvLSTM U-Net with Densely Connected Convolutions ", ICCV, 2019, download link.

M. Asadi, R. Azad, Mahmood Fathy and Sergio Escalera "Multi-level Context Gating of Embedded Collective Knowledge for Medical Image Segmentation", The first two authors contributed equally. arXiv:2003.05056, 2020, download link.

Please consider starring us, if you found it useful. Thanks

Updates

• July 20, 2020: SEDU model added to the Skin Lesion segmentation code (inside models.py), now you can use this model for higher performance on skin lesion segmentation, inside the train file call the SEDU_Net_D3 model).
• March 5, 2020: An extended version of the network has been released(Complete implemenation for SKin Lesion Segmentation on ISIC 217, Skin Lesion Segmentation PH2 Dataset and cell nuclei along with the network implementation will be update soon).
• December 4, 2019: Document Image Binarization using BCDU-Net on DIBCO Challenges has been implemented, best performance on DIBCO series link
• Augest 28, 2019: First release (Complete implemenation for SKin Lesion Segmentation on ISIC 218, Retina Blood Vessel Segmentation and Lung segmentation dataset added.)
• Augest 27, 2019: Paper Accepted in the ICCV workshop 2019 (Oral presentation).

Prerequisties and Run

This code has been implemented in python language using Keras libarary with tensorflow backend and tested in ubuntu OS, though should be compatible with related environment. following Environement and Library needed to run the code:

• Python 3
• Keras - tensorflow backend

Run Demo

For training deep model for each task, go to the related folder and follow the bellow steps:

Skin Lesion Segmentation

1- Download the ISIC 2018 train dataset from this link and extract both training dataset and ground truth folders inside the dataset_isic18.
2- Run Prepare_ISIC2018.py for data preperation and dividing data to train,validation and test sets.
3- Run train_isic18.py for training BCDU-Net model using trainng and validation sets. The model will be train for 100 epochs and it will save the best weights for the valiation set. You can also train U-net model for this dataset by changing model to unet, however, the performance will be low comparing to BCDU-Net.
4- For performance calculation and producing segmentation result, run evaluate.py. It will represent performance measures and will saves related figures and results in output folder.

Retina Blood Vessel Segmentation

1- Download Drive dataset from this link and extract both training and test folders in a folder name DRIVE (make a new folder with name DRIVE)
2- Run prepare_datasets_DRIVE.py for reading whole data. This code will read all the train and test samples and will saves them as a hdf5 file in the DRIVE_datasets_training_testing folder.
3- The next step is to extract random patches from the training set to train the model, to do so, Run save_patch.py, it will extract random patches with size 64*64 and will save them as numpy file. This code will use help_functions.py, spre_processing.py and extract_patches.py functions for data normalization and patch extraction.
4- For model training, run train_retina.py, it will load the training data and will use 20% of training samples as a validation set. The model will be train for 50 epochs and it will save the best weights for the valiation set.
4- For performance calculation and producing segmentation result, run evaluate.py. It will represent performance measures and will saves related figures and results in test folder.
Note: For image pre-processing and patch extraction we used this github's code.

Lung Segmentation

1- Download the Lung Segmentation dataset from Kaggle link and extract it.
2- Run Prepare_data.py for data preperation, train/test seperation and generating new masks around the lung tissues. 3- Run train_lung.py for training BCDU-Net model using trainng and validation sets (20 percent of the training set). The model will be train for 50 epochs and it will save the best weights for the valiation set. You can train either BCDU-net model with 1 or 3 densly connected convolutions.
4- For performance calculation and producing segmentation result, run evaluate_performance.py. It will represent performance measures and will saves related figures and results.

Quick Overview

Structure of the Bidirection Convolutional LSTM that used in our network

Structure of the BConvLSTM+SE that used in our network (MCGU-Net)

Results

For evaluating the performance of the proposed method, Two challenging task in medical image segmentaion has been considered. In bellow, results of the proposed approach illustrated.

Task 1: Retinal Blood Vessel Segmentation

Performance Comparision on Retina Blood Vessel Segmentation

In order to compare the proposed method with state of the art appraoches on retinal blood vessel segmentation, we considered Drive dataset.

Methods	Year	F1-scores	Sensivity	Specificaty	Accuracy	AUC
Chen etc. all Hybrid Features	2014	-	0.7252	0.9798	0.9474	0.9648
Azzopardi et. all Trainable COSFIRE filters	2015	-	0.7655	0.9704	0.9442	0.9614
Roychowdhury and et. all Three Stage Filtering	2016	-	0.7250	0.9830	0.9520	0.9620
Liskowsk etc. allDeep Model	2016	-	0.7763	0.9768	0.9495	0.9720
Qiaoliang et. all Cross-Modality Learning Approach	2016	-	0.7569	0.9816	0.9527	0.9738
Ronneberger and et. all U-net	2015	0.8142	0.7537	0.9820	0.9531	0.9755
Alom etc. all Recurrent Residual U-net	2018	0.8149	0.7726	0.9820	0.9553	0.9779
Oktay et. all Attention U-net	2018	0.8155	0.7751	0.9816	0.9556	0.9782
Alom et. all R2U-Net	2018	0.8171	0.7792	0.9813	0.9556	0.9784
Azad et. all Proposed BCDU-Net	2019	0.8222	0.8012	0.9784	0.9559	0.9788

Retinal blood vessel segmentation result on test data

Skin Lesion Segmentation

Performance Evalution on the Skin Lesion Segmentation task

Methods	Year	F1-scores	Sensivity	Specificaty	Accuracy	PC	JS
Ronneberger and etc. all U-net	2015	0.647	0.708	0.964	0.890	0.779	0.549
Alom et. all Recurrent Residual U-net	2018	0.679	0.792	0.928	0.880	0.741	0.581
Oktay et. all Attention U-net	2018	0.665	0.717	0.967	0.897	0.787	0.566
Alom et. all R2U-Net	2018	0.691	0.726	0.971	0.904	0.822	0.592
Azad et. all Proposed BCDU-Net	2019	0.847	0.783	0.980	0.936	0.922	0.936
Azad et. all MCGU-Net	2020	0.895	0.848	0.986	0.955	0.947	0.955

Skin Lesion Segmentation results

Lung Segmentation

Performance Evalution on the Lung Segmentation task

Methods	Year	F1-scores	Sensivity	Specificaty	Accuracy	AUC	JS
Ronneberger and etc. all U-net	2015	0.9658	0.9696	0.9872	0.9872	0.9784	0.9858
Alom et. all Recurrent Residual U-net	2018	0.9638	0.9734	0.9866	0.9836	0.9800	0.9836
Alom et. all R2U-Net	2018	0.9832	0.9944	0.9832	0.9918	0.9889	0.9918
Azad et. all Proposed BCDU-Net	2019	0.9904	0.9910	0.9982	0.9972	0.9946	0.9972

Lung Segmentation results

Cell Nuclei Segmentation

Cell Nuclei Segmentation results

Model weights

You can download the learned weights for each task in the following table.

Task	Dataset	Learned weights
Retina Blood Vessel Segmentation	Drive	BCDU_net_D3
Skin Lesion Segmentation	ISIC2018	BCDU_net_D3
Lung Segmentation	Lung kaggle	BCDU_net_D3

Query

All implementation done by Reza Azad. For any query please contact us for more information.

rezazad68@gmail.com