Deep Multimodal Fusion via Projective Networks

This is the official repository of the paper "Deep Multimodal Fusion of Data with Heterogeneous Dimensionality via Projective Networks," published in IEEE Journal of Biomedical and Health Informatics on January 2024.

[IEEE Xplore] [arXiv]

For questions about the code or the paper, please contact the first author, José Morano: [email protected], or open an issue in this repository (preferred).

Framework

Network architecture

Setting up the environment

Install pyenv.

curl https://pyenv.run | bash

Install clang. E.g.:

sudo dnf install clang

Install Python version 3.6.8.

CC=clang pyenv install -v 3.6.8

Create and activate Python environment.

~/.pyenv/versions/3.6.8/bin/python3 -m venv venv/
source venv/bin/activate  # bash
. venv/bin/activate.fish  # fish

Update pip.

pip install --upgrade pip

Install requirements using requirements.txt.

pip3 install -r requirements.txt

Running the code

See the options in config.py and sample commands in run.sh.

The main script for running the training is train.py, while validate_ensemble.py is used for testing.

The dataloaders expect the data to be (approximately) in the following format. The id0 is the image ID. Please check the dataloaders for more details and modify them as needed.

dataset_dir/
    id0/
        preprocessed_images/
            bscan_size.mask.id0.png
        slo.id0.png
        faf.id0.png
        bscan.id0.npy
        spacing.id0.npy

Spacing is needed for the computation of the Hausdorff distance.

Other information

Level5 versions of the architectures differ from the other versions in that the fusion is done at the 5 levels, not only the first 4.

Citation

If you use this code, please cite the following paper:

@ARTICLE{morano2024deep,
  author={Morano, José and Aresta, Guilherme and Grechenig, Christoph and Schmidt-Erfurth, Ursula and Bogunović, Hrvoje},
  journal={IEEE Journal of Biomedical and Health Informatics},
  title={Deep Multimodal Fusion of Data with Heterogeneous Dimensionality via Projective Networks},
  year={2024},
  volume={},
  number={},
  pages={1-12},
  doi={10.1109/JBHI.2024.3352970}
}