Numan Saeed, Ikboljon Sobirov, Roba Al Majzoub, and Mohammad Yaqub
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
{numan.saeed, ikboljon.sobirov, roba.majzoub, mohammad.yaqub}@mbzuai.ac.ae
When oncologists estimate cancer patient survival, they rely on multimodal data. Even though some multimodal deep learning methods have been proposed in the literature, the majority rely on having two or more independent networks that share knowledge at a later stage in the overall model. On the other hand, oncologists do not do this in their analysis but rather fuse the information in their brain from multiple sources such as medical images and patient history. This work proposes a deep learning method that mimics oncologists' analytical behavior when quantifying cancer and estimating patient survival. We propose TMSS, an end-to-end Transformer based Multimodal network for Segmentation and Survival predication that leverages the superiority of transformers that lies in their abilities to handle different modalities. The model was trained and validated for segmentation and prognosis tasks on the training dataset from the HEad & NeCK TumOR segmentation and the outcome prediction in PET/CT images challenge (HECKTOR). We show that the proposed prognostic model significantly outperforms state-of-the-art methods with a concordance index of 0.763 while achieving a comparable dice score of 0.772 to a standalone segmentation model. TMSS implementation code will be publicly available soon.
Figure 1. An illustration of the proposed TMSS architecture and the multimodal training strategy. TMSS linearly projects EHR and multimodal images into a feature vector and feeds it into a Transformer encoder. The CNN decoder is fed with the input images, skip connection outputs at different layers, and the final layer output to perform the segmentation, whereas the prognostic end utilizes the output of the last layer of the encoder to predict the risk score.
The checkpoint to the model is available at
