Paper accepted for publication at ICIP'2021: read the preprint here.
Code: Please, check the code for our extended work https://github.com/avagait/ugaitnet
Multimodal systems for gait recognition have gained a lot of attention. However, there is a clear gap in the study of missing modalities, which represents real-life scenarios where sensors fail or data get corrupted. Here, we investigate how to handle missing modalities for gait recognition. We propose a single and flexible framework that uses a variable number of input modalities. For each modality, it consists of a branch and a binary unit indicating whether the modality is available; these are gated and merged together. Finally, it generates a single and compact ‘multimodal’ gait signature that encodes biometric information of the input. Our framework outperforms the state of the art on TUM-GAID and extensive experiments reveal its effectiveness for handling missing modalities even in the multiview setup of CASIA-B.
Input: (OF?, Gray?, Depth? ellipses) binary input units indicating whether the modality is available – here, depth is not available (dashed red cross); (volumes) sequences of L frames for the different modalities. After fusing the single-modality signatures, a multimodal gait signature of d dimensions is further compressed by FC1. The final FC2 contains C classes (used just for training).
The proposed model is depicted in the following figure:
At training, the network learns multimodal signatures so that the distance D between a pair of signatures of the same subject is lower than the distance between signatures of different subjects, independently of the modalities used to generate the signatures. To imitate test situations, some modalities are disabled (i.e. missing) at training (empty shapes).
R. Delgado-Escaño, F. Castro, N. Guil, V. Kalogeiton, M. Marín-Jiménez. "Multimodal gait recognition under missing modalities". IEEE ICIP, 2021