How to perform the task of generating video captions

[cascat0]

Great work!

How to perform the task of generating video captions？

[Flaick]

Hello, thanks for the appreciation. Here we follow CapDec to perform the text-only training for image captioning.

[cascat0]

Hello, thanks for the appreciation. Here we follow CapDec to perform the text-only training for image captioning.

Thank you for your answer. I mean how I can use this code to infer captions directly. My requirement is to use this model to generate captions for my surgical video dataset (i.e., image-to-text). However, I only see the three inference modes of 'video', 'text' and 'all' in README, which seem to be unable to implement image-to-text transformation.

[Flaick]

Oh ok, so the current model is CLIP-like architecture, which only includes the visual and text encoders. The function of caption generation requires the trained text decoder trained from CapDec. We do not have this in the current repo, but let me check if we can integrate it. I will get to you back soon

[cascat0]

Oh ok, so the current model is CLIP-like architecture, which only includes the visual and text encoders. The function of caption generation requires the trained text decoder trained from CapDec. We do not have this in the current repo, but let me check if we can integrate it. I will get to you back soon

Thank you very much. I really need this feature for my graduation thesis.

[Flaick]

Hello, hope this is not too late, we have released the codebase for that, please refer to this repo: https://github.com/CAMMA-public/Surg-FTDA