all_splits.h5

[auh-12]

Dear author. Can you share how to generate all_splits.h5 files?

[davidnvq]

I am sorry to ask why do you need all_splits.h5 file? It is only to support pre-extracted features, which leads to lower performance.

Since the file is very large (70GB), it is not convenient for me to upload it. Instead, you can also generate the file by yourself using the provided code.
https://github.com/davidnvq/grit/blob/main/tools/extract_features.py

[auh-12]

Thank you. In the README.md, it is stated that in order to achieve higher CIDEr scores, the backbone and detector need to be frozen, and the detector requires all_splits.h5 files.

[davidnvq]

Sorry for this misunderstanding. If we use all_splits.h5 the performance will 134 CIDEr only.
Freezing the backbone and detector doesn't need the all_splits.h5 (this is only for pre-extraction of visual features), actually.

[auh-12]

Thank you. If I want to achieve 139.2 CIDEr. I must first freezing_xe and freezing_sc and then freeze freezing_xe and freezing_sc. Is it right?

[davidnvq]

Please run the following instead. In this way, you freeze the backbone and detector but the images are fed forward into the model every iteration you finetune the model:

export DATA_ROOT=path/to/coco_dataset
# with pretrained object detector on 4 datasets
python train_caption.py exp.name=caption_4ds model.detector.checkpoint=4ds_detector_path \
optimizer.freezing_xe_epochs=0 \
optimizer.freezing_sc_epochs=0 \
optimizer.finetune_xe_epochs=10 \
optimizer.finetune_sc_epochs=10 \
optimizer.freeze_backbone=True \
optimizer.freeze_detector=True