Issue training with DOCBANK COCO format annotations

[shivamsnaik]

Hello,
I am trying to use the X101 arch from the Model ZOO as a backbone for one of my experiments with the DOCBANK dataset.
I am using the COCO format provided for DOCBANK. However, I am getting really bad results on Inference.

Am I doing something wrong?

• Use pretrained weights for backbone.
• Freeze the top few layers and finetune the trainable layers along with the HEAD layers with a low LR.

Also, I would like to ask if the pretrained weights trained on the COCO based annotations or on the original token based annotations?.

Any help would be appreciated.