Transfer Learning

[jamesheatonrdm]

I have custom object classes defined in my own dataset which I am successfully enable to detect using pointpillars, however I wish to perform transfer learning as I would like to also detect classes in the KITTI dataset (Pedestrian, Car, etc.).

I would obviously rather not train on the combined dataset to save time, when multiple models trained on KITTI exist already and my custom dataset is relatively smaller.

I would like to know if anyone has performed transfer learning using this toolbox and how they performed it as I couldn't seem to find it in the documentation.

[MatthiasHinz]

Have you made any progress with this? Since there was no response so far, I will try to make some guesses from my observations. I am not sure if it will work, but it may be worth trying.

I think that you need to obtain a pre-trained Kitti-model first, e.g.:
https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pointpillars/README.md

Then in the runtime part of your config (similar to https://github.com/open-mmlab/mmdetection3d/blob/main/configs/_base_/default_runtime.py) you need to specify the path to this model within the load_from variable.

In the dataset part, you will probably need to specify the KITTI classes you want to recognize plus the classes from your dataset.

Then train with your data.

Please let me know if this was helpful. Feel free to correct me or add on.

[jamesheatonrdm]

Thanks, this makes sense. However I am struggling to see where the load_from variable is placed in the config file? In the file you linked to there doesn't appear to be an example of this.

[jamesheatonrdm]

Apologies, I see it at the bottom!

[jamesheatonrdm]

That appears to be training without errors, I'll let you know what the results are! Thanks again

[MatthiasHinz]

I am glad my comment reached you. I am looking forward to your update. :)

[MatthiasHinz]

Another way to achieve what you want may be initializing a model in your config with a checkpoint as follows:

model = dict(
    type='VoxelNet',
    init_cfg=dict(type="Pretrained", checkpoint="work_dirs/hv_second_secfpn_6x8_80e_kitti-3d-car_20200620_230238-393f000c.pth"),
    data_preprocessor=dict(
   ....

Even though this will run without errors, I am not sure where it makes sense. In the templates, the parameter init_cfg is only used in order to initialize backbone models, but technically, it should be possible to initialize almost every model like this, though the model parameters have to match with the checkpoint used, at least. Check the log and warnings carefully. I am also still experimenting with this.