Reduce Training time

[bkev13]

Hi, I installed mmdetection3d via docker to implement pointpillars for my project. The training runs smoothly except for one issue, I have a single gpu on ,y laptop for thr training time is estimated to a bit more than 5 days. Is there a way to cut down on training time? I had to reduce the batch size and samples per gpu to 1 for it to run (otherwise I get a a CUDA runtime error). So far I tried to reduce the number of epochs in the config file but that didn't affect the training time. I don't need the trained model to work with great accuracy, for now I want to lower the training time. Any tips?

[MatthiasHinz]

It would be helpful if you shared your model configuration and described the data you are using.

However, I figured out that voxel_size is an important parameter for balancing model accuracy vs. speed on point cloud datasets.
See https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/pure_point_cloud_dataset.html#adjust-voxel-size-in-config

If reducing the number of epochs did not affect the training time at all, you might have also made a mistake while doing so. If there is a RepeatDataSet definition in your train_dataloader, then you can reduce the number of iterations by decreasing the times parameter, e.g. times=1. Also, you may choose a shorter training schedule, e.g., use cyclic-20e if you used cyclic-40e until now.

You can also try increasing the number of workers if your system's resources are capable enough.