[Bug] mAOE is little bad in mmdet3d 1.0.0rc5

[junjie18]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• I have read the FAQ documentation but cannot get the expected help.
• The bug has not been fixed in the latest version (dev) or latest version (1.x).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

master branch https://github.com/open-mmlab/mmdetection3d

Environment

pytorch==1.9.0+cu111
mmdet==2.24.0
mmdet3d==1.0.0rc5

Reproduces the problem - code sample

None

Reproduces the problem - command or script

bash tools/dist_train.sh configs/centerpoint/centerpoint_01voxel_second_secfpn_4x8_cyclic_20e_nus.py 8

Reproduces the problem - error message

I run the official centerpoint without fade training schedule, and the mAOE seems to be degrade under mmdet 1.0.0rc5. The pkl file is generated under mmdet3d 1.0.0 and there is no problem with the coordinates.

mAP: 0.5623
mATE: 0.3118
mASE: 0.3442
mAOE: 0.5803
mAVE: 0.2864
mAAE: 0.1838
NDS: 0.6105
Eval time: 113.9s

Per-class results:
Object Class    AP      ATE     ASE     AOE     AVE     AAE
car     0.844   0.185   0.153   0.113   0.295   0.193
truck   0.548   0.341   0.204   0.163   0.252   0.236
bus     0.654   0.370   0.308   0.327   0.492   0.268
trailer 0.340   0.580   0.339   0.711   0.223   0.110
construction_vehicle    0.185   0.674   0.535   1.180   0.133   0.296
pedestrian      0.833   0.169   0.276   0.406   0.232   0.097
motorcycle      0.550   0.210   0.511   1.065   0.467   0.255
bicycle 0.369   0.183   0.502   1.165   0.197   0.015
traffic_cone    0.654   0.169   0.333   nan     nan     nan
barrier 0.646   0.236   0.279   0.093   nan     nan

Additional information

No response

[Tai-Wang]

Please @JingweiZhang12 help reproduce this problem and fix it if necessary.

[JingweiZhang12]

@junjie18 Hi, I will check it.

[JingweiZhang12]

@junjie18 Hi, thanks for your feedback. I am a little confused about your comparison.
Is the mAOE 0.5803 above your evaluation results on mmdet3d v1.0.0rc5? We checked the training log and the mAOE is 0.3081 log, and the NDS 0.6105 above is lower than the performance of our released mode 0.6461.
Alternatively, according to I run the official centerpoint without fade training schedule. Did you mean you run this repo: https://github.com/tianweiy/CenterPoint/blob/master/configs/nusc/README.md? But, the fading training schedule, maybe it refers to cyclic learning schedule, is used on mmdet3d. Did you compare the performance under the same learning schedule?

[JingweiZhang12]

We ran the same command and got the following results:

mAP: 0.5616
mATE: 0.2985
mASE: 0.2535
mAOE: 0.3065
mAVE: 0.2970
mAAE: 0.1876
NDS: 0.6465
Eval time: 90.6s

Per-class results:
Object Class    AP      ATE     ASE     AOE     AVE     AAE
car     0.845   0.184   0.153   0.114   0.284   0.190
truck   0.540   0.331   0.183   0.110   0.244   0.235
bus     0.670   0.324   0.173   0.078   0.513   0.261
trailer 0.348   0.564   0.210   0.504   0.229   0.124
construction_vehicle    0.173   0.647   0.423   0.824   0.136   0.311
pedestrian      0.831   0.168   0.278   0.395   0.229   0.099
motorcycle      0.560   0.215   0.235   0.278   0.534   0.260
bicycle 0.358   0.171   0.269   0.379   0.207   0.021
traffic_cone    0.646   0.162   0.329   nan     nan     nan
barrier 0.645   0.221   0.282   0.077   nan     nan

[junjie18]

@JingweiZhang12 Hi, thanks for your detailed reply. My official here means mmdet3d.
I check the log and find the orient_err of some classes(motorcycle, bycycle, trailer) is abnormal. What will make this difference in your opinion? thx.

[lucifer443]

I encountered the same problem too.

[JingweiZhang12]

@junjie18 Hi, there are some tips:

1. You can test the checkpoint we provide to make sure the inference is normal.
2. I noticed that the loss of your latest iteration in training is about 4.5, but ours is 4.0~4.1.

I provide our processed annotation file:
link: https://pan.baidu.com/s/1lUX4aPsYbcG9w40i1zroew?pwd=nuki code: nuki

You can use these annotations to retrain the model and evaluate it. Wait for your new results and we'll keep an eye on this issue.

[junjie18]

@JingweiZhang12 Hi,
1、I download the checkpoint, evaluate and get the corresponding mAP/NDS 56.4/64.8.
2、I have checked your pkl file with mine, everything works well.

import mmcv
import numpy as np

pkl1 = mmcv.load('path to your train pkl')
pkl2 = mmcv.load('path to mine')

assert len(pkl1['infos']) == len(pkl2['infos'])
total_num = len(pkl1['infos'])

for idx in range(total_num):
    info1 = pkl1['infos'][idx]
    info2 = pkl2['infos'][idx]
    for key in info1:
        if key in ['lidar_path', 'sweeps', 'cams']:
            continue
        if key in ['gt_boxes', 'gt_velocity']:
            assert ((np.nan_to_num(info1[key]) - np.nan_to_num(info2[key])) < 1e-6).all()
            continue
        if key in ['gt_names', 'num_lidar_pts', 'num_radar_pts', 'valid_flag']:
            assert (info1[key] == info2[key]).all()
            continue
        assert info1[key] == info2[key]

[JingweiZhang12]

I check the version of mmdet3d I used and its dependency.

TorchVision: 0.12.0+cu113
OpenCV: 4.6.0
MMCV: 1.7.1
MMCV Compiler: GCC 5.4
MMCV CUDA Compiler: 11.3
MMDetection: 2.26.0
MMSegmentation: 0.29.1
MMDetection3D: 1.0.0rc4+9556958
spconv2.0: False

Actually, the latest commit in my codebase is

commit 9556958fe1c6fe432d55a9f98781b8fdd90f4e9c
Author: lianqing01 <[email protected]>
Date:   Wed Aug 17 15:09:32 2022 +0800

    [Fix] Update the cn doc (#1715)

But these few differences should be not related to the performance of CenterPoint.
I suggest that you can try re-training the model on a version of mmdet3d 1.0.0rc4+9556958 using the ann file we provided.

[JingweiZhang12]

@junjie18 Hi, have you found the cause of the drop of performance about mAOE?

[junjie18]

@JingweiZhang12

I haven't verified more, I think it's mostly caused by my mistake in GT sample.