Confirmation of the results

[Lynne-Zheng-Linfang]

Hi Xiaolong,

I have tried to run the evaluation code by using the provided dataset. I followed the instructions, corrected some bugs though. The results shown on my server are as follows:

(articulated-pose) user@lambda-dual:~/Documents/Code/articulated-pose$ cd evaluation/
(articulated-pose) user@lambda-dual:~/Documents/Code/articulated-pose/evaluation$ python eval_pose_err.py --item='eyeglasses' --domain='unseen' --nocs='ANCSH'

/home/user/Documents/Code/articulated-pose/results/pickle/3.9/unseen_ANCSH_eyeglasses_rt.pkl
number of data for pn_gt : 2730
/home/user/Documents/Code/articulated-pose/results/pickle/3.9/unseen_NAOCS_eyeglasses_rt.pkl
number of data for gn_gt : 2730
/home/usesr/Documents/Code/articulated-pose/results/pickle/3.9/unseen_ANCSH_eyeglasses_rt_pn.pkl
number of data for baseline : 544
For unseen object, ANCSH nocs, mean rotation err per part is: 
baseline 3.8505 8.3401 7.0143
nonlinea 128.0051 126.1274 130.9753

For unseen object, ANCSH nocs, mean translation err per part is: 
baseline 0.0450 0.0842 0.0758
nonlinea 1.1314 1.2538 1.3448


For unseen object, ANCSH nocs, 5 degrees accuracy per part is: 
baseline 0.9320 0.5460 0.5607
nonlinea 0.0129 0.0037 0.0018


For unseen object, ANCSH nocs, 5 degrees, 5 cms accuracy per part is: 
baseline 0.8989 0.3768 0.4007
nonlinea 0.0000 0.0000 0.0018


fetch rts data for baseline
fetch rts data for nonlinear
/home/user/Documents/Code/articulated-pose/evaluation/../lib/d3_utils.py:148: RuntimeWarning: invalid value encountered in arccos
  return np.arccos( ( np.trace(np.matmul(rot1, rot2.T)) - 1 ) / 2 ) % (2*np.pi)
For unseen object, ANCSH nocs, mean relative rotation err per part is: 
baseline 9.3495 8.3491
nonlinea 87.4471 127.2190

Are the results right? I think the baseline is for NPCS, and the nonlinear one is for ANCSH. Please correct me if I was wrong.

[dragonlong]

Hi Linfang,

The NPCS baseline result looks correct to me, but the nonlinear one doesn't. While the ransac-based nonlinear solver doesn't always guarantee a better estimation, it shouldn't be such worse. It might help if you could check whether some rotations are wrongly predicted to be 180 degrees flipped. Also keep in mind that the predicted joint, which is used during the nonlinear iteration, needs to be in the right direction.

Best,
Xiaolong

[Lynne-Zheng-Linfang]

Hi Linfang,

The NPCS baseline result looks correct to me, but the nonlinear one doesn't. While the ransac-based nonlinear solver doesn't always guarantee a better estimation, it shouldn't be such worse. It might help if you could check whether some rotations are wrongly predicted to be 180 degrees flipped. Also keep in mind that the predicted joint, which is used during the nonlinear iteration, needs to be in the right direction.

Best,
Xiaolong

Hi, Xiaolong,
Thanks for replying. I have checked with the code and set the USE_BASELINE to False. The results look similar to the baseline now. But I am not sure if USE_BASELINE should be set to False. Also, the results are still lower than the baseline. About the dealing of the direction of the predicted joints, I think it should have been done in the written code?

(articulated-pose) linfang@lambda-dual:~/Documents/Code/articulated-pose/evaluation$ python eval_pose_err.py --item='eyeglasses' --domain='unseen' --nocs='ANCSH'
/home/usr/Documents/Code/articulated-pose/results/pickle/3.9/unseen_ANCSH_eyeglasses_rt.pkl
number of data for pn_gt : 2730
/home/usr/Documents/Code/articulated-pose/results/pickle/3.9/unseen_NAOCS_eyeglasses_rt.pkl
number of data for gn_gt : 2730
/home/usr/Documents/Code/articulated-pose/results/pickle/3.9/unseen_ANCSH_eyeglasses_rt_pn.pkl
number of data for baseline : 544
For unseen object, ANCSH nocs, mean rotation err per part is: 
baseline 3.8505 8.3401 7.0143
nonlinea 12.7156 14.2523 14.5457


For unseen object, ANCSH nocs, mean translation err per part is: 
baseline 0.0450 0.0842 0.0758
nonlinea 0.1086 0.1582 0.1794


For unseen object, ANCSH nocs, 5 degrees accuracy per part is: 
baseline 0.9320 0.5460 0.5607
nonlinea 0.8435 0.8158 0.7937


For unseen object, ANCSH nocs, 5 degrees, 5 cms accuracy per part is: 
baseline 0.8989 0.3768 0.4007
nonlinea 0.8177 0.6298 0.6077


fetch rts data for baseline
fetch rts data for nonlinear
For unseen object, ANCSH nocs, mean relative rotation err per part is: 
baseline 9.3327 8.3277
nonlinea 12.3156 14.5273

[Jianghanxiao]

Agree with that there may be some typo. Currently if USE_BASELINE == True, then both baseline and nonlinear uses the nocs_pred from baseline prediction. It doesn't make sense

[Lynne-Zheng-Linfang]

Hi Linfang,

The NPCS baseline result looks correct to me, but the nonlinear one doesn't. While the ransac-based nonlinear solver doesn't always guarantee a better estimation, it shouldn't be such worse. It might help if you could check whether some rotations are wrongly predicted to be 180 degrees flipped. Also keep in mind that the predicted joint, which is used during the nonlinear iteration, needs to be in the right direction.

Best,
Xiaolong

Hi Xiaolong,

I have tried to debug the code by myself but the results seems not correct. Could you please update the code which works for you so that we can reproduce the same results? It should be as easy as a simple git push command. Thanks in advance.

[dragonlong]

Hi Linfang,

I found there is actually a step missing in the cleaned ransac function, after each iteration, the best score should be updated as well, or this will result in degraded performance. Please feel free to check and test it. Thanks!

Best,
Xiaolong