How can I get the labels for each segmentation result?

[Griseo-Kaslana]

Thank you for your excellent work!
I successfully ran the visualization demo with my own data (ReplicaCAD) using the following command:
CUDA_VISIBLE_DEVICES=0 python vis_demo/stream_demo.py --data_root ./data/office2 --config ./configs/ESAM-E_CA/ESAM-E_online_stream.py --checkpoint ./work_dirs/ESAM_online_3rscan_CA_test/ESAM_CA_online_epoch_128.pth --online_vis

However, I now have a question: How can I get the labels for each segmentation result? For example, identifying whether a segmented object is a "tomato" or an "apple." What should I do?

[xuxw98]

Hi,
Thanks for your interest. We are glad that you successfully run ESAM on your own data! Currently you use the _CA model. which means class-agnostic. So ESAM will only conduct online 3D instance segmentation without semantic label. To incorporate labels, you can use the provided model without _CA suffix, or train one by yourself.

Besides, if you want to perform open-vocabulary classification, you can refer to here. We describe how to combine ESAM with OpenMask3D for this purpose.