Gaussian Splat labelling workflow

[murlock1000]

It would be great for (research purposes and others) if this editor could enable users to visualize and edit labelled gaussian point clouds.

Such a feature would require:

• Creation or adoption of a standardized label data format for Gaussian Splatting data. Perhaps using an existing label format developed by segments.ai would be a good approach to avoid standard proliferation.
• CRUD operations on labeled Gaussian Splat data. To enable users to load, edit, update, apply, and save labels for a .ply file within the editor.
• Option to visualize gaussian cloud based on class id. Perhaps by including additional metadata of the class RGB that a point would be rendered as belonging to that class. This would need to be data-visualization friendly by perhaps recommending ordinal data color gradient choices.

I've started the development of such a feature in a fork: supersplat-classifier. Currently, it also includes hot-reloading of .ply files. Ideas and suggestions are welcome :)

[murlock1000]

Latest work, perhaps someone is interested in helping out!

• Used Segments.ai 3D point cloud Segmentation Label
• Hot reload of splat and label data from python notebook using WebSocket communications with custom message format.
• 'Show Classes' option to color the splat centers based on class color.
• Annotation manager to highlight, select and hide certain annotations (WIP).

export.mp4

[slimbuck]

OMG this is awesome!

[simonbethke]

I wonder if this is creeping features into something complicated that might be usable by very few only. I don't see how hot-loading and python notebooking has to be combined with labeling.
My experience when contributing was always that contributions could only be merged if they addressed a single feature and not a combination of multiple ones

[murlock1000]

@simonbethke while the hot-loading feature is very limited in it's use cases, a wider application would be the implementation of splat streaming to visualize a training process for example. But you are correct, maybe this specific PR should be concerned only with annotation support.

[simonbethke]

Yea, streaming a training progress would be cool.