Support .i3dm tile payloads

[argallegos]

Add support for Instanced 3D Models.

Once this is completed, some work will need to be done in Cesium Native integrations (e.g. Cesium for Unreal, Cesium for O3DE) before the feature is fully operational in each engine.

[argallegos]

Pull request addressing this issue: #290

[argallegos]

Cesium for Unreal issue: CesiumGS/cesium-unreal#671

Cesium for O3DE issue: CesiumGS/cesium-o3de#32

[lilleyse]

Useful diagram from CesiumGS/cesium#10302 showing the expected transform order between i3dm and gltf + EXT_mesh_gpu_instancing

[lilleyse]

See CesiumGS/3d-tiles-tools#52 for ideas for upgrading i3dm to EXT_mesh_gpu_instancing. It's not trivial because of transformation order differences (see diagram above).

CC @javagl

[javagl]

One of the insights from the linked PR was that it is sometimes plainly (mathematically) not possible to upgrade I3DM to EXT_mesh_gpu_instancing. An I3DM could contain an animated GLB that was non-uniformly scaled via the instancing transform. This can not be represented as a standard glTF animation.

It could be possible to lay out the map of "what is and what isn't possible" more strictly and systematically. But for the main case of I3DMs that contain "standard (non-animated) geometry", the approach from CesiumGS/3d-tiles-tools#52 (comment) turned out to be relatively easy, and could probably be applied in cesium-native as well, when there is a function to "bake" the transform of glTF nodes into the attached meshes.

[lilleyse]

It could make sense to implement this in three phases:

• Add support for EXT_mesh_gpu_instancing in cesium-native and runtimes
  • Unreal: #671
  • Unity: #416
  • Omniverse: #687
• Add support for EXT_instance_features in cesium-native and runtimes
• Transcode .i3dm to EXT_mesh_gpu_instancing in cesium-native (caveats discussed above)

Sample data:

• https://github.com/CesiumGS/3d-tiles-samples/tree/main/glTF/GpuInstancesMetadata
• https://github.com/KhronosGroup/glTF-Sample-Assets/tree/main/Models/SimpleInstancing
• https://github.com/CesiumGS/cesium/tree/main/Specs/Data/Cesium3DTiles/Instanced

CC @timoore @kring

[timoore]

One of the insights from the linked PR was that it is sometimes plainly (mathematically) not possible to upgrade I3DM to EXT_mesh_gpu_instancing. An I3DM could contain an animated GLB that was non-uniformly scaled via the instancing transform. This can not be represented as a standard glTF animation.

It seems to me that even relatively simple animations can't be directly supported directly by EXT_mesh_gpu_instancing, for example wind turbines with the blade rotating around its local axis. This may be less useful in practice than as a thought experiment.

It could be possible to lay out the map of "what is and what isn't possible" more strictly and systematically. But for the main case of I3DMs that contain "standard (non-animated) geometry", the approach from CesiumGS/3d-tiles-tools#52 (comment) turned out to be relatively easy, and could probably be applied in cesium-native as well, when there is a function to "bake" the transform of glTF nodes into the attached meshes.

Animations aren't supported by the native clients yet AFAIK, but if it became important one could imagine cesium-native creating "extended instances" from standard glTF. That is, it could identify regular nodes that reference the same mesh and create instances from them, including animated instance data.

[javagl]

This may be less useful in practice than as a thought experiment.

For CesiumGS/3d-tiles-tools#84 I actually considered to create a draft/proof of concept that visualizes actual wind power plants from one of the public databases, and ... then noticed that this may be an actual limitation. Not that having them animated would be "useful", but ... at this point, it's not about "useful", but about "looking cool" 😎

In terms of the phases that @lilleyse mentioned:

• Add support for EXT_mesh_gpu_instancing
• Transcode .i3dm to EXT_mesh_gpu_instancing

One concern could be: Iff the implementation is (too) focussed on the EXT_mesh_gpu_instancing extension, then there's the risk that it will "soak up" the limitations of this extension. I think that technically, having "individual(ly) animated instances that use GPU instancing" could be possible (and correct me if I'm wrong).

Ideally, the fact that it is not possible to represent such animations in the extension should not necessarily restrict the implementation. Very roughly speaking: The implementation could be "powerful enough" to represent such animations. When it is fed with EXT_mesh_gpu_instancing information, then this capabilitiy will not be used. If it is fed with I3DM (or some hypothetical EXT_mesh_gpu_instancing_with_animations data...), then it could use that information.

NOTE: This is really just brainstorming. I have no idea whether trying to support such animations may imply a prohibitively large implementation effort, or whether it would just boil down to passing in some optional animation data to an existing function...

[andek714]

Just a short comment about this: Our users would benefit a lot from i3dm support in Unreal, even without animation support. That would make trees, power line pylons, street lights etc. so much better in a terrain exported from Remoscape. I assume we're not the only ones using static non-animated i3dm either, so implementing it without animation first could be a good idea.