Add support for spatial video metadata

[ladvoc]

To allow clients to properly handle spatial video (stereoscopic and/or spherical), this PR proposes adding a subset of the fields defined by the Spherical Video V2 RFC to TrackInfo.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 42d8e15

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

💥 An error occurred when fetching the changed packages and changesets in this PR

Some errors occurred when validating the changesets config:
The package or glob expression "github.com/livekit/protocol" specified in the `fixed` option does not match any package in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.

[lukasIO]

I think in theory for now we could comment out everything apart from stereo_mode ?

All fields make sense to me, I'm just not sure if defining all of them on the protocol level is the right choice.

Would love to hear some additional opinions from the rest of the team.

[lukasIO]

maybe let's go for a more generalVideoTrackFeatures ?

[bcherry]

I think it's a great idea to add something like this to the track. I think we should only add explicit detailed support for things we're actually planning to support at a framework level (e.g. the sdks + components libraries have special rendering support). i'm definitely in favor of a basic enum for very well understood formats that don't require extra metadata (e.g. stereo l/r) but the others i'm not so sure although i'm not an expert. I think that customers implementing more complex formats are also probably comfortable implementing their own rendering too, at least for the medium term. and i wouldn't want us to add a complex field that no one really uses because it's not quite comprehensive enough.

it would be great if track had a freeform attributes field or something but for now we put that on participant and i think its probably fine for a developer to use that to share detailed track rendering information across clients if needed.

wdyt @chenosaurus?

[lukasIO]

k. I think we should only add explicit detailed support for things we're actually planning to support at a framework level (e.g. the sdks + components libraries have special rendering support). i'm definitely in favor of a basic enum for very well understood formats that don't require extra metadata (e.g. stereo l/r) but the others i'm not so sure although i'm not an expert.

yeah, that's also what my feeling was, let's do stereo_mode for now and leave the decision for the rest for later / when users request it.

[chenosaurus]

I think it's a great idea to add something like this to the track. I think we should only add explicit detailed support for things we're actually planning to support at a framework level (e.g. the sdks + components libraries have special rendering support). i'm definitely in favor of a basic enum for very well understood formats that don't require extra metadata (e.g. stereo l/r) but the others i'm not so sure although i'm not an expert. I think that customers implementing more complex formats are also probably comfortable implementing their own rendering too, at least for the medium term. and i wouldn't want us to add a complex field that no one really uses because it's not quite comprehensive enough.

it would be great if track had a freeform attributes field or something but for now we put that on participant and i think its probably fine for a developer to use that to share detailed track rendering information across clients if needed.

wdyt @chenosaurus?

I agree, let's have only stereo_mode to start and add more of the parameters when we have a need for it.