Add Pico Passthrough

[Rongxuan2024]

I'm a newbie in XR development, if there is something to be corrected, please help me out XD

[mikeskydev]

Hi, thanks a lot for the passthrough PR! I'd planned to add it, but I hadn't looked at the relevant API.

As this is a PR to a branch that's not merged yet, would you mind if I took the specific changes we need for passthrough and add them directly to the current PR? I'll make sure the commits are set with you as the author. There's also some changes to the workflow needed as this PR stands, as you'll need to assume that the Pico SDK isn't in the project otherwise there will be compiler errors.

[Rongxuan2024]

Hi, thanks a lot for the passthrough PR! I'd planned to add it, but I hadn't looked at the relevant API.

As this is a PR to a branch that's not merged yet, would you mind if I took the specific changes we need for passthrough and add them directly to the current PR? I'll make sure the commits are set with you as the author. There's also some changes to the workflow needed as this PR stands, as you'll need to assume that the Pico SDK isn't in the project otherwise there will be compiler errors.

@merulaxr Sure, let's do it. In my code, the passthrough feature is realized with API related to the guardian boundary.
static void EnableSeeThroughManual (bool value)

[Rongxuan2024]

Hi, thanks a lot for the passthrough PR! I'd planned to add it, but I hadn't looked at the relevant API.

As this is a PR to a branch that's not merged yet, would you mind if I took the specific changes we need for passthrough and add them directly to the current PR? I'll make sure the commits are set with you as the author. There's also some changes to the workflow needed as this PR stands, as you'll need to assume that the Pico SDK isn't in the project otherwise there will be compiler errors.

@merulaxr
Plus, In my attempt, some of the brush shaders didn't work well with the video seethrough stream, probably we can work on it together.

[mikeskydev]

Great, thanks!

some of the brush shaders didn't work well with the video seethrough stream

Had literally just tested this myself, we had the same issue with Oculus passthrough and I'm trying to remember how it was solved. In Oculus, we set the passthrough to be an underlay layer but Pico doesn't use a Game Object, so I'm not sure how best to change that. It may also not be the solution that I'm remembering.

[Rongxuan2024]

Probably to change the shader blend mode or texture type of the brushes may help, I can't script shaders, so I'll try the texture later

[mikeskydev]

Ideally we shouldn't need to change the brushes, textures or shaders as we don't want to change how it looks between apps that use the toolkit, it should be something we tweak that's Pico related!

[Rongxuan2024]

Absolutely, I agree, keeping the brushes unchanged is good for the project‘s integrity as a whole. Maybe it's on the next update of Pico's VST configuration, hoping it's not too late haha (ToT)

[mikeage]

As this is a PR to a branch that's not merged yet, would you mind if I took the specific changes we need for passthrough and add them directly to the current PR? I'll make sure the commits are set with you as the author. There's also some changes to the workflow needed as this PR stands, as you'll need to assume that the Pico SDK isn't in the project otherwise there will be compiler errors.

You can also just change the target....

[Zomby2D]

In my attempt, some of the brush shaders didn't work well with the video seethrough stream

The author of this video compiled your PR and "fixed" the transparent brushes by making the passthrough darker.

[scottgwald]

@MuRongxuan1998 I don't have the build environment set up - is there any way you could send me an apk with Pico passthrough enabled? Also, are you working with Pico 3 or Pico 4? Shouldn't make a difference for passthrough, I don't think?!

[scottgwald]

@mikeskydev a kind comment that some people would be very appreciative of getting this merged - I'm eager to use it and I don't have my build tools set up at the moment. I really appreciate what you're doing by maintaining and enhancing this project. It's very powerful for the whole (open) ecosystem. Thank you for your time and effort.

[mikeskydev]

@scottgwald per the previous comments, I'm not going to merge an improper fix that compromises the look and feel of the app. We have recently got some leads on the issues facing the brushes though so hope to have them fixed properly soon!

[scottgwald]

Hope the recent leads will help get it across the finish line!

[Rongxuan2024]

@MuRongxuan1998 I don't have the build environment set up - is there any way you could send me an apk with Pico passthrough enabled? Also, are you working with Pico 3 or Pico 4? Shouldn't make a difference for passthrough, I don't think?!

Hi, you can get a better version via the link below this video, and PICO 3 & 4 share the same API for passthrough. Sorry for the late reply :)

[wei-kris]

this video

It looks good, but do you know what he did to achieve it?

[andybak]

It looks good, but do you know what he did to achieve it?

He implies he just compiled this PR.

[mikeskydev]

Thanks for the submission, now superseded by the work in #798 directly with OpenXR and composition layers!