MRTK3 NonNativeKeyboard & vanilla UGUI doesn't support near touch

[IssueSyncBot]

Original issue opened by:

@AMollis

---

Describe the bug

MRTK3 NonNativeKeyboard & vanilla UGUI doesn't support near touch. MRTK2's non native keyboard & vanilla UGUI supported near touch.

To reproduce

Steps to reproduce the behavior:

1. Open NonNativeKeyboard.scene
2. Click text box to open non-native keyboard
3. Try to enter text with near touch
4. Can't near touch the keyboard

Or

1. VanillaUGUIExample.scene
2. Try to interact with UGUI canvas via near touch
3. Near touch doesn't work

Expected behavior

Near touch to work on MRTK3's non-native keyboard & with vanilla UGUI, just like MRTK2

Screenshots

Your setup (please complete the following information)

• Unity Version 2021.3
• MRTK Version 3.0.0-pre

Target platform (please complete the following information)

• HoloLens 2

Additional context

Add any other context about the problem here.

---

ISSUE MIGRATION

Issue migrated from: microsoft/MixedRealityToolkit-Unity#11557

[IssueSyncBot]

Original comment by:

AMollis

---

To properly implement this we need to switch to using Unity XRI's XRPokeInteractor. The XRPokeInteractor implements IUIInteractor and registers itself with the scene's XRUIInputModule.

We're unable to implement and use IUIInteractor on MRTK's PokeInteractor. To correctly implement IUIInteractor, we need to be able to set TrackedDeviceModel.pokeDepth and TrackedDeviceModel.interactionType. However, interactionType and pokeDepth are internal fields and can't be accessed by MRTK.

[IssueSyncBot]

Original comment by:

AMollis

---

I attempted to update the MRTK3's PokeInteractor to derive from XRPokeInteractor. This almost work. However, MRTK3's InteractionModeHandler is getting in the way. The InteractionModeHandler is using MRTK3's IInteractionModeDetector to device what mode to change to (e.g. "near", "far", ect.). However, there is no detector for detecting if an interactor is near UGUI. This results in the PokeInteractor being disabled. If the PokeInteractor is disable, XRPokeInteractor can't do it's work to update UGUI interactions.

To properly use XRPokeInteractor, MRTK should consider switch to use XRI's XRInteractionGroup, instead of the IInteractionModeDetector method.

At this point, I don't believe there to be an easy way for MRTK3 to support UGUI with the poke interactor. Not until the MRTK3's default rig is modified to use XRI's 'XRPokeInteractorand XRInteractionGroup`.

As workaround, if someone really wanted to use UGUI, they'll have to add colliders and IXRInteractable to the UGUI components.

[IssueSyncBot]

Original comment by:

AMollis

---

Given the limitations here, the best option seems to be update MRTK3's rig to use the new XRI components. As such, this issue is depended upon the following:

• Update MRTK3 to use latest XRI components microsoft/MixedRealityToolkit-Unity#11679