Skip to content

Commit a08666b

Browse files
carlos-zamoraDHowett
authored andcommitted
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915 Let's start with the easy change: - `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl` now. Ok. Now we've got a story to tell here.... ### TermControlAP - the Automation Peer Here's an in-depth guide on custom automation peers: https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers We have a custom XAML element (TermControl). So XAML can't really hold our hands and determine an accessible behavior for us. So this automation peer is responsible for enabling that interaction. We made it a FrameworkElementAutomationPeer to get as much accessibility as possible from it just being a XAML element (i.e.: where are we on the screen? what are my dimensions?). This is recommended. Any functions with "Core" at the end, are overwritten here to tweak this automation peer into what we really need. But what kind of interactions can a user expect from this XAML element? Introducing ControlPatterns! There's a ton of interfaces that just define "what can I do". Thankfully, we already know that we're supposed to be `ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the TermControlAP an `ITextProvider` too. So now we have a way to define what accessible actions can be performed on us, but what should those actions do? Well let's just use the automation providers from ConHost that are now in a shared space! (Note: this is a great place to stop and get some coffee. We're about to hop into the .cpp file in the next section) ### Wrapping our shared Automation Providers Unfortunately, we can't just use the automation providers from ConHost. Or, at least not just hook them up as easily as we wish. ConHost's UIA Providers were written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces ITextProvider and ITextRangeProvider are lined up to be exactly the same. So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML ones. We had two providers, so that means we have two wrappers. #### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore) Each of the functions in the pragma region `ITextProvider` for TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and returning an acceptable version of it. Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need to wrap that too. That's this next section... #### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore) Same idea. We're wrapping everything that we could do with `UiaTextRange` and putting it inside of `XamlUiaTextRange`. ### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider` If you don't know what I just said, please read this background: - #1691: how accessibility works and the general responsibility of these two classes - #1915: how we pulled these Accessibility Providers into a shared area TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text. `UiaTextRange` is specific ranges of text in the display and navigate the text. Thankfully, we didn't do many changes here. I feel like some of it is hacked together but now that we have a somewhat working system, making changes shouldn't be too hard...I hope. #### UiaTextRange We don't have access to the window handle. We really only need it to draw the bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I need to figure out how to get around this. In the meantime, I made the window handle optional. And if we don't have one....well, we need to figure that out. But other than that, we have a `UiaTextRange`. #### ScreenInfoUiaProvider At some point, we need to hook up this automation provider to the WindowUiaProvider. This should help with navigation of the UIA Tree and make everything just look waaaay better. For now, let's just do the same approach and make the pUiaParent optional. This one's the one I'm not that proud of, but it works. We need the parent to get a bounding rect of the terminal. While we figure out how to attach the WindowUiaProvider, we should at the very least be able to get a bunch of info from our xaml automation peer. So, I've added a _getBoundingRect optional function. This is what's called when we don't have a WindowUiaProvider as our parent. ## Validation Steps Performed I've been using inspect.exe to see the UIA tree. I was able to interact with the terminal mostly fine. A few known issues below. Unfortunately, I tried running Narrator on this and it didn't seem to like it (by that I mean WT crashed). Then again, I don't really know how to use narrator other than "click on object" --> "listen voice". I feel like there's a way to get the other interactions with narrator, but I'll be looking into more of that soon. I bet if I fix the two issues below, Narrator will be happy. ## Miscellaneous Known Issues - `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through these. I want to include them in this PR. Fixes #1353.
1 parent 1afab78 commit a08666b

20 files changed

+811
-86
lines changed

src/cascadia/TerminalApp/App.cpp

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -881,7 +881,7 @@ namespace winrt::TerminalApp::implementation
881881
_UpdateTitle(tab);
882882
});
883883

884-
term.GetControl().GotFocus([this, weakTabPtr](auto&&, auto&&) {
884+
term.GotFocus([this, weakTabPtr](auto&&, auto&&) {
885885
auto tab = weakTabPtr.lock();
886886
if (!tab)
887887
{

src/cascadia/TerminalApp/Pane.cpp

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ Pane::Pane(const GUID& profile, const TermControl& control, const bool lastFocus
1919
_lastFocused{ lastFocused },
2020
_profile{ profile }
2121
{
22-
_root.Children().Append(_control.GetControl());
22+
_root.Children().Append(_control);
2323
_connectionClosedToken = _control.ConnectionClosed({ this, &Pane::_ControlClosedHandler });
2424

2525
// Set the background of the pane to match that of the theme's default grid
@@ -426,7 +426,7 @@ bool Pane::_HasFocusedChild() const noexcept
426426
// We're intentionally making this one giant expression, so the compiler
427427
// will skip the following lookups if one of the lookups before it returns
428428
// true
429-
return (_control && _control.GetControl().FocusState() != FocusState::Unfocused) ||
429+
return (_control && _control.FocusState() != FocusState::Unfocused) ||
430430
(_firstChild && _firstChild->_HasFocusedChild()) ||
431431
(_secondChild && _secondChild->_HasFocusedChild());
432432
}
@@ -445,7 +445,7 @@ void Pane::UpdateFocus()
445445
if (_IsLeaf())
446446
{
447447
const auto controlFocused = _control &&
448-
_control.GetControl().FocusState() != FocusState::Unfocused;
448+
_control.FocusState() != FocusState::Unfocused;
449449

450450
_lastFocused = controlFocused;
451451
}
@@ -468,7 +468,7 @@ void Pane::_FocusFirstChild()
468468
{
469469
if (_IsLeaf())
470470
{
471-
_control.GetControl().Focus(FocusState::Programmatic);
471+
_control.Focus(FocusState::Programmatic);
472472
}
473473
else
474474
{
@@ -564,11 +564,11 @@ void Pane::_CloseChild(const bool closeFirst)
564564
_separatorRoot = { nullptr };
565565

566566
// Reattach the TermControl to our grid.
567-
_root.Children().Append(_control.GetControl());
567+
_root.Children().Append(_control);
568568

569569
if (_lastFocused)
570570
{
571-
_control.GetControl().Focus(FocusState::Programmatic);
571+
_control.Focus(FocusState::Programmatic);
572572
}
573573

574574
_splitState = SplitState::None;

src/cascadia/TerminalApp/Tab.cpp

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,7 @@ void Tab::_Focus()
124124
auto lastFocusedControl = _rootPane->GetFocusedTerminalControl();
125125
if (lastFocusedControl)
126126
{
127-
lastFocusedControl.GetControl().Focus(FocusState::Programmatic);
127+
lastFocusedControl.Focus(FocusState::Programmatic);
128128
}
129129
}
130130

@@ -181,7 +181,7 @@ void Tab::SetTabText(const winrt::hstring& text)
181181
void Tab::Scroll(const int delta)
182182
{
183183
auto control = GetFocusedTerminalControl();
184-
control.GetControl().Dispatcher().RunAsync(CoreDispatcherPriority::Normal, [control, delta]() {
184+
control.Dispatcher().RunAsync(CoreDispatcherPriority::Normal, [control, delta]() {
185185
const auto currentOffset = control.GetScrollOffset();
186186
control.KeyboardScrollViewport(currentOffset + delta);
187187
});

src/cascadia/TerminalControl/TermControl.cpp

Lines changed: 18 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
#include "..\..\types\inc\GlyphWidth.hpp"
1212

1313
#include "TermControl.g.cpp"
14+
#include "TermControlAutomationPeer.h"
1415

1516
using namespace ::Microsoft::Console::Types;
1617
using namespace ::Microsoft::Terminal::Core;
@@ -30,7 +31,6 @@ namespace winrt::Microsoft::Terminal::TerminalControl::implementation
3031
_connection{ connection },
3132
_initializedTerminal{ false },
3233
_root{ nullptr },
33-
_controlRoot{ nullptr },
3434
_swapChainPanel{ nullptr },
3535
_settings{ settings },
3636
_closing{ false },
@@ -52,11 +52,6 @@ namespace winrt::Microsoft::Terminal::TerminalControl::implementation
5252

5353
void TermControl::_Create()
5454
{
55-
// Create a dummy UserControl to use as the "root" of our control we'll
56-
// build manually.
57-
Controls::UserControl myControl;
58-
_controlRoot = myControl;
59-
6055
Controls::Grid container;
6156

6257
Controls::ColumnDefinition contentColumn{};
@@ -108,20 +103,20 @@ namespace winrt::Microsoft::Terminal::TerminalControl::implementation
108103
_bgImageLayer = bgImageLayer;
109104

110105
_swapChainPanel = swapChainPanel;
111-
_controlRoot.Content(_root);
106+
this->Content(_root);
112107

113108
_ApplyUISettings();
114109

115110
// These are important:
116111
// 1. When we get tapped, focus us
117-
_controlRoot.Tapped([this](auto&, auto& e) {
118-
_controlRoot.Focus(FocusState::Pointer);
112+
this->Tapped([this](auto&, auto& e) {
113+
this->Focus(FocusState::Pointer);
119114
e.Handled(true);
120115
});
121116
// 2. Make sure we can be focused (why this isn't `Focusable` I'll never know)
122-
_controlRoot.IsTabStop(true);
117+
this->IsTabStop(true);
123118
// 3. Actually not sure about this one. Maybe it isn't necessary either.
124-
_controlRoot.AllowFocusOnInteraction(true);
119+
this->AllowFocusOnInteraction(true);
125120

126121
// DON'T CALL _InitializeTerminal here - wait until the swap chain is loaded to do that.
127122

@@ -345,14 +340,16 @@ namespace winrt::Microsoft::Terminal::TerminalControl::implementation
345340
Close();
346341
}
347342

348-
UIElement TermControl::GetRoot()
343+
Windows::UI::Xaml::Automation::Peers::AutomationPeer TermControl::OnCreateAutomationPeer()
349344
{
350-
return _root;
345+
// create a custom automation peer with this code pattern:
346+
// (https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers)
347+
return winrt::make<winrt::Microsoft::Terminal::TerminalControl::implementation::TermControlAutomationPeer>(*this);
351348
}
352349

353-
Controls::UserControl TermControl::GetControl()
350+
::Microsoft::Console::Render::IRenderData* TermControl::GetRenderData() const
354351
{
355-
return _controlRoot;
352+
return _terminal.get();
356353
}
357354

358355
void TermControl::SwapChainChanged()
@@ -506,9 +503,9 @@ namespace winrt::Microsoft::Terminal::TerminalControl::implementation
506503
// through CharacterRecieved.
507504
// I don't believe there's a difference between KeyDown and
508505
// PreviewKeyDown for our purposes
509-
// These two handlers _must_ be on _controlRoot, not _root.
510-
_controlRoot.PreviewKeyDown({ this, &TermControl::_KeyDownHandler });
511-
_controlRoot.CharacterReceived({ this, &TermControl::_CharacterHandler });
506+
// These two handlers _must_ be on this, not _root.
507+
this->PreviewKeyDown({ this, &TermControl::_KeyDownHandler });
508+
this->CharacterReceived({ this, &TermControl::_CharacterHandler });
512509

513510
auto pfnTitleChanged = std::bind(&TermControl::_TerminalTitleChanged, this, std::placeholders::_1);
514511
_terminal->SetTitleChangedCallback(pfnTitleChanged);
@@ -542,14 +539,14 @@ namespace winrt::Microsoft::Terminal::TerminalControl::implementation
542539
// import value from WinUser (convert from milli-seconds to micro-seconds)
543540
_multiClickTimer = GetDoubleClickTime() * 1000;
544541

545-
_gotFocusRevoker = _controlRoot.GotFocus(winrt::auto_revoke, { this, &TermControl::_GotFocusHandler });
546-
_lostFocusRevoker = _controlRoot.LostFocus(winrt::auto_revoke, { this, &TermControl::_LostFocusHandler });
542+
_gotFocusRevoker = this->GotFocus(winrt::auto_revoke, { this, &TermControl::_GotFocusHandler });
543+
_lostFocusRevoker = this->LostFocus(winrt::auto_revoke, { this, &TermControl::_LostFocusHandler });
547544

548545
// Focus the control here. If we do it up above (in _Create_), then the
549546
// focus won't actually get passed to us. I believe this is because
550547
// we're not technically a part of the UI tree yet, so focusing us
551548
// becomes a no-op.
552-
_controlRoot.Focus(FocusState::Programmatic);
549+
this->Focus(FocusState::Programmatic);
553550

554551
_connection.Start();
555552
_initializedTerminal = true;

src/cascadia/TerminalControl/TermControl.h

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,6 @@ namespace winrt::Microsoft::Terminal::TerminalControl::implementation
3535
TermControl();
3636
TermControl(Settings::IControlSettings settings, TerminalConnection::ITerminalConnection connection);
3737

38-
Windows::UI::Xaml::UIElement GetRoot();
39-
Windows::UI::Xaml::Controls::UserControl GetControl();
4038
void UpdateSettings(Settings::IControlSettings newSettings);
4139

4240
hstring Title();
@@ -55,6 +53,9 @@ namespace winrt::Microsoft::Terminal::TerminalControl::implementation
5553
void SwapChainChanged();
5654
~TermControl();
5755

56+
Windows::UI::Xaml::Automation::Peers::AutomationPeer OnCreateAutomationPeer();
57+
::Microsoft::Console::Render::IRenderData* GetRenderData() const;
58+
5859
static Windows::Foundation::Point GetProposedDimensions(Microsoft::Terminal::Settings::IControlSettings const& settings, const uint32_t dpi);
5960

6061
// clang-format off
@@ -71,7 +72,6 @@ namespace winrt::Microsoft::Terminal::TerminalControl::implementation
7172
TerminalConnection::ITerminalConnection _connection;
7273
bool _initializedTerminal;
7374

74-
Windows::UI::Xaml::Controls::UserControl _controlRoot;
7575
Windows::UI::Xaml::Controls::Grid _root;
7676
Windows::UI::Xaml::Controls::Image _bgImageLayer;
7777
Windows::UI::Xaml::Controls::SwapChainPanel _swapChainPanel;

src/cascadia/TerminalControl/TermControl.idl

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,15 +14,13 @@ namespace Microsoft.Terminal.TerminalControl
1414
}
1515

1616
[default_interface]
17-
runtimeclass TermControl
17+
runtimeclass TermControl : Windows.UI.Xaml.Controls.UserControl
1818
{
1919
TermControl();
2020
TermControl(Microsoft.Terminal.Settings.IControlSettings settings, Microsoft.Terminal.TerminalConnection.ITerminalConnection connection);
2121

2222
static Windows.Foundation.Point GetProposedDimensions(Microsoft.Terminal.Settings.IControlSettings settings, UInt32 dpi);
2323

24-
Windows.UI.Xaml.UIElement GetRoot();
25-
Windows.UI.Xaml.Controls.UserControl GetControl();
2624
void UpdateSettings(Microsoft.Terminal.Settings.IControlSettings newSettings);
2725

2826
event TitleChangedEventArgs TitleChanged;
Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
// Copyright (c) Microsoft Corporation.
2+
// Licensed under the MIT license.
3+
4+
#include "pch.h"
5+
#include <UIAutomationCore.h>
6+
#include "TermControlAutomationPeer.h"
7+
#include "TermControl.h"
8+
#include "TermControlAutomationPeer.g.cpp"
9+
10+
#include "XamlUiaTextRange.h"
11+
12+
using namespace Microsoft::Console::Types;
13+
using namespace winrt::Windows::UI::Xaml::Automation::Peers;
14+
15+
namespace UIA
16+
{
17+
using ::ITextRangeProvider;
18+
using ::SupportedTextSelection;
19+
}
20+
21+
namespace XamlAutomation
22+
{
23+
using winrt::Windows::UI::Xaml::Automation::SupportedTextSelection;
24+
using winrt::Windows::UI::Xaml::Automation::Provider::IRawElementProviderSimple;
25+
using winrt::Windows::UI::Xaml::Automation::Provider::ITextRangeProvider;
26+
}
27+
28+
namespace winrt::Microsoft::Terminal::TerminalControl::implementation
29+
{
30+
TermControlAutomationPeer::TermControlAutomationPeer(winrt::Microsoft::Terminal::TerminalControl::implementation::TermControl const& owner) :
31+
TermControlAutomationPeerT<TermControlAutomationPeer>(owner), // pass owner to FrameworkElementAutomationPeer
32+
_uiaProvider{ owner.GetRenderData(), nullptr, std::bind(&TermControlAutomationPeer::GetBoundingRectWrapped, this) } {};
33+
34+
winrt::hstring TermControlAutomationPeer::GetClassNameCore() const
35+
{
36+
return L"TermControl";
37+
}
38+
39+
AutomationControlType TermControlAutomationPeer::GetAutomationControlTypeCore() const
40+
{
41+
return AutomationControlType::Text;
42+
}
43+
44+
winrt::hstring TermControlAutomationPeer::GetLocalizedControlTypeCore() const
45+
{
46+
// TODO GitHub #2142: Localize string
47+
return L"TerminalControl";
48+
}
49+
50+
winrt::Windows::Foundation::IInspectable TermControlAutomationPeer::GetPatternCore(PatternInterface patternInterface) const
51+
{
52+
switch (patternInterface)
53+
{
54+
case PatternInterface::Text:
55+
return *this;
56+
break;
57+
default:
58+
return nullptr;
59+
}
60+
}
61+
62+
#pragma region ITextProvider
63+
winrt::com_array<XamlAutomation::ITextRangeProvider> TermControlAutomationPeer::GetSelection()
64+
{
65+
SAFEARRAY* pReturnVal;
66+
THROW_IF_FAILED(_uiaProvider.GetSelection(&pReturnVal));
67+
return WrapArrayOfTextRangeProviders(pReturnVal);
68+
}
69+
70+
winrt::com_array<XamlAutomation::ITextRangeProvider> TermControlAutomationPeer::GetVisibleRanges()
71+
{
72+
SAFEARRAY* pReturnVal;
73+
THROW_IF_FAILED(_uiaProvider.GetVisibleRanges(&pReturnVal));
74+
return WrapArrayOfTextRangeProviders(pReturnVal);
75+
}
76+
77+
XamlAutomation::ITextRangeProvider TermControlAutomationPeer::RangeFromChild(XamlAutomation::IRawElementProviderSimple childElement)
78+
{
79+
UIA::ITextRangeProvider* returnVal;
80+
// ScreenInfoUiaProvider doesn't actually use parameter, so just pass in nullptr
81+
THROW_IF_FAILED(_uiaProvider.RangeFromChild(/* IRawElementProviderSimple */ nullptr,
82+
&returnVal));
83+
84+
auto parentProvider = this->ProviderFromPeer(*this);
85+
auto xutr = winrt::make_self<XamlUiaTextRange>(returnVal, parentProvider);
86+
return xutr.as<XamlAutomation::ITextRangeProvider>();
87+
}
88+
89+
XamlAutomation::ITextRangeProvider TermControlAutomationPeer::RangeFromPoint(Windows::Foundation::Point screenLocation)
90+
{
91+
UIA::ITextRangeProvider* returnVal;
92+
THROW_IF_FAILED(_uiaProvider.RangeFromPoint({ screenLocation.X, screenLocation.Y }, &returnVal));
93+
94+
auto parentProvider = this->ProviderFromPeer(*this);
95+
auto xutr = winrt::make_self<XamlUiaTextRange>(returnVal, parentProvider);
96+
return xutr.as<XamlAutomation::ITextRangeProvider>();
97+
}
98+
99+
XamlAutomation::ITextRangeProvider TermControlAutomationPeer::DocumentRange()
100+
{
101+
UIA::ITextRangeProvider* returnVal;
102+
THROW_IF_FAILED(_uiaProvider.get_DocumentRange(&returnVal));
103+
104+
auto parentProvider = this->ProviderFromPeer(*this);
105+
auto xutr = winrt::make_self<XamlUiaTextRange>(returnVal, parentProvider);
106+
return xutr.as<XamlAutomation::ITextRangeProvider>();
107+
}
108+
109+
Windows::UI::Xaml::Automation::SupportedTextSelection TermControlAutomationPeer::SupportedTextSelection()
110+
{
111+
UIA::SupportedTextSelection returnVal;
112+
THROW_IF_FAILED(_uiaProvider.get_SupportedTextSelection(&returnVal));
113+
return static_cast<XamlAutomation::SupportedTextSelection>(returnVal);
114+
}
115+
116+
#pragma endregion
117+
118+
RECT TermControlAutomationPeer::GetBoundingRectWrapped()
119+
{
120+
auto rect = GetBoundingRectangle();
121+
return {
122+
gsl::narrow<LONG>(rect.X),
123+
gsl::narrow<LONG>(rect.Y),
124+
gsl::narrow<LONG>(rect.X + rect.Width),
125+
gsl::narrow<LONG>(rect.Y + rect.Height)
126+
};
127+
}
128+
129+
// Method Description:
130+
// - extracts the UiaTextRanges from the SAFEARRAY and converts them to Xaml ITextRangeProviders
131+
// Arguments:
132+
// - SAFEARRAY of UIA::UiaTextRange (ITextRangeProviders)
133+
// Return Value:
134+
// - com_array of Xaml Wrapped UiaTextRange (ITextRangeProviders)
135+
winrt::com_array<XamlAutomation::ITextRangeProvider> TermControlAutomationPeer::WrapArrayOfTextRangeProviders(SAFEARRAY* textRanges)
136+
{
137+
// transfer ownership of UiaTextRanges to this new vector
138+
auto providers = SafeArrayToOwningVector<::Microsoft::Console::Types::UiaTextRange>(textRanges);
139+
int count = providers.size();
140+
141+
std::vector<XamlAutomation::ITextRangeProvider> vec;
142+
vec.reserve(count);
143+
auto parentProvider = this->ProviderFromPeer(*this);
144+
for (int i = 0; i < count; i++)
145+
{
146+
auto xutr = winrt::make_self<XamlUiaTextRange>(providers[i].detach(), parentProvider);
147+
vec.emplace_back(xutr.as<XamlAutomation::ITextRangeProvider>());
148+
}
149+
150+
winrt::com_array<XamlAutomation::ITextRangeProvider> result{ vec };
151+
152+
return result;
153+
}
154+
}

0 commit comments

Comments
 (0)