Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement iOS VoiceOver support #18016

Open
wants to merge 35 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
863e06c
Implement iOS VoiceOver support
IsaMorphic Jan 21, 2025
709c212
Merge branch 'master' into feature/ios-voiceover
IsaMorphic Jan 21, 2025
e632ea7
Some investigation (it is not working)
Jan 21, 2025
87376b3
Progress
Jan 21, 2025
230a542
Functional prototype for iOS VoiceOver support
IsaMorphic Jan 22, 2025
ddff15a
Improved VoiceOver control function
Jan 22, 2025
bf9933d
Implement traits and accessibility actions for VoiceOver
IsaMorphic Jan 23, 2025
f9ed243
Merge branch 'master' into feature/ios-voiceover
IsaMorphic Jan 23, 2025
9287158
Remove unused using statement
IsaMorphic Jan 23, 2025
a3df775
Default IsOffscreenBehavior should be based on IsEffectivelyVisible
IsaMorphic Jan 23, 2025
5aa082c
Use most specific activation function first
IsaMorphic Jan 24, 2025
b0ac5ab
Merge branch 'master' into feature/ios-voiceover
IsaMorphic Jan 24, 2025
d627697
Ensure that offscreen UIAccessibilityElements are removed
IsaMorphic Jan 24, 2025
1abc92b
Fix interrupted speech upon focusing element
IsaMorphic Jan 24, 2025
093cfbd
Accessibility improvements based on tests with low vision users
IsaMorphic Jan 25, 2025
9b28c9c
More improvements to iOS VoiceOver access
Jan 26, 2025
ccdf460
Final fixes for iOS VoiceOver
Jan 26, 2025
e4af6d2
Merge branch 'master' into feature/ios-voiceover
IsaMorphic Feb 1, 2025
c39f2c0
Merge branch 'master' into feature/ios-voiceover
IsaMorphic Feb 5, 2025
173fcab
Merge branch 'master' into feature/ios-voiceover
IsaMorphic Feb 6, 2025
8327938
Merge branch 'master' into feature/ios-voiceover
IsaMorphic Feb 9, 2025
58bd060
Merge branch 'master' into feature/ios-voiceover
IsaMorphic Feb 11, 2025
3cffb12
Add informative comments
IsaMorphic Feb 12, 2025
ae28f62
Merge branch 'master' into feature/ios-voiceover
IsaMorphic Feb 15, 2025
e6fb5e0
Merge branch 'master' into feature/ios-voiceover
IsaMorphic Feb 17, 2025
b11b5af
Implement review suggestions for iOS VoiceOver
IsaMorphic Feb 17, 2025
2fb061f
Fixed bugs in determining whether controls are offscreen
IsaMorphic Feb 17, 2025
1e33213
Ensure that AutomationPeer children are updated
IsaMorphic Feb 17, 2025
0b12e0f
Fix failing tests for IsEffectivelyVisible & CompositorHitTesting
IsaMorphic Feb 18, 2025
0abc5b5
Merge branch 'master' into feature/ios-voiceover
IsaMorphic Feb 21, 2025
6d37a96
Merge branch 'master' into feature/ios-voiceover
IsaMorphic Feb 24, 2025
254313c
Merge branch 'master' into feature/ios-voiceover
IsaMorphic Feb 26, 2025
eccdba6
Merge branch 'master' into feature/ios-voiceover
IsaMorphic Mar 1, 2025
7783e26
Merge branch 'master' into feature/ios-voiceover
IsaMorphic Mar 3, 2025
7b65e56
Merge branch 'master' into feature/ios-voiceover
IsaMorphic Mar 4, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/Avalonia.Base/Visual.cs
Original file line number Diff line number Diff line change
Expand Up @@ -548,7 +548,7 @@ protected virtual void OnDetachedFromVisualTreeCore(VisualTreeAttachmentEventArg
}

DisableTransitions();
UpdateIsEffectivelyVisible(true);
UpdateIsEffectivelyVisible(false);
OnDetachedFromVisualTree(e);
DetachFromCompositor();

Expand Down
2 changes: 1 addition & 1 deletion src/Avalonia.Controls/Automation/IsOffscreenBehavior.cs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ namespace Avalonia.Automation
public enum IsOffscreenBehavior
{
/// <summary>
/// The AutomationProperty IsOffscreen is calculated based on IsVisible.
/// The AutomationProperty IsOffscreen is calculated based on IsEffectivelyVisible.
/// </summary>
Default,
/// <summary>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,20 @@ protected ContentControlAutomationPeer(ContentControl owner)

protected override string? GetNameCore()
{
var result = base.GetNameCore();

if (result is null && Owner.Presenter?.Child is TextBlock text)
{
result = text.Text;
}

if (result is null)
{
result = Owner.Content?.ToString();
}
Control? childControl = Owner.Presenter?.Child;
AutomationPeer? childPeer = childControl is null ? null :
CreatePeerForElement(childControl);
return base.GetNameCore() ?? (childControl as TextBlock)?.Text ??
childPeer?.GetName() ?? Owner.Content?.ToString();
}

return result;
protected override string? GetHelpTextCore()
{
Control? childControl = Owner.Presenter?.Child;
AutomationPeer? childPeer = childControl is null ? null :
CreatePeerForElement(childControl);
return base.GetHelpTextCore() ??
childPeer?.GetHelpText();
}

protected override bool IsContentElementCore() => false;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,7 @@ protected override bool IsOffscreenCore()
IsOffscreenBehavior.FromClip => Owner.GetTransformedBounds() is not { } bounds ||
MathUtilities.IsZero(bounds.Clip.Width) ||
MathUtilities.IsZero(bounds.Clip.Height),
_ => !Owner.IsVisible,
_ => !Owner.IsEffectivelyVisible,
Copy link
Member

@maxkatz6 maxkatz6 Feb 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @grokys, is it fine to switch to Effectively Visible? Or do other backends handle inherited visibility on their own?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other backends handle this only in-so-far as they handle the hit testing in Avalonia code. This is in contrast to iOS which is the only target platform that performs the hit testing on the OS side. Therefore, I argue that this adaptation is necessary.

};
}

Expand Down
284 changes: 284 additions & 0 deletions src/iOS/Avalonia.iOS/AutomationPeerWrapper.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,284 @@
using System;
using System.Collections.Generic;
using System.Linq;
using Avalonia.Automation;
using Avalonia.Automation.Peers;
using Avalonia.Automation.Provider;
using CoreGraphics;
using Foundation;
using UIKit;

namespace Avalonia.iOS
{
internal class AutomationPeerWrapper : UIAccessibilityElement
{
private static readonly IReadOnlyDictionary<AutomationProperty, Action<AutomationPeerWrapper>> s_propertySetters =
new Dictionary<AutomationProperty, Action<AutomationPeerWrapper>>()
{
{ AutomationElementIdentifiers.NameProperty, UpdateName },
{ AutomationElementIdentifiers.HelpTextProperty, UpdateHelpText },
{ AutomationElementIdentifiers.BoundingRectangleProperty, UpdateBoundingRectangle },

{ RangeValuePatternIdentifiers.IsReadOnlyProperty, UpdateIsReadOnly },
{ RangeValuePatternIdentifiers.ValueProperty, UpdateValue },

{ ValuePatternIdentifiers.IsReadOnlyProperty, UpdateIsReadOnly },
{ ValuePatternIdentifiers.ValueProperty, UpdateValue },
};

private readonly AvaloniaView _view;
private readonly AutomationPeer _peer;

public AutomationPeerWrapper(AvaloniaView view, AutomationPeer? peer = null) : base(view)
{
_view = view;
_peer = peer ?? ControlAutomationPeer.CreatePeerForElement(view.TopLevel);

_peer.PropertyChanged += PeerPropertyChanged;
_peer.ChildrenChanged += PeerChildrenChanged;

AccessibilityContainer = _view;
AccessibilityIdentifier = _peer.GetAutomationId();
}

private static void UpdateName(AutomationPeerWrapper self)
{
AutomationPeer peer = self;
self.AccessibilityLabel = peer.GetName();
}

private static void UpdateHelpText(AutomationPeerWrapper self)
{
AutomationPeer peer = self;
self.AccessibilityHint = peer.GetHelpText();
}

private static void UpdateBoundingRectangle(AutomationPeerWrapper self)
{
AutomationPeer peer = self;
Rect bounds = peer.GetBoundingRectangle();
PixelRect screenRect = new PixelRect(
self._view.TopLevel.PointToScreen(bounds.TopLeft),
self._view.TopLevel.PointToScreen(bounds.BottomRight)
);
CGRect nativeRect = new CGRect(
screenRect.X, screenRect.Y,
screenRect.Width, screenRect.Height
);
if (self.AccessibilityFrame != nativeRect)
{
self.AccessibilityFrame = nativeRect;
UIAccessibility.PostNotification(UIAccessibilityPostNotification.LayoutChanged, null);
}
}

private static void UpdateIsReadOnly(AutomationPeerWrapper self)
{
AutomationPeer peer = self;
self.AccessibilityRespondsToUserInteraction =
peer.GetProvider<IValueProvider>()?.IsReadOnly ??
peer.GetProvider<IRangeValueProvider>()?.IsReadOnly ??
peer.IsEnabled();
}

private static void UpdateValue(AutomationPeerWrapper self)
{
AutomationPeer peer = self;
string? newValue =
peer.GetProvider<IRangeValueProvider>()?.Value.ToString("0.##") ??
peer.GetProvider<IValueProvider>()?.Value;
if (self.AccessibilityValue != newValue)
{
self.AccessibilityValue = newValue;
UIAccessibility.PostNotification(UIAccessibilityPostNotification.Announcement, null);
}
}

private void PeerChildrenChanged(object? sender, EventArgs e)
{
_view.UpdateChildren(_peer);
UIAccessibility.PostNotification(UIAccessibilityPostNotification.ScreenChanged, null);
}

private void PeerPropertyChanged(object? sender, AutomationPropertyChangedEventArgs e) =>
UpdateProperties(e.Property);

private void UpdateProperties(params AutomationProperty[] properties)
{
HashSet<Action<AutomationPeerWrapper>> calledSetters = new();
foreach (AutomationProperty property in properties)
{
if (s_propertySetters.TryGetValue(property,
out Action<AutomationPeerWrapper>? setter) &&
!calledSetters.Contains(setter))
{
calledSetters.Add(setter);
setter.Invoke(this);
}
}
}

public bool UpdatePropertiesIfValid()
{
if (_peer.IsContentElement() && !_peer.IsOffscreen())
{
UpdateProperties(s_propertySetters.Keys.ToArray());
return IsAccessibilityElement = true;
}
else
{
return IsAccessibilityElement = false;
}
}

public void UpdateTraits()
{
UIAccessibilityTrait traits = UIAccessibilityTrait.None;

switch (_peer.GetAutomationControlType())
{
case AutomationControlType.Button:
traits |= UIAccessibilityTrait.Button;
break;
case AutomationControlType.Header:
traits |= UIAccessibilityTrait.Header;
break;
case AutomationControlType.Hyperlink:
traits |= UIAccessibilityTrait.Link;
break;
case AutomationControlType.Image:
traits |= UIAccessibilityTrait.Image;
break;
}

if (_peer.GetProvider<IRangeValueProvider>()?.IsReadOnly == false)
{
traits |= UIAccessibilityTrait.Adjustable;
}

if (_peer.GetProvider<ISelectionItemProvider>()?.IsSelected == true)
{
traits |= UIAccessibilityTrait.Selected;
}

if (_peer.GetProvider<IValueProvider>()?.IsReadOnly == false)
{
traits |= UIAccessibilityTrait.UpdatesFrequently;
}

if (_peer.IsEnabled() == false)
{
traits |= UIAccessibilityTrait.NotEnabled;
}

AccessibilityTraits = (ulong)traits;
}

[Export("accessibilityActivate")]
public bool AccessibilityActivate()
{
IToggleProvider? toggleProvider = _peer.GetProvider<IToggleProvider>();
IInvokeProvider? invokeProvider = _peer.GetProvider<IInvokeProvider>();
if (toggleProvider is not null)
{
toggleProvider.Toggle();
return true;
}
else if (invokeProvider is not null)
{
invokeProvider.Invoke();
return true;
}
else
{
return false;
}
}

public override bool AccessibilityElementIsFocused()
{
base.AccessibilityElementIsFocused();
return _peer.HasKeyboardFocus();
}

public override void AccessibilityElementDidBecomeFocused()
{
base.AccessibilityElementDidBecomeFocused();
_peer.BringIntoView();
}

public override void AccessibilityDecrement()
{
base.AccessibilityDecrement();
IRangeValueProvider? provider = _peer.GetProvider<IRangeValueProvider>();
if (provider is not null)
{
double value = provider.Value;
provider.SetValue(value - provider.SmallChange);
}
}

public override void AccessibilityIncrement()
{
base.AccessibilityIncrement();
IRangeValueProvider? provider = _peer.GetProvider<IRangeValueProvider>();
if (provider is not null)
{
double value = provider.Value;
provider.SetValue(value + provider.SmallChange);
}
}

public override bool AccessibilityScroll(UIAccessibilityScrollDirection direction)
{
base.AccessibilityScroll(direction);
IScrollProvider? scrollProvider = _peer.GetProvider<IScrollProvider>();
if (scrollProvider is not null)
{
bool didScroll;
ScrollAmount verticalAmount, horizontalAmount;
switch (direction)
{
case UIAccessibilityScrollDirection.Up:
verticalAmount = ScrollAmount.SmallIncrement;
horizontalAmount = ScrollAmount.NoAmount;
didScroll = true;
break;
case UIAccessibilityScrollDirection.Down:
verticalAmount = ScrollAmount.SmallDecrement;
horizontalAmount = ScrollAmount.NoAmount;
didScroll = true;
break;
case UIAccessibilityScrollDirection.Left:
verticalAmount = ScrollAmount.NoAmount;
horizontalAmount = ScrollAmount.SmallIncrement;
didScroll = true;
break;
case UIAccessibilityScrollDirection.Right:
verticalAmount = ScrollAmount.NoAmount;
horizontalAmount = ScrollAmount.SmallDecrement;
didScroll = true;
break;
default:
verticalAmount = ScrollAmount.NoAmount;
horizontalAmount = ScrollAmount.NoAmount;
didScroll = false;
break;
}

scrollProvider.Scroll(verticalAmount, horizontalAmount);
if (didScroll)
{
UIAccessibility.PostNotification(UIAccessibilityPostNotification.PageScrolled, null);
return true;
}
}
return false;
}

public static implicit operator AutomationPeer(AutomationPeerWrapper instance)
{
return instance._peer;
}
}
}
Loading
Loading