Skip to content

test(e2e): Adds Feedback Widget Capture Screenshot Maestro E2E tests #4795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 261 commits into
base: main
Choose a base branch
from

Conversation

antonis
Copy link
Collaborator

@antonis antonis commented May 1, 2025

📢 Type of change

  • Bugfix
  • New feature
  • Enhancement
  • Refactoring

Based on #4604

📜 Description

Adds Feedback Widget Capture Screenshot tests

💡 Motivation and Context

Part of #4302

💚 How did you test it?

CI (example of submitted feedback via the CI, Local:

Screen.Recording.2025-05-01.at.8.20.15.AM.mov

📝 Checklist

  • I added tests to verify changes
  • No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled
  • I updated the docs if needed.
  • I updated the wizard if needed.
  • All tests passing
  • No breaking changes

🔮 Next steps

#skip-changelog

Copy link
Contributor

github-actions bot commented May 1, 2025

Android (legacy) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 423.94 ms 450.56 ms 26.62 ms
Size 17.75 MiB 20.15 MiB 2.40 MiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
e754436 472.72 ms 480.96 ms 8.23 ms
7f6950b 438.74 ms 430.71 ms -8.03 ms
148f924 492.65 ms 500.28 ms 7.63 ms
7301787 412.13 ms 405.46 ms -6.67 ms
60d7316 418.46 ms 414.12 ms -4.34 ms
1c9e040 325.02 ms 311.57 ms -13.45 ms
86818c7 444.76 ms 465.25 ms 20.49 ms
d8668ce 452.13 ms 435.06 ms -17.07 ms
87d396c 463.52 ms 500.31 ms 36.79 ms
73fc872 468.12 ms 483.07 ms 14.95 ms

App size

Revision Plain With Sentry Diff
e754436 17.75 MiB 20.13 MiB 2.38 MiB
7f6950b 17.74 MiB 20.10 MiB 2.36 MiB
148f924 17.73 MiB 19.94 MiB 2.21 MiB
7301787 17.75 MiB 20.13 MiB 2.38 MiB
60d7316 17.75 MiB 20.11 MiB 2.36 MiB
1c9e040 17.75 MiB 20.11 MiB 2.37 MiB
86818c7 17.75 MiB 20.12 MiB 2.37 MiB
d8668ce 17.74 MiB 20.08 MiB 2.34 MiB
87d396c 17.75 MiB 20.13 MiB 2.38 MiB
73fc872 17.75 MiB 20.13 MiB 2.38 MiB

Copy link
Contributor

github-actions bot commented May 1, 2025

iOS (new) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 1232.76 ms 1232.47 ms -0.29 ms
Size 3.19 MiB 4.39 MiB 1.20 MiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
8f0282e+dirty 1225.33 ms 1229.92 ms 4.59 ms
f8fa2ad+dirty 1227.51 ms 1212.89 ms -14.62 ms
686b3bc+dirty 1219.65 ms 1231.73 ms 12.08 ms
d7401ac+dirty 1288.10 ms 1289.54 ms 1.44 ms
9de3d45+dirty 1228.37 ms 1231.79 ms 3.42 ms
148f924+dirty 1220.72 ms 1221.30 ms 0.58 ms
800171e+dirty 1222.94 ms 1235.82 ms 12.88 ms
a0b0298+dirty 1227.71 ms 1234.12 ms 6.41 ms
ce76a0b+dirty 1220.49 ms 1229.29 ms 8.80 ms
0d3e677+dirty 1239.02 ms 1241.22 ms 2.20 ms

App size

Revision Plain With Sentry Diff
8f0282e+dirty 3.19 MiB 4.33 MiB 1.14 MiB
f8fa2ad+dirty 3.19 MiB 4.35 MiB 1.16 MiB
686b3bc+dirty 3.19 MiB 4.31 MiB 1.12 MiB
d7401ac+dirty 2.92 MiB 3.40 MiB 488.06 KiB
9de3d45+dirty 3.19 MiB 4.33 MiB 1.14 MiB
148f924+dirty 2.92 MiB 3.60 MiB 701.88 KiB
800171e+dirty 3.19 MiB 4.25 MiB 1.06 MiB
a0b0298+dirty 3.19 MiB 4.32 MiB 1.13 MiB
ce76a0b+dirty 3.19 MiB 4.35 MiB 1.16 MiB
0d3e677+dirty 2.92 MiB 3.66 MiB 758.42 KiB

Copy link
Contributor

github-actions bot commented May 1, 2025

iOS (legacy) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 1216.15 ms 1222.94 ms 6.79 ms
Size 2.63 MiB 3.82 MiB 1.19 MiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
8f0282e+dirty 1215.46 ms 1219.64 ms 4.18 ms
f8fa2ad+dirty 1239.08 ms 1235.49 ms -3.59 ms
686b3bc+dirty 1210.89 ms 1221.14 ms 10.25 ms
d7401ac+dirty 1252.38 ms 1275.04 ms 22.66 ms
9de3d45+dirty 1229.16 ms 1229.58 ms 0.42 ms
148f924+dirty 1214.76 ms 1215.73 ms 0.97 ms
800171e+dirty 1229.96 ms 1232.87 ms 2.91 ms
a0b0298+dirty 1220.73 ms 1226.87 ms 6.14 ms
ce76a0b+dirty 1212.27 ms 1217.83 ms 5.56 ms
0d3e677+dirty 1214.39 ms 1225.70 ms 11.31 ms

App size

Revision Plain With Sentry Diff
8f0282e+dirty 2.63 MiB 3.76 MiB 1.13 MiB
f8fa2ad+dirty 2.63 MiB 3.78 MiB 1.14 MiB
686b3bc+dirty 2.63 MiB 3.74 MiB 1.11 MiB
d7401ac+dirty 2.36 MiB 2.83 MiB 481.14 KiB
9de3d45+dirty 2.63 MiB 3.77 MiB 1.13 MiB
148f924+dirty 2.36 MiB 3.04 MiB 696.25 KiB
800171e+dirty 2.63 MiB 3.69 MiB 1.05 MiB
a0b0298+dirty 2.63 MiB 3.75 MiB 1.12 MiB
ce76a0b+dirty 2.63 MiB 3.78 MiB 1.14 MiB
0d3e677+dirty 2.36 MiB 3.10 MiB 753.12 KiB

Copy link
Contributor

github-actions bot commented May 1, 2025

Android (new) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 389.34 ms 377.36 ms -11.98 ms
Size 7.15 MiB 8.42 MiB 1.26 MiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
76d1baf+dirty 339.02 ms 408.65 ms 69.63 ms
bd0ca38+dirty 272.43 ms 290.67 ms 18.24 ms
c830127+dirty 352.35 ms 388.96 ms 36.61 ms
86818c7+dirty 362.79 ms 376.04 ms 13.25 ms
d7401ac+dirty 373.98 ms 394.08 ms 20.10 ms
43e66e0+dirty 378.20 ms 404.59 ms 26.40 ms
e754436+dirty 424.60 ms 478.19 ms 53.59 ms
d8668ce+dirty 372.43 ms 403.84 ms 31.41 ms
834729f+dirty 413.31 ms 400.80 ms -12.52 ms
5446992+dirty 371.61 ms 390.00 ms 18.39 ms

App size

Revision Plain With Sentry Diff
76d1baf+dirty 7.15 MiB 8.09 MiB 964.41 KiB
bd0ca38+dirty 7.15 MiB 8.40 MiB 1.25 MiB
c830127+dirty 7.15 MiB 8.38 MiB 1.23 MiB
86818c7+dirty 7.15 MiB 8.39 MiB 1.23 MiB
d7401ac+dirty 7.15 MiB 8.04 MiB 910.85 KiB
43e66e0+dirty 7.15 MiB 8.37 MiB 1.22 MiB
e754436+dirty 7.15 MiB 8.40 MiB 1.25 MiB
d8668ce+dirty 7.15 MiB 8.35 MiB 1.20 MiB
834729f+dirty 7.15 MiB 8.40 MiB 1.25 MiB
5446992+dirty 7.15 MiB 8.12 MiB 999.45 KiB

@antonis
Copy link
Collaborator Author

antonis commented May 1, 2025

The Android app fails to open the widget on CI after the capture screenshot step. Leaving this PR in draft till the issue is investigated and fixed.

Maestro logs

08:43:06.268 [ INFO] maestro.cli.runner.TestSuiteInteractor.invoke: Input text [email protected] COMPLETED
08:43:06.268 [ INFO] maestro.cli.runner.TestSuiteInteractor.invoke: Tap on "What's the bug? What did you expect?" RUNNING
08:43:06.398 [ INFO] maestro.Maestro.tap-BUbHBYE: Tapping on element:  UiElement(treeNode=TreeNode(attributes={text=What's the bug? What did you expect?, accessibilityText=, hintText=What's the bug? What did you expect?, ignoreBoundsFiltering=false, resource-id=, clickable=true, bounds=[20,345][300,445], enabled=true, focused=false, checked=false, scrollable=false, selected=false, class=android.widget.EditText}, children=[], clickable=true, enabled=true, focused=false, checked=false, selected=false), bounds=Bounds(x=20, y=345, width=280, height=100))
08:43:07.287 [ INFO] maestro.Maestro.tap-BUbHBYE: Refreshed element
08:43:07.287 [ INFO] maestro.Maestro.hierarchyBasedTap-hbl3e4M: Tapping at (160, 395) using hierarchy based logic for wait
08:43:07.287 [TRACE] maestro.utils.ScreenshotUtils.takeScreenshot: Taking screenshot to byte array
08:43:07.287 [TRACE] maestro.utils.ScreenshotUtils.takeScreenshot: Taking screenshot to output sink
08:43:08.157 [ INFO] maestro.Maestro.hierarchyBasedTap-hbl3e4M: Something have changed in the UI judging by view hierarchy. Proceed.
08:43:08.158 [ INFO] maestro.cli.runner.TestSuiteInteractor.invoke: Tap on "What's the bug? What did you expect?" COMPLETED
08:43:08.158 [ INFO] maestro.cli.runner.TestSuiteInteractor.invoke: Input text This is a test feedback message with a screenshot from CI e2e tests RUNNING
08:43:08.158 [ INFO] maestro.Maestro.inputText: Inputting text: This is a test feedback message with a screenshot from CI e2e tests
08:43:15.437 [ INFO] maestro.cli.runner.TestSuiteInteractor.invoke: Input text This is a test feedback message with a screenshot from CI e2e tests COMPLETED
08:43:15.437 [ INFO] maestro.cli.runner.TestSuiteInteractor.invoke: Scrolling DOWN until "Take a screenshot" is visible. RUNNING
08:43:15.608 [ INFO] maestro.orchestra.Orchestra.scrollUntilVisible: Scrolling try count: 0, DeviceWidth: 320, DeviceWidth: 640
08:43:15.608 [ INFO] maestro.orchestra.Orchestra.scrollUntilVisible: Element bounds: Bounds(x=96, y=466, width=128, height=22)
08:43:15.608 [ INFO] maestro.orchestra.Orchestra.scrollUntilVisible: Visibility Percent: 0
08:43:15.609 [ INFO] maestro.orchestra.Orchestra.scrollUntilVisible: Command centerElement: ScrollUntilVisibleCommand(selector=ElementSelector(textRegex=Take a screenshot, idRegex=null, size=null, below=null, above=null, leftOf=null, rightOf=null, containsChild=null, containsDescendants=null, traits=null, index=null, enabled=null, optional=false, selected=null, checked=null, focused=null, childOf=null), direction=DOWN, scrollDuration=601, visibilityPercentage=100, timeout=20000, waitToSettleTimeoutMs=null, centerElement=false, label=null, optional=false).centerElement
08:43:15.609 [ INFO] maestro.orchestra.Orchestra.scrollUntilVisible: visibilityPercentageNormalized: 1.0
08:43:15.609 [ INFO] maestro.cli.runner.TestSuiteInteractor.invoke: Scrolling DOWN until "Take a screenshot" is visible. COMPLETED
08:43:15.609 [ INFO] maestro.cli.runner.TestSuiteInteractor.invoke: Tap on "Take a screenshot" RUNNING
08:43:15.696 [ INFO] maestro.Maestro.tap-BUbHBYE: Tapping on element:  UiElement(treeNode=TreeNode(attributes={text=Take a screenshot, accessibilityText=, hintText=, ignoreBoundsFiltering=false, resource-id=sentry-feedback-take-screenshot-button, clickable=false, bounds=[96,466][224,488], enabled=true, focused=false, checked=false, scrollable=false, selected=false, class=android.widget.TextView}, children=[], clickable=false, enabled=true, focused=false, checked=false, selected=false), bounds=Bounds(x=96, y=466, width=128, height=22))
08:43:16.603 [ INFO] maestro.Maestro.tap-BUbHBYE: Refreshed element
08:43:16.603 [ INFO] maestro.Maestro.hierarchyBasedTap-hbl3e4M: Tapping at (160, 477) using hierarchy based logic for wait
08:43:16.603 [TRACE] maestro.utils.ScreenshotUtils.takeScreenshot: Taking screenshot to byte array
08:43:16.603 [TRACE] maestro.utils.ScreenshotUtils.takeScreenshot: Taking screenshot to output sink
08:43:17.755 [ INFO] maestro.Maestro.hierarchyBasedTap-hbl3e4M: Something have changed in the UI judging by view hierarchy. Proceed.
08:43:17.755 [ INFO] maestro.cli.runner.TestSuiteInteractor.invoke: Tap on "Take a screenshot" COMPLETED
08:43:17.755 [ INFO] maestro.cli.runner.TestSuiteInteractor.invoke: Tap on "Take Screenshot" RUNNING
08:43:17.883 [ INFO] maestro.Maestro.tap-BUbHBYE: Tapping on element:  UiElement(treeNode=TreeNode(attributes={text=Take Screenshot, accessibilityText=, hintText=, ignoreBoundsFiltering=false, resource-id=sentry-feedback-screenshot-button, clickable=false, bounds=[140,569][274,594], enabled=true, focused=false, checked=false, scrollable=false, selected=false, class=android.widget.TextView}, children=[], clickable=false, enabled=true, focused=false, checked=false, selected=false), bounds=Bounds(x=140, y=569, width=134, height=25))
08:43:18.778 [ INFO] maestro.Maestro.tap-BUbHBYE: Refreshed element
08:43:18.779 [ INFO] maestro.Maestro.hierarchyBasedTap-hbl3e4M: Tapping at (207, 581) using hierarchy based logic for wait
08:43:18.779 [TRACE] maestro.utils.ScreenshotUtils.takeScreenshot: Taking screenshot to byte array
08:43:18.779 [TRACE] maestro.utils.ScreenshotUtils.takeScreenshot: Taking screenshot to output sink
08:43:20.000 [ INFO] maestro.Maestro.hierarchyBasedTap-hbl3e4M: Something have changed in the UI judging by view hierarchy. Proceed.
08:43:20.000 [ INFO] maestro.cli.runner.TestSuiteInteractor.invoke: Tap on "Take Screenshot" COMPLETED
08:43:20.000 [ INFO] maestro.cli.runner.TestSuiteInteractor.invoke: Assert that "Report a Bug" is visible RUNNING
08:43:25.267 [ INFO] maestro.cli.runner.TestSuiteInteractor.invoke: Assert that "Report a Bug" is visible FAILED
08:43:25.268 [ INFO] maestro.Maestro.takeScreenshot: Taking screenshot
08:43:25.268 [TRACE] maestro.utils.ScreenshotUtils.takeScreenshot: Taking screenshot to output sink
08:43:25.327 [ERROR] maestro.orchestra.Orchestra.executeCommands: [Command execution] CommandFailed: Assertion is false: "Report a Bug" is visible
08:43:25.327 [ INFO] maestro.cli.runner.TestSuiteInteractor.invoke: Run feedback/captureFlow-android.yml when Platform is ANDROID FAILED

timeout: 5_000

# Hide keyboard by tapping on a non-tappable element
- tapOn: 'Email'
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tapping on a non-tappable region is a work around to hide the keyboard since:

Base automatically changed from antonis/feedback-e2e to feedback-ui-2 May 9, 2025 07:22
# Conflicts:
#	dev-packages/e2e-tests/maestro/feedback.yml
#	dev-packages/e2e-tests/maestro/feedback/happyFlow-android.yml
#	dev-packages/e2e-tests/maestro/feedback/happyFlow-ios.yml
Base automatically changed from feedback-ui-2 to main May 20, 2025 14:10
Copy link
Collaborator Author

@antonis antonis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Android app fails to open the widget on CI after the capture screenshot step. Leaving this PR in draft till the issue is investigated and fixed.

Did a couple of attempts with higher timeouts but the issue persists indicating that the app hangs on CI after the screenshot is captured.

@@ -391,8 +391,7 @@ jobs:
force-avd-creation: false
disable-animations: true
disable-spellchecker: true
target: 'aosp_atd'
channel: canary # Necessary for ATDs
target: 'google_apis'
Copy link
Collaborator Author

@antonis antonis May 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Google image seem to provide a more complete system image support for graphics operations fixing the capture screenshot failure. The downside is that it is heavier for the CI operations. Comparing two runs on main vs this PR showed an average 44s bump in the test duration.

Test main this PR
RN 0.65.3 legacy jsc android production no 5m 32s 6m 16s
RN 0.79.1 legacy hermes android production no 5m 36s 6m 49s
RN 0.79.1 new hermes android production no 6m 53s 7m 7s

Note that the screenshots functionality failed with the AOSP image even for the screenshots taken by the Maestro environment in case of failure (E.g. see this log artifcact were the image is blank).

@antonis antonis marked this pull request as ready for review May 22, 2025 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants