-
-
Notifications
You must be signed in to change notification settings - Fork 344
test(e2e): Adds Feedback Widget Capture Screenshot Maestro E2E tests #4795
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Co-authored-by: LucasZF <[email protected]>
# Conflicts: # dev-packages/e2e-tests/patch-scripts/rn.patch.app.js # packages/core/src/js/feedback/FeedbackWidgetProvider.tsx
Android (legacy) Performance metrics 🚀
|
Revision | Plain | With Sentry | Diff |
---|---|---|---|
e754436 | 472.72 ms | 480.96 ms | 8.23 ms |
7f6950b | 438.74 ms | 430.71 ms | -8.03 ms |
148f924 | 492.65 ms | 500.28 ms | 7.63 ms |
7301787 | 412.13 ms | 405.46 ms | -6.67 ms |
60d7316 | 418.46 ms | 414.12 ms | -4.34 ms |
1c9e040 | 325.02 ms | 311.57 ms | -13.45 ms |
86818c7 | 444.76 ms | 465.25 ms | 20.49 ms |
d8668ce | 452.13 ms | 435.06 ms | -17.07 ms |
87d396c | 463.52 ms | 500.31 ms | 36.79 ms |
73fc872 | 468.12 ms | 483.07 ms | 14.95 ms |
App size
Revision | Plain | With Sentry | Diff |
---|---|---|---|
e754436 | 17.75 MiB | 20.13 MiB | 2.38 MiB |
7f6950b | 17.74 MiB | 20.10 MiB | 2.36 MiB |
148f924 | 17.73 MiB | 19.94 MiB | 2.21 MiB |
7301787 | 17.75 MiB | 20.13 MiB | 2.38 MiB |
60d7316 | 17.75 MiB | 20.11 MiB | 2.36 MiB |
1c9e040 | 17.75 MiB | 20.11 MiB | 2.37 MiB |
86818c7 | 17.75 MiB | 20.12 MiB | 2.37 MiB |
d8668ce | 17.74 MiB | 20.08 MiB | 2.34 MiB |
87d396c | 17.75 MiB | 20.13 MiB | 2.38 MiB |
73fc872 | 17.75 MiB | 20.13 MiB | 2.38 MiB |
iOS (new) Performance metrics 🚀
|
Revision | Plain | With Sentry | Diff |
---|---|---|---|
8f0282e+dirty | 1225.33 ms | 1229.92 ms | 4.59 ms |
f8fa2ad+dirty | 1227.51 ms | 1212.89 ms | -14.62 ms |
686b3bc+dirty | 1219.65 ms | 1231.73 ms | 12.08 ms |
d7401ac+dirty | 1288.10 ms | 1289.54 ms | 1.44 ms |
9de3d45+dirty | 1228.37 ms | 1231.79 ms | 3.42 ms |
148f924+dirty | 1220.72 ms | 1221.30 ms | 0.58 ms |
800171e+dirty | 1222.94 ms | 1235.82 ms | 12.88 ms |
a0b0298+dirty | 1227.71 ms | 1234.12 ms | 6.41 ms |
ce76a0b+dirty | 1220.49 ms | 1229.29 ms | 8.80 ms |
0d3e677+dirty | 1239.02 ms | 1241.22 ms | 2.20 ms |
App size
Revision | Plain | With Sentry | Diff |
---|---|---|---|
8f0282e+dirty | 3.19 MiB | 4.33 MiB | 1.14 MiB |
f8fa2ad+dirty | 3.19 MiB | 4.35 MiB | 1.16 MiB |
686b3bc+dirty | 3.19 MiB | 4.31 MiB | 1.12 MiB |
d7401ac+dirty | 2.92 MiB | 3.40 MiB | 488.06 KiB |
9de3d45+dirty | 3.19 MiB | 4.33 MiB | 1.14 MiB |
148f924+dirty | 2.92 MiB | 3.60 MiB | 701.88 KiB |
800171e+dirty | 3.19 MiB | 4.25 MiB | 1.06 MiB |
a0b0298+dirty | 3.19 MiB | 4.32 MiB | 1.13 MiB |
ce76a0b+dirty | 3.19 MiB | 4.35 MiB | 1.16 MiB |
0d3e677+dirty | 2.92 MiB | 3.66 MiB | 758.42 KiB |
iOS (legacy) Performance metrics 🚀
|
Revision | Plain | With Sentry | Diff |
---|---|---|---|
8f0282e+dirty | 1215.46 ms | 1219.64 ms | 4.18 ms |
f8fa2ad+dirty | 1239.08 ms | 1235.49 ms | -3.59 ms |
686b3bc+dirty | 1210.89 ms | 1221.14 ms | 10.25 ms |
d7401ac+dirty | 1252.38 ms | 1275.04 ms | 22.66 ms |
9de3d45+dirty | 1229.16 ms | 1229.58 ms | 0.42 ms |
148f924+dirty | 1214.76 ms | 1215.73 ms | 0.97 ms |
800171e+dirty | 1229.96 ms | 1232.87 ms | 2.91 ms |
a0b0298+dirty | 1220.73 ms | 1226.87 ms | 6.14 ms |
ce76a0b+dirty | 1212.27 ms | 1217.83 ms | 5.56 ms |
0d3e677+dirty | 1214.39 ms | 1225.70 ms | 11.31 ms |
App size
Revision | Plain | With Sentry | Diff |
---|---|---|---|
8f0282e+dirty | 2.63 MiB | 3.76 MiB | 1.13 MiB |
f8fa2ad+dirty | 2.63 MiB | 3.78 MiB | 1.14 MiB |
686b3bc+dirty | 2.63 MiB | 3.74 MiB | 1.11 MiB |
d7401ac+dirty | 2.36 MiB | 2.83 MiB | 481.14 KiB |
9de3d45+dirty | 2.63 MiB | 3.77 MiB | 1.13 MiB |
148f924+dirty | 2.36 MiB | 3.04 MiB | 696.25 KiB |
800171e+dirty | 2.63 MiB | 3.69 MiB | 1.05 MiB |
a0b0298+dirty | 2.63 MiB | 3.75 MiB | 1.12 MiB |
ce76a0b+dirty | 2.63 MiB | 3.78 MiB | 1.14 MiB |
0d3e677+dirty | 2.36 MiB | 3.10 MiB | 753.12 KiB |
Android (new) Performance metrics 🚀
|
Revision | Plain | With Sentry | Diff |
---|---|---|---|
76d1baf+dirty | 339.02 ms | 408.65 ms | 69.63 ms |
bd0ca38+dirty | 272.43 ms | 290.67 ms | 18.24 ms |
c830127+dirty | 352.35 ms | 388.96 ms | 36.61 ms |
86818c7+dirty | 362.79 ms | 376.04 ms | 13.25 ms |
d7401ac+dirty | 373.98 ms | 394.08 ms | 20.10 ms |
43e66e0+dirty | 378.20 ms | 404.59 ms | 26.40 ms |
e754436+dirty | 424.60 ms | 478.19 ms | 53.59 ms |
d8668ce+dirty | 372.43 ms | 403.84 ms | 31.41 ms |
834729f+dirty | 413.31 ms | 400.80 ms | -12.52 ms |
5446992+dirty | 371.61 ms | 390.00 ms | 18.39 ms |
App size
Revision | Plain | With Sentry | Diff |
---|---|---|---|
76d1baf+dirty | 7.15 MiB | 8.09 MiB | 964.41 KiB |
bd0ca38+dirty | 7.15 MiB | 8.40 MiB | 1.25 MiB |
c830127+dirty | 7.15 MiB | 8.38 MiB | 1.23 MiB |
86818c7+dirty | 7.15 MiB | 8.39 MiB | 1.23 MiB |
d7401ac+dirty | 7.15 MiB | 8.04 MiB | 910.85 KiB |
43e66e0+dirty | 7.15 MiB | 8.37 MiB | 1.22 MiB |
e754436+dirty | 7.15 MiB | 8.40 MiB | 1.25 MiB |
d8668ce+dirty | 7.15 MiB | 8.35 MiB | 1.20 MiB |
834729f+dirty | 7.15 MiB | 8.40 MiB | 1.25 MiB |
5446992+dirty | 7.15 MiB | 8.12 MiB | 999.45 KiB |
The Android app fails to open the widget on CI after the capture screenshot step. Leaving this PR in draft till the issue is investigated and fixed. Maestro logs
|
timeout: 5_000 | ||
|
||
# Hide keyboard by tapping on a non-tappable element | ||
- tapOn: 'Email' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tapping on a non-tappable region is a work around to hide the keyboard since:
# Conflicts: # dev-packages/e2e-tests/maestro/feedback.yml # dev-packages/e2e-tests/maestro/feedback/happyFlow-android.yml # dev-packages/e2e-tests/maestro/feedback/happyFlow-ios.yml
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Android app fails to open the widget on CI after the capture screenshot step. Leaving this PR in draft till the issue is investigated and fixed.
Did a couple of attempts with higher timeouts but the issue persists indicating that the app hangs on CI after the screenshot is captured.
@@ -391,8 +391,7 @@ jobs: | |||
force-avd-creation: false | |||
disable-animations: true | |||
disable-spellchecker: true | |||
target: 'aosp_atd' | |||
channel: canary # Necessary for ATDs | |||
target: 'google_apis' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Google image seem to provide a more complete system image support for graphics operations fixing the capture screenshot failure. The downside is that it is heavier for the CI operations. Comparing two runs on main vs this PR showed an average 44s
bump in the test duration.
Test | main | this PR |
---|---|---|
RN 0.65.3 legacy jsc android production no | 5m 32s | 6m 16s |
RN 0.79.1 legacy hermes android production no | 5m 36s | 6m 49s |
RN 0.79.1 new hermes android production no | 6m 53s | 7m 7s |
Note that the screenshots functionality failed with the AOSP image even for the screenshots taken by the Maestro environment in case of failure (E.g. see this log artifcact were the image is blank).
📢 Type of change
Based on #4604
📜 Description
Adds Feedback Widget Capture Screenshot tests
💡 Motivation and Context
Part of #4302
💚 How did you test it?
CI (example of submitted feedback via the CI, Local:
Screen.Recording.2025-05-01.at.8.20.15.AM.mov
📝 Checklist
sendDefaultPII
is enabled🔮 Next steps
#skip-changelog