"patrol test ..." isn't running 2nd test after 1st, while via "patrol develop ..." it is Ok #1588

ghost · 2023-07-28T08:40:59Z

ghost
Jul 28, 2023

Hi, guys! I'm migrating our Flutter integration tests to the Patrol framework & currently facing an issue - "patrol test ..." isn't running 2nd test after 1st, while via "patrol develop ..." it is Ok. Maybe someone already had something similar? TY in advance!

Flutter: 3.7.12
Patrol: 2.1.0
Patrol CLI version: 2.0.4

flutter logs

Showing sdk gphone64 arm64 logs:
I/flutter (10423): The Dart VM service is listening on http://127.0.0.1:39731/9HNwZGzoENA=/
D/FlutterBranchSDK(10423): onAttachedToEngine call
D/FlutterBranchSDK(10423): setupChannels call
D/FlutterBranchSDK(10423): Branch SDK with log enable
D/FlutterBranchSDK(10423): onAttachedToActivity call
D/FlutterBranchSDK(10423): setActivity call
D/FlutterBranchSDK(10423): BranchReferralInitListener - params: {"+clicked_branch_link":false,"+is_first_session":false}
I/flutter (10423): Patrol (native): initialize() started
I/flutter (10423): Patrol (native): initialize() succeeded
I/flutter (10423): 00:00 +0: patrol_test_explorer
I/flutter (10423): PatrolAppService started on port 8082
I/flutter (10423): Patrol (native): markPatrolAppServiceReady() started
I/flutter (10423): 00:00 +1: patrol_ci_smoke_test Sign-up Only With Required Fields e2e
I/flutter (10423): Patrol (native): markPatrolAppServiceReady() succeeded
I/flutter (10423): PatrolAppService: registered "patrol_ci_smoke_test"
I/flutter (10423): PatrolAppService.listDartTests() called
I/flutter (10697): The Dart VM service is listening on http://127.0.0.1:41481/hjPGWN452UY=/
D/FlutterBranchSDK(10697): onAttachedToEngine call
D/FlutterBranchSDK(10697): setupChannels call
D/FlutterBranchSDK(10697): Branch SDK with log enable
D/FlutterBranchSDK(10697): onAttachedToActivity call
D/FlutterBranchSDK(10697): setActivity call
D/FlutterBranchSDK(10697): BranchReferralInitListener - params: {"+clicked_branch_link":false,"+is_first_session":false}
I/flutter (10697): Patrol (native): initialize() started
I/flutter (10697): Patrol (native): initialize() succeeded
I/flutter (10697): 00:00 +0: patrol_test_explorer
I/flutter (10697): PatrolAppService started on port 8082
I/flutter (10697): Patrol (native): markPatrolAppServiceReady() started
I/flutter (10697): 00:00 +1: patrol_ci_smoke_test Sign-up Only With Required Fields e2e
I/flutter (10697): Patrol (native): markPatrolAppServiceReady() succeeded
I/flutter (10697): PatrolAppService: registered "patrol_ci_smoke_test"
I/flutter (10697): PatrolAppService.listDartTests() called
I/flutter (10697): PatrolAppService.runDartTest(patrol_ci_smoke_test) called
I/flutter (10697): PatrolAppService: requested execution of test "patrol_ci_smoke_test"
I/flutter (10697): Patrol (native): configure() started
I/flutter (10697): Patrol (native): configure() succeeded
I/flutter (10697): Braze content card subscription not present. Removing any queued cards and adding only the recent refresh.
D/FlutterBranchSDK(10697): setRequestMetadata call
D/FlutterBranchSDK(10697): onListen call
I/flutter (10697): Patrol (native): isPermissionDialogVisible() started
I/flutter (10697): Patrol (native): isPermissionDialogVisible() succeeded
I/flutter (10697): Patrol (native): tap() started
I/flutter (10697): Patrol (native): tap() succeeded
I/flutter (10697): version = "Android 12"
I/flutter (10697): Init with applicationId=""
I/flutter (10697): Init result ="false"
D/FlutterBranchSDK(10697): onCancel call
I/flutter (10697): PatrolBinding: tearDown(): count: 1, results: {Sign-up Only With Required Fields e2e: success}
I/flutter (10697): PatrolBinding: tearDown(): test "Sign-up Only With Required Fields e2e" in group "patrol_ci_smoke_test", passed: true
I/flutter (10697): PatrolAppService.markDartTestAsCompleted(): patrol_ci_smoke_test
I/flutter (10697): 00:41 +2: patrol_ci_smoke_test Delete Account w/o reason e2e
I/flutter (10697): PatrolAppService: registered "patrol_ci_smoke_test"
I/flutter (10697): PatrolAppService: requested execution of test "patrol_ci_smoke_test"
I/flutter (10697): Patrol (native): configure() started
I/flutter (10697): Patrol (native): configure() succeeded

fylyppo · 2023-07-28T17:56:48Z

fylyppo
Jul 28, 2023
Maintainer

Hi, can you show me the code of these two tests?
Also please attach logs from patrol test with --verbose flag and use collapsed section (makes it easier to read the thread) to hide logs.

4 replies

ghost Jul 28, 2023

@fylyppo unfortunately, I can't, but I guess the code of tests is not the case, cuz I'm trying 2nd day with all possible options to understand what can be the reason for this. I'm so sure about the code because I even left in both tests only the first method, which just launches the app & waits for Finder to be visible:

Future patrolRunApp(PatrolTester $, Finder finder) async {
await app.mainTests();
await $.waitUntilVisible(finder, timeout: const Duration(seconds: 10));
}

ghost Jul 28, 2023

patrol test ... -v

Received 1 test target(s)
Received test target: .../patrol_smoke_test.dart
Generated entrypoint .../test_bundle.dart with 1 bundled test(s)
Received Android flavor: ..._dev
Received iOS flavor: ..._dev
$ flutter --no-version-check devices --machine
No device specified, using the first one (emulator-5554)
Received 1 device(s) to run on
Received device: emulator-5554
Received 9 --dart-define(s) (2 custom, 7 internal)
Received custom --dart-define: ..._ENVIRONMENT
Received custom --dart-define: RUNNING_E2E
Received internal --dart-define: PATROL_WAIT=0
Received internal --dart-define: PATROL_APP_PACKAGE_NAME=com.pack.name.dev
Received internal --dart-define: PATROL_APP_BUNDLE_ID=com.pack.name.dev
Received internal --dart-define: PATROL_ANDROID_APP_NAME=AppName
Received internal --dart-define: PATROL_IOS_APP_NAME=AppName
Received internal --dart-define: INTEGRATION_TEST_SHOULD_REPORT_RESULTS_TO_NATIVE=false
Received internal --dart-define: PATROL_TEST_LABEL_ENABLED=true
• Building apk with entrypoint test_bundle.dart...

✓ Completed building apk with entrypoint test_bundle.dart (84.5s)
Will uninstall apps before running tests
Uninstalling com.pack.name.dev from sdk gphone64 arm64
Uninstalling com.pack.name.dev.test from sdk gphone64 arm64
• Executing tests of apk with entrypoint test_bundle.dart on emulator-5554...

Task :app:connectedAppName_devDebugAndroidTest
: Starting 1 tests on Pixel_5_API_31(AVD) - 12
:
: Test results saved as file:.../build/app/outputs/androidTest-results/connected/flavors/..._dev/test-result.pb. Inspect these results in Android Studio by selecting Run > Import Tests From File from the menu bar and importing test-result.pb.
:
: BUILD SUCCESSFUL in 1m 20s
: 1080 actionable tasks: 4 executed, 4 from cache, 1072 up-to-date
✓ Completed executing apk with entrypoint test_bundle.dart on emulator-5554 (81.4s)
Uninstalling com.pack.name.dev from sdk gphone64 arm64
Uninstalling com.pack.name.dev.test from sdk gphone64 arm64

fylyppo Jul 29, 2023
Maintainer

From patrol test logs I see that only 1 test target was received but from flutter logs two targets. I suppose they come from two inconsistent invocations (the parameters were different). Run patrol test and flutter logs at once and update logs in the thread.
For now I can tell you that patrol is exploring integration_test directory and looking for *_test.dart files in order to execute these tests. So do your all test targets:

run with patrol develop
are placed inside that folder
are stored in own target file which ends with "_test.dart"
?

If that didn't help, could you provide some repo with minimal reproducible example? I'm sure it'll be much easier and faster to investigate this issue there.

ghost Jul 29, 2023

@fylyppo it was one invocation - "flutter logs" was launched 1st & in 2nd terminal tab was launched this command:
patrol test --target=integration_test/patrol_smoke_test.dart --flavor=..._dev --dart-define=..._ENVIRONMENT=..._dev --dart-define=RUNNING_E2E=true

My "patrol develop" is the same:
patrol develop --target=integration_test/patrol_smoke_test.dart --flavor=..._dev --dart-define=..._ENVIRONMENT=..._dev --dart-define=RUNNING_E2E=true

flutter logs (patrol develop)

Showing sdk gphone64 arm64 logs:
I/flutter ( 7120): The Dart VM service is listening on http://127.0.0.1:36229/Tyc5DeKBfbc=/
D/FlutterBranchSDK( 7120): onAttachedToEngine call
D/FlutterBranchSDK( 7120): setupChannels call
D/FlutterBranchSDK( 7120): Branch SDK with log enable
D/FlutterBranchSDK( 7120): onAttachedToActivity call
D/FlutterBranchSDK( 7120): setActivity call
D/FlutterBranchSDK( 7120): BranchReferralInitListener - params: {"+clicked_branch_link":false,"+is_first_session":false}
I/flutter ( 7120): Patrol (native): initialize() started
I/flutter ( 7120): Patrol (native): initialize() succeeded
I/flutter ( 7120): 00:00 +0: patrol_smoke_test Sign-up Only With Required Fields e2e
I/flutter ( 7120): Patrol (native): configure() started
I/flutter ( 7120): Patrol (native): configure() succeeded

D/FlutterBranchSDK( 7120): setRequestMetadata call
D/FlutterBranchSDK( 7120): onListen call

I/flutter ( 7120): Patrol (native): isPermissionDialogVisible() started
I/flutter ( 7120): Patrol (native): isPermissionDialogVisible() succeeded
I/flutter ( 7120): Patrol (native): tap() started
I/flutter ( 7120): Patrol (native): tap() succeeded
I/flutter ( 7120): version = "Android 12"
I/flutter ( 7120): Init with applicationId="..."
I/flutter ( 7120): Init result ="false"
D/FlutterBranchSDK( 7120): onCancel call
I/flutter ( 7120): 00:49 +1: patrol_smoke_test Delete Account w/o reason e2e
I/flutter ( 7120): Patrol (native): configure() started
I/flutter ( 7120): Patrol (native): configure() succeeded

D/FlutterBranchSDK( 7120): setRequestMetadata call
D/FlutterBranchSDK( 7120): onListen call
I/flutter ( 7120): 01:01 +1: patrol_smoke_test Sign-up Only With Required Fields e2e
I/flutter ( 7120): version = "Android 12"
I/flutter ( 7120): Init with applicationId="..."
I/flutter ( 7120): Init result ="false"

D/FlutterBranchSDK( 7120): onCancel call
I/flutter ( 7120): 01:06 +2: (tearDownAll)
I/flutter ( 7120): 01:06 +3: All tests passed!

patrol develop ... -v

Verbose mode enabled. More logs will be printed.
Received test target: .../integration_test/patrol_smoke_test.dart
Generated entrypoint.../integration_test/test_bundle.dart
Received Android flavor: ..._dev
Received iOS flavor: ..._dev
$ flutter --no-version-check devices --machine
No device specified, using the first one (emulator-5554)
Received device: emulator-5554
Received 10 --dart-define(s) (2 custom, 8 internal)
Received custom --dart-define: ..._ENVIRONMENT
Received custom --dart-define: RUNNING_E2E
Received internal --dart-define: PATROL_WAIT=0
Received internal --dart-define: PATROL_APP_PACKAGE_NAME=ccom.pack.name.dev
Received internal --dart-define: PATROL_APP_BUNDLE_ID=com.pack.name.dev
Received internal --dart-define: PATROL_ANDROID_APP_NAME=AppName
Received internal --dart-define: PATROL_IOS_APP_NAME=AppName
Received internal --dart-define: INTEGRATION_TEST_SHOULD_REPORT_RESULTS_TO_NATIVE=false
Received internal --dart-define: PATROL_TEST_LABEL_ENABLED=true
Received internal --dart-define: PATROL_HOT_RESTART=true
• Building apk with entrypoint test_bundle.dart...

BUILD SUCCESSFUL in 12s

Hot Restart: logs connected
Hot Restart: not attached to the app yet

The Dart VM service is listening on http://127.0.0.1:36229/Tyc5DeKBfbc=/

Patrol (native): initialize() started

Patrol (native): initialize() succeeded
00:00 +0: patrol_smoke_test Sign-up Only With Required Fields e2e
Patrol (native): configure() started
Patrol (native): configure() succeeded

D/FlutterBranchSDK( 7120): setRequestMetadata call
D/FlutterBranchSDK( 7120): onListen call

Task :app:connected..._devDebugAndroidTest
Patrol (native): isPermissionDialogVisible() started
Patrol (native): isPermissionDialogVisible() succeeded
Patrol (native): tap() started
Syncing files to device sdk gphone64 arm64... 37.1s

Hot Restart: attached to the app (press "r" to restart)
: Flutter run key commands.
: r Hot reload. 🔥🔥🔥
: R Hot restart.
: h List all available interactive commands.
: d Detach (terminate "flutter run" but leave application running).
: c Clear the screen
: q Quit (terminate the application on the device).
:
: 💪 Running with sound null safety 💪
:
: An Observatory debugger and profiler on sdk gphone64 arm64 is available at: http://127.0.0.1:52133/TKaQdiuT7FY=/
: The Flutter DevTools debugger and profiler on sdk gphone64 arm64 is available at: http://127.0.0.1:9100?uri=http://127.0.0.1:52133/TKaQdiuT7FY=/
Patrol (native): tap() succeeded
version = "Android 12"
Init with applicationId="..."
Init result ="false"
D/FlutterBranchSDK( 7120): onCancel call
00:49 +1: patrol_smoke_test Delete Account w/o reason e2e
Patrol (native): configure() started
Patrol (native): configure() succeeded

D/FlutterBranchSDK( 7120): setRequestMetadata call
D/FlutterBranchSDK( 7120): onListen call
01:01 +1: patrol_smoke_test Sign-up Only With Required Fields e2e
version = "Android 12"
Init with applicationId="..."
Init result ="false"

D/FlutterBranchSDK( 7120): onCancel call
01:06 +2: (tearDownAll)
01:06 +3: All tests passed!

ivan-horchakov · 2023-07-30T12:42:53Z

ivan-horchakov
Jul 30, 2023

@fylyppo I've created a public repo with Counter App as a minimal reproducible example (I'm using Flutter 3.7.12). The "integration_test/example_test.dart" has 2 same tests but only one is getting run during the "patrol test ..." command execution.
Please, checkout & (after all of Flutter's prelude) run the "patrol test --target integration_test/example_test.dart".

PatrolOnlyOneTestsGetsRun.23-07-30.at.14.39.mov

0 replies

fylyppo · 2023-07-31T06:36:53Z

fylyppo
Jul 31, 2023
Maintainer

Oh, now I see the problem. Inside main() you can have only one patrolTest(). So you just need to create separate *_test.dart file for each patrolTest() invocation.

2 replies

ghost Jul 31, 2023

Hm, a very interesting & strange technical limitation. I don't remember mention of this somewhere in docs...am I blind?
So for each new test we need to have separate file? Is there a technical possibility & plans to change this?
Ty in advance!

fylyppo Jul 31, 2023
Maintainer

Yes, we have a lack of this in the docs. We'll need to update them.
For now you need to have a separate file for each test but we plan to bring back test suites (you can follow this task here) so in the near future we'll be able to have many tests inside one file.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"patrol test ..." isn't running 2nd test after 1st, while via "patrol develop ..." it is Ok #1588

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

"patrol test ..." isn't running 2nd test after 1st, while via "patrol develop ..." it is Ok #1588

ghost Jul 28, 2023

Replies: 3 comments · 6 replies

fylyppo Jul 28, 2023 Maintainer

ghost Jul 28, 2023

ghost Jul 28, 2023

fylyppo Jul 29, 2023 Maintainer

ghost Jul 29, 2023

ivan-horchakov Jul 30, 2023

fylyppo Jul 31, 2023 Maintainer

ghost Jul 31, 2023

fylyppo Jul 31, 2023 Maintainer

ghost
Jul 28, 2023

Replies: 3 comments 6 replies

fylyppo
Jul 28, 2023
Maintainer

fylyppo Jul 29, 2023
Maintainer

ivan-horchakov
Jul 30, 2023

fylyppo
Jul 31, 2023
Maintainer

fylyppo Jul 31, 2023
Maintainer