Weird Failure when executing multiple tests with shutdown simulator

[yongjincho92]

Hi,

So I have set up a KIF on a small simple app which only has a table view with 50 cells.

And I have written two tests like this.

- (void)testSwipeAction
{
    // Go find one cell out of the screen.
    [tester waitForCellAtIndexPath:[NSIndexPath indexPathForRow:40 inSection:0]
        inTableViewWithAccessibilityIdentifier:@"table"];

    // Move back to top of the table.
    [tester waitForCellAtIndexPath:[NSIndexPath indexPathForRow:1 inSection:0]
        inTableViewWithAccessibilityIdentifier:@"table"];
}

- (void)testTapVisibleCell
{
    [tester tapViewWithAccessibilityIdentifier:@"Cell 1"];
}

So there are two scenarios when I execute the tests: 1) when the destination simulator is shutdown and 2) when the destination simulator is already booted.

• When I run tests via Xcode, in both scenarios, tests worked fine.
• However, when I run the tests via BUCK, it would work fine on scenario 2. But on scenario 1, it would hang on the second test method testTapVisibleCell. And the failure I see here is that KIFUITestActor(IdentifierTests) tapViewWithAccessibilityIdentifier: would lead to NSBundle accessibilityBundleWithLastPathComponent: which only exists until iOS 9.0.

Below is the trace exception

Test Suite 'All tests' started at 2020-10-14 13:53:59.502
Test Suite 'SCSampleScreenTests.xctest' started at 2020-10-14 13:53:59.503
Test Suite 'SCSampleScreenTests' started at 2020-10-14 13:53:59.503
Test Case '-[SCSampleScreenTests testSwipeAction]' started.
Test Case '-[SCSampleScreenTests testSwipeAction]' passed (2.482 seconds).
Test Case '-[SCSampleScreenTests testTapVisibleCell]' started.
2020-10-14 13:54:02.191 TestHostApplication[37224:37071515] +[NSBundle accessibilityBundleWithLastPathComponent:]: unrecognized selector sent to class 0x7fff87d0b248
2020-10-14 13:54:02.192 TestHostApplication[37224:37071515] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '+[NSBundle accessibilityBundleWithLastPathComponent:]: unrecognized selector sent to class 0x7fff87d0b248'
*** First throw call stack:
(
	0   CoreFoundation                      0x00007fff23c7127e __exceptionPreprocess + 350
	1   libobjc.A.dylib                     0x00007fff513fbb20 objc_exception_throw + 48
	2   CoreFoundation                      0x00007fff23c91ed4 +[NSObject(NSObject) doesNotRecognizeSelector:] + 132
	3   CoreFoundation                      0x00007fff23c75c4c ___forwarding___ + 1436
	4   CoreFoundation                      0x00007fff23c77f78 _CF_forwarding_prep_0 + 120
	5   UIKit                               0x0000000112007084 __46+[AXUIKitGlue _handleWebKitLegacyInstallation]_block_invoke + 112
	6   libdispatch.dylib                   0x00007fff5223f7b9 _dispatch_client_callout + 8
	7   libdispatch.dylib                   0x00007fff522409e5 _dispatch_once_callout + 20
	8   UIKit                               0x0000000112007012 +[AXUIKitGlue _handleWebKitLegacyInstallation] + 34
	9   libdispatch.dylib                   0x00007fff5223f7b9 _dispatch_client_callout + 8
	10  libdispatch.dylib                   0x00007fff52241b75 _dispatch_continuation_pop + 440
	11  libdispatch.dylib                   0x00007fff522529cc _dispatch_source_invoke + 1997
	12  libdispatch.dylib                   0x00007fff5224bb38 _dispatch_main_queue_callback_4CF + 857
	13  CoreFoundation                      0x00007fff23bd4049 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 9
	14  CoreFoundation                      0x00007fff23bceca9 __CFRunLoopRun + 2329
	15  CoreFoundation                      0x00007fff23bce066 CFRunLoopRunSpecific + 438
	16  SCSampleScreenTests                 0x0000000108505f29 -[KIFTestActor tryRunningBlock:complete:timeout:error:] + 537
	17  SCSampleScreenTests                 0x000000010850614a -[KIFTestActor runBlock:complete:timeout:] + 138
	18  SCSampleScreenTests                 0x0000000108506256 -[KIFTestActor runBlock:complete:] + 150
	19  SCSampleScreenTests                 0x0000000108506331 -[KIFTestActor runBlock:] + 65
	20  SCSampleScreenTests                 0x000000010850e600 -[KIFUITestActor waitForAccessibilityElement:view:withElementMatchingPredicate:tappable:] + 208
	21  SCSampleScreenTests                 0x000000010850e28e -[KIFUITestActor waitForAccessibilityElement:view:withIdentifier:tappable:] + 318
	22  SCSampleScreenTests                 0x000000010850b856 -[KIFUITestActor(IdentifierTests) tapViewWithAccessibilityIdentifier:] + 118
	23  SCSampleScreenTests                 0x00000001084f5fca -[SCSampleScreenTests testTapVisibleCell] + 154
	24  CoreFoundation                      0x00007fff23c7820c __invoking___ + 140
	25  CoreFoundation                      0x00007fff23c753af -[NSInvocation invoke] + 319
	26  XCTest                              0x00000001081f2037 __24-[XCTestCase invokeTest]_block_invoke_2 + 52
	27  XCTest                              0x00000001081f1fe3 __24-[XCTestCase invokeTest]_block_invoke.206 + 320
	28  XCTest                              0x000000010824cdc2 +[XCTestCase(Failures) performFailableBlock:testCase:testCaseRun:shouldInterruptTest:] + 69
	29  XCTest                              0x000000010824ccd4 -[XCTestCase(Failures) _performTurningExceptionsIntoFailuresInterruptAfterHandling:block:] + 115
	30  XCTest                              0x00000001081f19f6 -[XCTestCase invokeTest] + 1183
	31  XCTest                              0x00000001081f3329 __26-[XCTestCase performTest:]_block_invoke_2 + 43
	32  XCTest                              0x000000010824cdc2 +[XCTestCase(Failures) performFailableBlock:testCase:testCaseRun:shouldInterruptTest:] + 69
	33  XCTest                              0x000000010824ccd4 -[XCTestCase(Failures) _performTurningExceptionsIntoFailuresInterruptAfterHandling:block:] + 115
	34  XCTest                              0x00000001081f3260 __26-[XCTestCase performTest:]_block_invoke.359 + 86
	35  XCTest                              0x000000010825fa0d +[XCTContext runInContextForTestCase:block:] + 211
	36  XCTest                              0x00000001081f2b14 -[XCTestCase performTest:] + 566
	37  XCTest                              0x000000010823938e -[XCTest runTest] + 57
	38  XCTest                              0x00000001081ecd50 __27-[XCTestSuite performTest:]_block_invoke + 354
	39  XCTest                              0x00000001081ec4a2 __59-[XCTestSuite _performProtectedSectionForTest:testSection:]_block_invoke + 24
	40  XCTest                              0x000000010825fa0d +[XCTContext runInContextForTestCase:block:] + 211
	41  XCTest                              0x00000001081ec459 -[XCTestSuite _performProtectedSectionForTest:testSection:] + 148
	42  XCTest                              0x00000001081ec7be -[XCTestSuite performTest:] + 348
	43  XCTest                              0x000000010823938e -[XCTest runTest] + 57
	44  XCTest                              0x00000001081ecd50 __27-[XCTestSuite performTest:]_block_invoke + 354
	45  XCTest                              0x00000001081ec4a2 __59-[XCTestSuite _performProtectedSectionForTest:testSection:]_block_invoke + 24
	46  XCTest                              0x000000010825fa0d +[XCTContext runInContextForTestCase:block:] + 211
	47  XCTest                              0x00000001081ec459 -[XCTestSuite _performProtectedSectionForTest:testSection:] + 148
	48  XCTest                              0x00000001081ec7be -[XCTestSuite performTest:] + 348
	49  XCTest                              0x000000010823938e -[XCTest runTest] + 57
	50  XCTest                              0x00000001081ecd50 __27-[XCTestSuite performTest:]_block_invoke + 354
	51  XCTest                              0x00000001081ec4a2 __59-[XCTestSuite _performProtectedSectionForTest:testSection:]_block_invoke + 24
	52  XCTest                              0x000000010825fa0d +[XCTContext runInContextForTestCase:block:] + 211
	53  XCTest                              0x00000001081ec459 -[XCTestSuite _performProtectedSectionForTest:testSection:] + 148
	54  XCTest                              0x00000001081ec7be -[XCTestSuite performTest:] + 348
	55  XCTest                              0x000000010823938e -[XCTest runTest] + 57
	56  XCTest                              0x000000010826ef14 __44-[XCTTestRunSession runTestsAndReturnError:]_block_invoke + 171
	57  XCTest                              0x000000010826f001 __44-[XCTTestRunSession runTestsAndReturnError:]_block_invoke.100 + 96
	58  XCTest                              0x0000000108207746 -[XCTestObservationCenter _observeTestExecutionForBlock:] + 682
	59  XCTest                              0x000000010826ec9f -[XCTTestRunSession runTestsAndReturnError:] + 615
	60  XCTest                              0x00000001081d0744 -[XCTestDriver runTestsAndReturnError:] + 456
	61  XCTest                              0x000000010825b64c _XCTestMain + 2496
	62  libXCTestBundleInject.dylib         0x0000000108056bfa __copy_helper_block_e8_32s + 0
	63  CoreFoundation                      0x00007fff23bd429c __CFRUNLOOP_IS_CALLING_OUT_TO_A_BLOCK__ + 12
	64  CoreFoundation                      0x00007fff23bd3a08 __CFRunLoopDoBlocks + 312
	65  CoreFoundation                      0x00007fff23bce894 __CFRunLoopRun + 1284
	66  CoreFoundation                      0x00007fff23bce066 CFRunLoopRunSpecific + 438
	67  GraphicsServices                    0x00007fff384c0bb0 GSEventRunModal + 65
	68  UIKitCore                           0x00007fff48092d4d UIApplicationMain + 1621
	69  TestHostApplication                 0x0000000107fa9900 main + 112
	70  libdyld.dylib                       0x00007fff5227ec25 start + 1
)
libc++abi.dylib: terminating with uncaught exception of type NSException
2020-10-14 13:54:17.073 xcodebuild[37193:37070122]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:

What's really weird here is that it only happens when I am running the KIF tests with shutdown simulator (i.e. running test will start the simulator and then run the tests).

My Environment

• Xcode version 11.6
• Simulator I tried : iPhone 8 (iOS 12.4, iOS 13.3, iOS 13.5)

[dostrander]

Very interesting I haven't run into this before adding it to the bug list, thanks for letting us know!