Test report does not get generated from Caraya CLI when generating multiple assertions

[Robbie20]

I'm currently running into a problem where I'm executing different tests through running Caraya CLI.vi.
By running from the command line something like:
"<LV_Path>\Labview.exe" "<Caraya_Path>\Caraya CLI.vi" -- -x "<TestFolder_Path>\report.xml" -s "<TestFolder_path>" -r -v

I think I've narrowed down my issue to the fact that my test is generating too many assertions such that it slows down significantly whatever process is handling the assertions (which I presume is related to this)

I replicated my issue through a simpler example. Here is a single test VI:

And then I have a wrapper that runs the test with different parameters:

I tried to debug this a little by running Caraya\subVIs\Run Tests Programmatically.vi directly.
This took me to the actual TestRunner.lvclass:Run.vi's sequence:

What I think is happening here is that the tests finish running (TestRunner.lvclass:OnTestExecution.vi being the 'Run' step shown), but the process that's populating the assertions information has not (if you run interactively you can see the pop up window still populating). As far as I understand, TestRunner.lvclass:DestroyTestSuite.vi is the one in charge of sending the signal to create the XML report. So we get to this step while a concurrent process is still busy populating the assertions' information. Hence, I can see that the destroy test suite step is currently silently failing with this error:

Error 7003 occurred at Caraya.lvlib:Test Suite.lvclass:Destroy Test Suite.vi
Possible reason(s):
Caraya Test Manager: Test Manager shutdown timed out

Which again I'm attributing to the fact that the other process is busy still processing the assertions.
Note that since this is a Caraya error Run Tests Programmatically.vi clears it up.

Going back to the original use case, adding '-q 0' when running Caraya CLI.vi seems to give it time to actually generate the report (since LV does not close). My guess is that when I don't add that option, from Caraya CLI.vi we end up running Quit LabVIEW before we have enough time to process the assertions, and thus we don't get to generate the report. However, we don't want to leave LV open since this is intended to run on a test agent and we intend to run more tests onwards.

One of the problems I see, is that there's no real handshake or guarantee that we'll be able to generate the report.
The timeout is the only form of 'confirmation', which we don't get to configure, or isn't exposed through Caraya CLI.vi.

Any thoughts on how to go about this?

[francois-normandin]

Hi @Robbie20
I've tried reproducing what you see and I get a report generated. No errors.
I'm running LabVIEW 2021 32-bit and I have Caraya 1.2.1.

I'm not certain what could cause your setup to fail. We did improve the execution speed with version 1.2, so make sure you run the latest version. If you can't get to the bottom of it, please upload a zip of a simple piece of code that reproduces the issue, so we can try and run it. I've attached the source I ran, but it's in LV2021...

test.zip

[Robbie20]

Here's the sample code I wrote:
Multiple Tests Sample.zip

Though I don't see anything significantly different from what you showed me.
Sorry for not specifying before, I am currently using version 1.2.1.131 on LV 2019.

Could you try increasing the number of iterations maybe? I think something ridiculous like 2000 should make it more obvious, though I'm still a bit surprised you wouldn't see it before. I've been able to reproduce it on different machines, though with a similar setup.

[francois-normandin]

With 2000 iterations, which takes substantially more time to execute, I reproduce the problem you are seeing.

Adding a 10ms timeout to throttle the For Loop, the problem goes away, but with 2 or 5ms delays, it was still not producing the report.
I agree this is a problem of Test-Manager parsing algorithm that is too slow at high count. There must be a build up of a large enough delay between processing two test results that goes beyond the Test Suite's timeout value, which will kill the whole process.

We could increase the Test Suite's timeout value, but this is a temporary (and not at all satisfying) patch.
I'll mark this as linked to the need to decouple the Test Manager UI from the actual Test Processing Engine.

Workaround: Open the Test Suite creation methods and change the default timeout from 2000ms to 20000ms