Rerun All Tests does not do the same thing every time

[logmanoriginal]

Caraya 1.2.1, LV2017SP1f3 (32-bit)

I've been playing with the latest patch and found that the Rerun All Tests feature is behaving strangely. Here are a few scenarios I have tested:

---

Scenario 1 - simple test case

• Create a blank project
• Create a new VI
• Place the New Test quick-drop template
• Run the VI
• Select Action > Rerun All Tests

Notice that the same test executes again in the same window. This is the expected behavior.

Save the VI for use in the next scenarios.

---

Scenario 2 - simple test suite

(using the VI from scenario 1)

• Place Define Test Suite (Default) before Define Test
  • Connect a true constant to the interactive? terminal
• Place Destroy Test Suite after the assertion
• Run the VI
• Select Action > Rerun All Tests

Notice that every time a new dialog appears (only when interactive? is set to true). I'm not sure if this is the intended behavior, because I'd expect it to rerun all tests in the same dialog.

---

Scenario 3 - using the project test runner

(using the VI from scenario 1)

• Close all VIs and only keep the project open
• Select Tools > Caraya > Run Tests in Active Project
• Select Action > Rerun All Tests
• (in the new dialog) Select Action > Rerun All Tests

Notice that here a second dialog appears the first time we rerun the tests, but if we rerun again it reuses the second dialog. I believe the behavior of the second dialog is the expected behavior and this is somehow related to the application context issue discussed in #138.

---

Scenario 4 - using the disk test runner

(using the VI from scenario 1)

• Close all VIs and only keep the project open
• Select Tools > Caraya > Select and Run Tests on Disk
• Under Output Folder define an output folder
• Press Run All Tests
• Select Action > Rerun All Tests

Notice that an error message appears. (if the error message does not appear, make sure that all VIs are closed before running the test)

---

Scenario 5 - test case inside project library

(using the VI from scenario 1)

• Add a project library
• Move the VI from scenario 1 inside the library
• Make sure all VIs are closed and only the project is open.
• Select either one:
  • Select Tools > Caraya > Select and Run Tests on Disk
  • Select Tools > Caraya > Run Tests in Active Project
• Select Action > Rerun All Tests

Notice that nothing happens.
Now here comes the strange part.

• Open the test case VI
• Repeat the procedure above

Notice that this time the VI executes again and another dialog appears.

[francois-normandin]

Scenario 1 : I confirm this is expected behavior.

Scenario 2: This is also expected behavior.
The way the framework has been built, the Test Manager is tied to a Test Suite. Every test suite will generate a new UI. The reason is obscure and was not a conscious choice in the early days of Caraya, as the framework did not support nested suites. This fact persists to this day. A refactor (decoupling) of the Basic Test Manager is necessary to unlink these two models.

Scenario 3: I agree with you that this is another case of issue #138. My assumption is that the rerun does not start from the project's context. To be investigated (bug).

Scenario 4 and 5: Different flavors of "Test not in memory". When we rerun tests which were run using VI Server, we must try to find the VI's path on disk based on the Call Chain. However, asynchronous launch of VIs generate a call chain through ACBR Proxy Caller. There is a quick check to see if we could get lucky and find the path from an alternate mean (project base path, app folder, etc.) but I would call this feature "unfinished" at best.

[francois-normandin]

I have found another incongruity... Let's call it Scenario 6:

• Create a project
• Add two simple tests like scenario 1. (A.vi and B.vi)
• In A.vi, add a RunTest (byPath) and point to path of B.vi

We expect that:

• Running A.vi alone will assert A and run B.vi using VI Server
• Running B.vi alone will assert B
• Running from Tools "Run Tests in Active Project" should assert A once and B twice (can't filter a link by Path)

What is weird is, in the 3rd case, Rerun will only assert B and completely skip A. I suspect the "root test" does not resolve, and the next item in the call chain is used as root.

[logmanoriginal]

Scenario 2: This is also expected behavior.
The way the framework has been built, the Test Manager is tied to a Test Suite. Every test suite will generate a new UI. The reason is obscure and was not a conscious choice in the early days of Caraya, as the framework did not support nested suites. This fact persists to this day. A refactor (decoupling) of the Basic Test Manager is necessary to unlink these two models.

Ah okay, so this was a design decision after all.
It looks like #72 is related to this scenario.

Scenario 4 and 5: Different flavors of "Test not in memory". When we rerun tests which were run using VI Server, we must try to find the VI's path on disk based on the Call Chain. However, asynchronous launch of VIs generate a call chain through ACBR Proxy Caller. There is a quick check to see if we could get lucky and find the path from an alternate mean (project base path, app folder, etc.) but I would call this feature "unfinished" at best.

This is also a dangerous path, because if we get unlucky it executes random VIs from disk simply because the name matches.

---

Regarding scenario 6, when tests are registered, they are stored as variant attributes by their Test ID, which means that order of execution is lost. So if B.vi has a lower ID than A.vi, it will be used as root.