Caraya 1.0 BETA Release Candidate 6 -- Refactored Test Runners

This is a sixth beta release for the 1.0 branch.

This build refactors the proposed Test Runner base class to emulate nested Test Suites.

As the previous line suggest, this refactoring was necessary to support a feature that basically replaces the current Test Suite class and allows for nested Test Suites. It also features a tool menu shortcut to create derived Test Runners that can be run as a collection of tests.

For those who had already started playing with Caraya 1.0's CLI and Test Runners for your continuous integration servers, you will find that a few changes will be required to adapt to this new architecture. In a nutshell, the Test Execution and Discovery methods will now both be protected and part of the overridable stream of actions that the Run and Discover Tests public API will call.

Personal note to the few people impacted by this choice: I felt that breaking the few existing installations was the price to pay considering that, once out of pre-release status, it would have been much more painful to break backward compatibility. In my opinion, now was the time to make use of the Test Runner base class (thanks @ChrisStrykesAgain) and support nested Tests execution with a minimal change of API. Since the few developers who probably already started developing with this feature are experts, and with the knowledge that a few examples are found in the Caraya project file, I felt it was the right (only) time to propose this change. I hope you'll agree, and I am available to answer any refactoring questions, should the refactor be less straightforward than I anticipate.

CARAYA 1.0 PRE-Release Notes

https://github.com/JKISoftware/Caraya/wiki/Caraya-1.0-Release-Notes

Caraya 1.0 BETA Release Candidate 5 -- QuickDrop Plugin support

This is a fifth beta release for the 1.0 branch.

This build restores the RT support which was broken in the previous few builds. Thanks to the community for spotting this.

It also adds support for "Project-based Test Runners" (previously supported only "Folder-based Test Runners"). This feature allows Caraya to introduce a Quick Drop Plugin (default shortcut Ctrl-Space + Ctrl-U) that scans the active project and runs all the Caraya Unit Tests. The project-based test runner can also be used in automated testing pipeline to run the same tests on a project as what the Quick Drop plugin would do. The symmetry of test coverage between a developer in the IDE and the CI service was what prompted for this late addition to the code base.

Hopefully, this is the last release candidate before the official 1.0 release of Caraya.

Don't forget that this version also allows G-CLI(**) integration with Caraya to fully automate your CI server and produce a useful Standard Output. Your feedback on the stdout content is welcome.

CARAYA 1.0 PRE-Release Notes

https://github.com/JKISoftware/Caraya/wiki/Caraya-1.0-Release-Notes

(**) The following repository is not part of Caraya, but is an example of the G-CLI working in tandem with Caraya:
https://github.com/LabVIEW-Open-Source/Caraya-CLI-extension
Install it as an add-on. It works with Caraya 1.0+

Caraya 1.0 BETA Release Candidate 4 -- Documented Error Codes

This is a fourth beta release for the 1.0 branch.

Following some bug reports during installation, undocumented error messages and tests for extending CLI with Standard Output support, this version includes documented error codes which can be found in the wiki page:

https://github.com/JKISoftware/Caraya/wiki/Caraya-Error-Codes

Tests made with an extension to call the Run Tests Programmatically method from a G-CLI compatible extension showed that referencing a method outside of vi.lib was causing problems to support multiple version, therefore a wrapper was created to wrap the code previously located under the Tools directory.

CARAYA 1.0 PRE-Release Notes

https://github.com/JKISoftware/Caraya/wiki/Caraya-1.0-Release-Notes

Caraya 1.0 BETA Release Candidate 3 -- Command Line Interface support

This is a third beta release for the 1.0 branch.
Following some very useful user feedback, we've implemented a new feature for this beta release.

A new "Test Runner" class has been created to discover tests from a source folder, and execute those tests without a predefined test suite.

An interface was built to run this new functionality manually and to serve as an example implementation for anyone wishing to use the Test Runner class as part of their CI setup. The interface can be loaded from the "Tools\Caraya" menu and currently supports both Plain Text and J-Unit XML reports. The VI can be run in Silent mode to be compatible with the "Command Line Interface" example code.

An example is available in the palette, under Utilities:

Furthermore, a pre-build action VI is supplied to run in your build specs. It is located in the root folder of Caraya under "vi.lib\addons_JKI Toolkits\Caraya".

*This is the pre-release of Caraya 1.0.

CUMULATIVE 1.0 Release Notes

Fixes:
[Fix: 4] Fixed the intermittent error 1 when destroying Test Suites
[Fix: 34] Added an optional "Execution Time (us)" terminal to assertion methods for reporting
[Fix: 40] Assertions are now sorted in order they occur instead of ascending alphabetically
[Fix: 44] Fixed the conditions that overewrote assertions when test names were identical
[Fix: 47] Double-clicking on assertion result in Test Manager now works after VI has left memory for lvlibs inside lvlibs.
[Fix: 50] Updated the Math Test Suite to use non-deprecated method for Test Report
[Fix: 53] Fixed issues with comparison (greater/less) of variant representations when integers and floats were compared

New Features:
[Feature: 34] Time is filtered from the JUnit report if execution time is not provided
[Feature: 41] Upgraded sources to LV2013
[Feature: 43] Added a verbose option the the Test Suite to export detailed explanation of failures
[Feature: 45] Attribute a unique identifier to each assertions performed
[Feature: 48] Added a Rerun All Tests menu in the Test Manager UI.
(NEW) [Feature: 52] Added support for Command Line Interface which allows discovery and execution of all tests found under a folder. Outputs to a J-Unit compatible report or a plain text report.
(NEW) [Feature: 55] Creation of a Test Runner class to search and execute tests programmatically. Includes a pre-build action for build integration.

Thanks for all the contributions that led to this release, whether by reporting or fixing issues:

Carlo (@carlodri)
Cyril Gambini (@CygaLV)
Jim Kring (@jimkring)
Piotr Kruczkowski (@primary-key)
OlivierLabo (@OlivierLabo)
Matthew Jacobson (@jacobson3)
Omar Mussa (@omarmussa)
Francois Normandin (@francois-normandin)
James D. Powell (@drjdpowell)
Chris Stryker (@ChrisStrykesAgain)
Sam Taggart (@stagg54)

Caraya 1.0 BETA Release Candidate 2 -- Programmatic Test Discovery and Execution

This is a second beta release for the 1.0 branch.
Following some very useful user feedback, we've implemented a new feature for this beta release.

A new "Test Runner" class has been created to discover tests from a source folder, and execute those tests without a predefined test suite.

An interface was built to run this new functionality manually and to serve as an example implementation for anyone wishing to use the Test Runner class as part of their CI setup. The interface can be loaded from the "Tools\Caraya" menu and currently supports both Plain Text and J-Unit XML reports. The VI can be run in Silent mode to be compatible with the upcoming "Command Line Interface" example code (yet to be released).

Furthermore, a pre-build action VI is supplied to run in your build specs. It is located in the root folder of Caraya under "vi.lib\addons_JKI Toolkits\Caraya".

*This is the pre-release of Caraya 1.0.

CUMULATIVE 1.0 Release Notes

Fixes:
[Fix: 4] Fixed the intermittent error 1 when destroying Test Suites
[Fix: 34] Added an optional "Execution Time (us)" terminal to assertion methods for reporting
[Fix: 40] Assertions are now sorted in order they occur instead of ascending alphabetically
[Fix: 44] Fixed the conditions that overewrote assertions when test names were identical
[Fix: 47] Double-clicking on assertion result in Test Manager now works after VI has left memory for lvlibs inside lvlibs.
[Fix: 50] Updated the Math Test Suite to use non-deprecated method for Test Report
[Fix: 53] Fixed issues with comparison (greater/less) of variant representations when integers and floats were compared

New Features:
[Feature: 34] Time is filtered from the JUnit report if execution time is not provided
[Feature: 41] Upgraded sources to LV2013
[Feature: 43] Added a verbose option the the Test Suite to export detailed explanation of failures
[Feature: 45] Attribute a unique identifier to each assertions performed
[Feature: 48] Added a Rerun All Tests menu in the Test Manager UI.
(NEW) [Feature: 55] Creation of a Test Runner class to search and execute tests programmatically. Includes a pre-build action for build integration.

Thanks for all the contributions that led to this release, whether by reporting or fixing issues:

Carlo (@carlodri)
Chris Stryker (@ChrisStrikesAgain)
Cyril Gambini (@CygaLV)
Jim Kring (@jimkring)
OlivierLabo (@OlivierLabo)
Matthew Jacobson (@jacobson3)
Omar Mussa (@omarmussa)
Francois Normandin (@francois-normandin)
James D. Powell (@drjdpowell)
Sam Taggart (@stagg54)

Caraya 1.0 BETA Release Candidate 1

*This is the pre-release of Caraya 1.0.

Release Notes:

Fixes:
[Fix: 4] Fixed the intermittent error 1 when destroying Test Suites
[Fix: 34] Added an optional "Execution Time (us)" terminal to assertion methods for reporting
[Fix: 40] Assertions are now sorted in order they occur instead of ascending alphabetically
[Fix: 44] Fixed the conditions that overewrote assertions when test names were identical
[Fix: 47] Double-clicking on assertion result in Test Manager now works after VI has left memory for lvlibs inside lvlibs.
[Fix: 50] Updated the Math Test Suite to use non-deprecated method for Test Report
[Fix: 53] Fixed issues with comparison (greater/less) of variant representations when integers and floats were compared

New Features:
[Feature: 34] Time is filtered from the JUnit report if execution time is not provided
[Feature: 41] Upgraded sources to LV2013
[Feature: 43] Added a verbose option the the Test Suite to export detailed explanation of failures
[Feature: 45] Attribute a unique identifier to each assertions performed
[Feature: 48] Added a Rerun All Tests menu in the Test Manager UI.

Thanks for all the contributions that led to this release, whether by reporting or fixing issues:

Carlo (@carlodri)
Cyril Gambini (@CygaLV)
Jim Kring (@jimkring)
OlivierLabo (@OlivierLabo)
Matthew Jacobson (@jacobson3)
Omar Mussa (@omarmussa)
Francois Normandin (@francois-normandin)
James D. Powell (@drjdpowell)
Sam Taggart (@stagg54)

Hotfix: Added ability to show only failed tests on the main UI (and a minor UI bugfix)

Cumulative release note with 0.6.0 (0.6.3 changes in bold):

[Feature: 7] Adding support for JUnit-compatible XML file export
[Feature: 11] Adding an "Assert Equal_Float" assertion to compare float representation within a certain precision
[Feature: 14] Hook to inject an alternate "Export" format
[Fix: 15] Running Caraya Assertions in EXE builds
[Feature: 20] Double-click on test result in Test Manager GUI now points to the Assertion node if Text of label attached to assertion node matches the exact name of the test.
[Fix: 24] Default report generation now fixes hierarchy of assertions
[Feature: 25] Change of specification for Assert Equal to separate "Value" and "Value and Type" assertions
[Feature: 26] Assertions are now grouped in a polymorphic VIfor palette menu.
[Fix: 27] Double-clicking on empty space in the tree of the Test Manager UI no longer generates an error.
[Fix: 28] Fixed the issue where opening a test VI after it had left memory would give an error.
[Fix: 31] Equal Value Comparison now asserts correctly when comparing equal typedef with non-typedef values
[Fix: 32] Added recursive verification inside cluster to assert value equality for nested values with different type representations
[Feature: 5] Added ability to show only failed tests on the main UI
[Fix: 3] Clicking on expand/collapse (+/-) tree node quickly will no longer open the test VI (as a double-click on test does)

Hotfix: Nested type representation equality

Cumulative release note with 0.6.0 (0.6.1 changes in bold):

[Feature: 7] Adding support for JUnit-compatible XML file export
[Feature: 11] Adding an "Assert Equal_Float" assertion to compare float representation within a certain precision
[Feature: 14] Hook to inject an alternate "Export" format
[Fix: 15] Running Caraya Assertions in EXE builds
[Feature: 20] Double-click on test result in Test Manager GUI now points to the Assertion node if Text of label attached to assertion node matches the exact name of the test.
[Fix: 24] Default report generation now fixes hierarchy of assertions
[Feature: 25] Change of specification for Assert Equal to separate "Value" and "Value and Type" assertions
[Feature: 26] Assertions are now grouped in a polymorphic VIfor palette menu.
[Fix: 27] Double-clicking on empty space in the tree of the Test Manager UI no longer generates an error.
[Fix: 28] Fixed the issue where opening a test VI after it had left memory would give an error.
[Fix: 31] Equal Value Comparison now asserts correctly when comparing equal typedef with non-typedef values
[Fix: 32] Added recursive verification inside cluster to assert value equality for nested values with different type representations

0.6.0 Caraya with Test Reports and Poly Assertions

This version features many improvements and is marked as a stable release.
It includes Test Report classes that allow the developer to create his or her own report formats. Example JUnit and SimpleText reports are provided. Please refer to the Wiki page associated with release 0.6.0 for more information. (https://github.com/JKISoftware/Caraya/wiki/Caraya-0.6.0-Release-Notes)
It also includes multiple fixes, reorganization of the palette and a few UI improvements such as double-click on a test to highlight the source node associated with that test.

0.6.0 Release Candidate 1

[Feature: 7] Adding support for JUnit-compatible XML file export
[Feature: 11] Adding an "Assert Equal_Float" assertion to compare float representation within a certain precision
[Feature: 14] Hook to inject an alternate "Export" format
[Fix: 15] Running Caraya Assertions in EXE builds
[Feature: 20] Double-click on test result in Test Manager GUI now points to the Assertion node if Text of label attached to assertion node matches the exact name of the test.
[Fix: 24] Default report generation now fixes hierarchy of assertions
[Feature: 25] Change of specification for Assert Equal to separate "Value" and "Value and Type" assertions
[Feature: 26] Assertions are now grouped in a polymorphic VIfor palette menu.
[Fix: 27] Double-clicking on empty space in the tree of the Test Manager UI no longer generates an error.
[Fix: 28] Fixed the issue where opening a test VI after it had left memory would give an error.

Thanks to the community members for their issue reports, feature requests and/or code contributions for this release:

Christopher Field (@volks73)
Stefan Droege (@StefanD986)
Kosist (@kosist)
Francois Normandin (@francois-normandin)
Omar Mussa (@omarmussa)
Jim Kring (@jimkring)
OlivierLabo (@OlivierLabo)
James Powell (@drjdpowell)
Neil Pate (@neilpate)