-
Notifications
You must be signed in to change notification settings - Fork 33
Caraya 1.0 Release Notes
This is a change to take advantage of improvements brought by newer versions of LabVIEW, such as separate compile code, as well as benefitting from increased stability of classes and compiler optimizations.
We've added a unique identifier to each assertion calls in order to facilitate the identification of individual tests even in the case of use of redundant test labels such as when assertions are run in a loop or in parallel with the same call chain. This feature also allowed Caraya to fix a few bugs resulting from the name collisions, as well as allowing the assertions to be sorted by time of execution instead of being sorted alphabetically.
We've added an optional "Verbose" flag to the Test Suites to allow Assertions to report a more detailed explanation of the reason for failures. Because Caraya is renowned for being the fastest Unit Test framework in LabVIEW, we wanted to keep it that way and decided to make this feature default to False.
Add an image here for verbose option
In release 0.6, custom test reports were introduced, including a JUnit compatible report. In this version, we've extended the report to include an optional "Execution Time (s)" terminal on each assertion. Because Caraya is assertion-based, it can run headless without a Test Manager aggregating the results. It is therfore the responsibility of the developer to provide execution time metrics if this is needed.
Add an image here for execution time option + example of test report
Caraya introduces a Test Runner base class which declinates in three flavors available in the palette: Folder, Project and Collection. This enhancement allows programmatic discovery of tests that are not part of a specific test suite (nested Test suites are still not supported) and execute those tests programmatically in a sequential manner.
- The "Test Runner.Folder" implementation works on disk and discover all tests under a specific folder, recursively or not.
- The "Test Runner.Project" implementation works on a project reference and discovers all tests under a specific virtual folder.
- The "Test Runner.Collection" allows for a collection of Test Runners to be configured and run as a single object. This feature is for advanced users who will want to have flexibility in defining their own test automation integrations from multiple sources.
The "Test Runner.Folder" and "Test Runner.Project" methods are slower that the strictly-defined test cases because they rely on VI Server for test discovery, but they offer flexibility and allow for both "Command Line execution" and "Quick Drop plugin" integration.
The Test Runner base class also comes with a Tools menu shortcut which scripts the instantiation of a new derived Test Runner class. This Test Runner derivation permits to create the equivalent of recursive Test Suites. The base class handles pre- and post-Test Suite callbacks as well as setUp and tearDown for each Test Runner. This strategy can be leveraged to create complex collections of Test Runners, which essentially act as a traditional Test Suite.
Add an image here for the Test Runner palette + a link to a demo of the Test Runner in action
Caraya now integrates with automation tools for CI/CD. It supports command line interface executable for testing all VIs in a folder and returning jUnit xml into a specified file. Syntax is available in this Wiki page: Command Line Interface.
Caraya now installs a quick-drop plugin (Ctrl-U) which runs all Caraya Unit Tests found in the active project. Optionally, the text in the QuickDrop combo box serves to match the best virtual folder to run tests from. Default behavior is to run all tests under "My Computer".
The 7000-7999 code range has been chosen for all Caraya framework errors. A wiki page has been created to keep track of those error codes. Error Codes
We've fixed some minor issues in the UI where sometimes an error dialog might occur.
- [Fix: 4] Fixed the intermittent error 1 when destroying Test Suites
- [Fix: 34] Added an optional "Execution Time (us)" terminal to assertion methods for reporting
- [Fix: 40] Assertions are now sorted in order they occur instead of ascending alphabetically
- [Fix: 44] Fixed the conditions that overwrote assertions when test names were identical
- [Fix: 47] Double-clicking on assertion result in Test Manager now works after VI has left memory for lvlibs inside lvlibs.
- [Fix: 50] Updated the Math Test Suite to use non-deprecated method for Test Report
- [Fix: 53] Fixed issues with comparison (greater/less) of variant representations when integers and floats were compared
- [Fix: 65] Restored RT compatibility for Process VI after CLI integration
- [Fix: 68] Fixed an issue where the number of skipped and failed tests in JUnit report were wrong.
- [Fix: 79] Added XML escaping to ensure JUnit parses correctly
Thanks to the community members for their issue reports, feature requests, code contributions, offline discussions and/or review&testing of beta versions for this release:
- arjen-hiemstra (@arjen-hiemstra)
- Carlo (@carlodri)
- Cyril Gambini (@CygaLV)
- Jim Kring (@jimkring)
- kosist (@kosist)
- Piotr Kruczkowski (@Primary-Key)
- OlivierLabo (@OlivierLabo)
- Matthew Jacobson (@jacobson3)
- Tomi Maila (@tmaila)
- Omar Mussa (@omarmussa)
- Francois Normandin (@francois-normandin)
- Matt Pollock (@pollockm)
- James D. Powell (@drjdpowell)
- Chris Stryker (@ChrisStrykesAgain)
- Sam Taggart (@stagg54)