-
Notifications
You must be signed in to change notification settings - Fork 33
Caraya 1.0 Release Notes
This is a change to take advantage of improvements brought by newer versions of LabVIEW, such as separate compile code, as well as benefitting from increased stability of classes and compiler optimizations.
We've added a unique identifier to each assertion calls in order to facilitate the identification of individual tests even in the case of use of redundant test labels such as when assertions are run in a loop or in parallel with the same call chain. This feature also allowed Caraya to fix a few bugs resulting from the name collisions, as well as allowing the assertions to be sorted by time of execution instead of being sorted alphabetically.
A test is a VI that uses the "Define Test" method to declare that any subsequent assertions should be registered with the Test Manager. Each test is now being given a unique identifier which the assertions use for hierarchization instead of relying on Call Chain. The call chain is still used in the framework, but this unique identifier for tests serves at separating tests that share the same call chain but with different labels.
We've added an optional "Verbose" flag to the Test Suites to allow Assertions to report a more detailed explanation of the reason for failures. Because Caraya is renowned for being the fastest Unit Test framework in LabVIEW, we wanted to keep it that way and decided to make this feature default to False.
In release 0.6, custom test reports were introduced, including a JUnit compatible report. In this version, we've extended the report to include an optional "Execution Time (s)" terminal on each assertion. Because Caraya is assertion-based, it can run headless without a Test Manager aggregating the results. It is therfore the responsibility of the developer to provide execution time metrics if this is needed.
For advanced users, Caraya introduces a Test Runner base class which declinates in four flavors available in the Advanced Palette: Folder, Project, Class and Collection. This enhancement allows programmatic discovery of tests that are not part of a specific test suite and execute those tests programmatically in a sequential manner.
- The "Test Runner.Folder" implementation works on disk and discover all tests under a specific folder, recursively or not.
- The "Test Runner.Project" implementation works on a project reference and discovers all tests under a specific virtual folder.
- The "Test Runner.Class" allows to discover and execute the tests that are found in the class' public methods. Class instance injection is supported to configure a test.
- The "Test Runner.Collection" allows for a collection of Test Runners to be configured and run as a single object. This feature is for advanced users who will want to have flexibility in defining their own test automation integrations from multiple sources.
The "Test Runner.Folder", "Test Runner.Class" and "Test Runner.Project" methods are slower that the strictly-defined test cases because they rely on VI Server for test discovery, but they offer flexibility and allow for both "Command Line execution" and "Quick Drop plugin" integration. They should be reserved to those users that want to extend the framework or use it in as part of their automated integration strategy. Test Runners do not work on RT targets or in the Run time engine.
The Test Runners are definitely for advanced users with very specific use cases which are not easily handled. But we've thought about the more traditional workflow as well and created a convenient polymorphic "Run Tests" method to wrap the most useful experience. It is available on the main palette page, next to the "Define Test" and "Assert" methods.
The "Run Tests" method supports a wide variety of scalar or array input types such as VI Path, VI Refnum, Folder, Class path, Object instance and project refnum. In each case, the framework discovers the defined tests which can be run and executes them serially. A Test Report or Test Path (xml or txt) can be supplied to directly output the report to file.
The class instance variant is particularly interesting with the capability to inject a pre-configured object (standard 4x2x2x4 connector supported only) and run all tests defined in a publicaly accessible method.
Because the "Run Tests" method uses the "Test Runners", it cannot be used in the run-time engine nor on RT targets.
The Test Runner base class also comes with a Tools menu shortcut which scripts the instantiation of a new derived Test Runner class. This Test Runner derivation permits to create the equivalent of recursive Test Suites. The base class handles pre- and post-Test Suite callbacks as well as setUp and tearDown for each Test Runner. This strategy can be leveraged to create complex collections of Test Runners, which essentially act as a traditional Test Suite.
Caraya now integrates with automation tools for CI/CD. It supports command line interface executable for testing all VIs in a folder and returning jUnit xml into a specified file. Syntax is available in this Wiki page: Command Line Interface.
Caraya now installs a quick-drop plugin (Ctrl-U) which runs all Caraya Unit Tests found in the active project. Optionally, the text in the QuickDrop combo box serves to match the best virtual folder to run tests from. Default behavior is to run all tests under "My Computer".
The 7000-7999 code range has been chosen for all Caraya framework errors. A wiki page has been created to keep track of those error codes. Error Codes
We've fixed some minor issues in the UI where sometimes an error dialog might occur.
- [Fix: 4] Fixed the intermittent error 1 when destroying Test Suites
- [Fix: 34] Added an optional "Execution Time (us)" terminal to assertion methods for reporting
- [Fix: 40] Assertions are now sorted in order they occur instead of ascending alphabetically
- [Fix: 44] Fixed the conditions that overwrote assertions when test names were identical
- [Fix: 47] Double-clicking on assertion result in Test Manager now works after VI has left memory for lvlibs inside lvlibs.
- [Fix: 50] Updated the Math Test Suite to use non-deprecated method for Test Report
- [Fix: 53] Fixed issues with comparison (greater/less) of variant representations when integers and floats were compared
- [Fix: 65] Restored RT compatibility for Process VI after CLI integration
- [Fix: 68] Fixed an issue where the number of skipped and failed tests in JUnit report were wrong.
- [Fix: 79] Added XML escaping to ensure JUnit parses correctly
Thanks to the community members for their issue reports, feature requests, code contributions, offline discussions and/or review&testing of beta versions for this release:
- arjen-hiemstra (@arjen-hiemstra)
- Carlo (@carlodri)
- Cyril Gambini (@CygaLV)
- Jim Kring (@jimkring)
- kosist (@kosist)
- Piotr Kruczkowski (@Primary-Key)
- OlivierLabo (@OlivierLabo)
- Matthew Jacobson (@jacobson3)
- Tomi Maila (@tmaila)
- Omar Mussa (@omarmussa)
- Francois Normandin (@francois-normandin)
- Matt Pollock (@pollockm)
- James D. Powell (@drjdpowell)
- Chris Stryker (@ChrisStrykesAgain)
- Sam Taggart (@stagg54)