Automated conformance tests

Introduction

Conformance test quality in toolkit release can be improved by testing the actual release downloadable before the release. Automating these tests will make it easier and more convenient to test the status of the embedded conformance tests that are bundled with a Toolkit release, which is packaged as a WAR file.

Webui-tests maven module

The webui-tests module, part of the toolkit build and optionally run using the WebUI-Tests maven profile, has automated this approach using a combination of HtmlUnit, Jetty, and Spock tests. HtmlUnit is "a GUI-Less browser for Java programs", so that interactions between Toolkit and the browser are simulated through the API. A WAR file is required to load it into a Servlet container that supports war file deployment. Jetty is used to deploy the WAR file that is packaged by the xdstools2 module. Spock specification tests are used program the interations to launch all conformance tests for a given actor.

Spock Test design

Each actor webUI conformance test class extends from ConformanceActor. This abstract class calls three methods, overridden in the subclass, in its construtor: composeToolkitBaseUrl() setupSpi() setupSim()

1. composeToolkitBaseUrl composes the toolkit Url with the port number to send SPI commands to.

2. setupSpi creates a new instance of the SimulatorBuilder class.

3. setupSim deletes any old simulator actor with the same name and creates a new simulator actor that will be used for the conformance actor tests.

4. When the constructor finished the setup, all actor specific tests will be run in a sequantial order starting with the base class first.

(It is required because the UI steps need to be executed in a particular order just as a user would navigate through the application.)

5 a. The Simulators page will be checked to see if the name of the newly created sim exists in the page.

5 b. The conformance page will be opened to select the newly created sim in the Test Context box.

Class annotations

The test method ordering is managed throught the @Stepwise class level annotation. The @Timeout(120) annotation limits the entire execution of this test to 5 minutes.

Actor specific steps

For example the repository actor has

1. def 'Get repository conformance actor page.' This method will retrive the conformance actor web page using the conformance GWT place URL method.
2. def 'No weird popup or error message presented in a dialog box.' This method looks for any unexcepted alert type popup messages using the HTML element class name.
3. def 'Check Conformance page loading status and its title.' This method looks for the Initializing and the Loading messages that are displayed while loading the conformance actor page.
4. def 'Click Reset (or Initialize) Environment using defaults.' This method clicks the Reset checkbox in the orchestrated test environment (OTE) panel.
5. def 'Click Initialize.' This method clicks the Initialize button in the OTE panel.
6. def 'Update register endpoint using the supporting sim.' According to the Repository OTE instructions, the SUT must to configured to use the registry from the OTE and then click Reset again.
7. def 'Get (again) repository conformance actor page.' This method reloads the page again to have a fresh start.
8. def 'Click Reset Environment (again) using defaults after updating the Register endpoint.' According to the orchestration environment instructions, this method resets the environment.
9. def 'Find and Click the RunAll Test Registry Conformance Actor image button.' This method will run all conformance tests for the SUT.
10. def 'Number of failed tests count should be zero.' This method looks for the failed test count cell by the 'testFail' CSS class in the HTML test summary table.

Viewing the test output

Since all of the automated tests are run behind the scenes, viewing the log output is critical to troubleshooting.

The log output of the server startup and the spock tests are saved off to the disk in the webui-tests module's target folder. For example, on a Windows machine, it would look like this: C:\tk\webui-tests\target\2017_07_03.stderrout.log.

Logging configuration

See webui-tests\src\test\etc\jetty-logging.xml

and

The jetty-maven-plugin bound to the pre-integration-test and post-integration-test phases of the maven build lifecycle in the webui-tests maven module POM file.