Skip to content

SCL Validator Testing Guide

Jump to bottom
Armel Chausse edited this page Jan 6, 2022 · 11 revisions

Introduction

This guide aims at explaining the testing principles for the SCL validator CLI tool. The tests covered in this document are automatically run by RiseClipse's CI pipeline, but this guide will show you how to run these tests locally and update them if needed.

Requirements

First off, verify that the environment in which you plan to run the tests is UNIX based, and that Bash is installed (this will be needed to run the test shell scripts).
You will then need to have a local copy of the riseclipse-validator-scl2003 and riseclipse-validators-test repositories.
You will also need a working java installation, as the tests will run a java -jar command on the SCL validator fatjar.

Speaking of the SCL validator, you will need to have it built locally in order to run the tests.
The resulting RiseClipseValidatorSCL-*version*.jar should be located in the riseclipse-validator-scl2003/fr.centralesupelec.edf.riseclipse.iec61850.scl.validator/target directory.

Tests structure

Below you can find the test directory structure:

riseclipse-validators-test/
├── nsd
│   ├── input
│   │   └── *.nsd
│   └── snapshots
│       └── *.out
├── ocl
│   ├── input
│   │   └── *.ocl
│   └── snapshots
│       └── *.out
├── run_tests.sh
├── scl
│   ├── input
│   │   └── *.xml
│   └── snapshots
│       └── *.out
├── setup_vars.sh
└── update_snapshots.sh

The input folders in each format directory are where you will place the files that need to be validated, and the snapshots folder will contain the validator output for each input file.

The use for the remaining files of this directory will be detailed in the testing guide.

Testing guide

Snapshot setup

Upon adding a new file to be validated in the input folder, there will be no corresponding snapshot in the snapshots folder.
This is where the Bash script run_tests.sh comes into play. It has two purposes:

  1. Compare the SCL validator output for each input file with its corresponding snapshot in the snapshots folder
  2. Create a new snapshot based on the validator output for each input file that has no existing snapshot

Now let's say you added a test.xml file in the scl/input folder. Your directory structure should look like this:

riseclipse-validators-test/
|   ...
├── run_tests.sh
├── scl
│   ├── input
│   |   └── test.xml
├── setup_vars.sh
└── update_snapshots.sh

As you can see there is no scl/snapshots/other_test.out file yet, that is because we need to generate it.
In order to do this, we will use the run_tests.sh script. In your terminal, run:

./run_tests.sh scl

This will run tests for the SCL format, and create snapshots for the tests that did not have one yet.
If your installation is working correctly, you should get an output of this kind:

Running tests for the 'scl' format... Done !

=== Created snapshots: 1 ===
> scl/snapshots/test.out

And now your directory tree should look like this:

riseclipse-validators-test/
|   ...
├── run_tests.sh
├── scl
│   ├── input
│   |   └── test.xml
│   ├── snapshots
│   |   └── test.out
├── setup_vars.sh
└── update_snapshots.sh

Congratulations, you just created your first snapshot!
If you plan on integrating the test you just added to the production test suite, you should commit the input and snapshot files and make a pull request with your changes. Once your changes are integrated, your test will be checked at each CI pipeline run.

Testing a new validator version

If the validator was modified, chances are that some tests will not produce the same output as before.
This is where snapshot testing comes in handy: if the output for one of the validated files changes, the run_tests.sh script will let you know about it.

Let's say you have a validator version for which all tests pass. When you run the run_tests.sh script, you should get an output of this kind:

Running tests for the 'scl' format... Done !

=== Passed tests: 1 ===
> scl/input/test.xml

Now let's say you just built a new version of the validator, and it breaks one of your tests.
When you run the test script again, you should get an output of this kind:

Running tests for the 'scl' format... Done !

=== Failed tests: 1 ===

> scl/input/test.xml
        Expected:
INFO:    Copyright (c) 2016-2021 CentraleSupélec & EDF.
INFO:    All rights reserved. This program and the accompanying materials
INFO:    are made available under the terms of the Eclipse Public License v2.0
INFO:    which accompanies this distribution, and is available at
INFO:    https://www.eclipse.org/legal/epl-v20.html
INFO:    
INFO:    This tool is part of RiseClipse.
INFO:    Contributors:
INFO:        Computer Science Department, CentraleSupélec
INFO:        EDF R&D
INFO:    Contacts:
INFO:        [email protected]
INFO:        [email protected]
INFO:    Web site:
INFO:        https://riseclipse.github.io/
INFO:    
INFO:    RiseClipseValidatorSCL version: 1.1.0 a24 (15 october 2021)
INFO:    
        Got:
INFO:    Copyright (c) 2016-2021 CentraleSupélec & EDF.
INFO:    All rights reserved. This program and the accompanying materials
INFO:    are made available under the terms of the Eclipse Public License v2.0
INFO:    which accompanies this distribution, and is available at
INFO:    https://www.eclipse.org/legal/epl-v20.html
INFO:    
INFO:    This tool is part of RiseClipse.
INFO:    Contributors:
INFO:        Computer Science Department, CentraleSupélec
INFO:        EDF R&D
INFO:    Contacts:
INFO:        [email protected]
INFO:        [email protected]
INFO:    Web site:
INFO:        https://riseclipse.github.io/
INFO:    
INFO:    RiseClipseValidatorSCL version: 1.1.1 a25 (15 december 2021)
INFO:

When faced with such a snapshot difference upon running the tests, there are two possibilities:

  1. The new validator is malfunctioning, in which case it needs to be fixed and the snapshot needs to remain unchanged
  2. The new validator output is the expected one, and we need to update the snapshot accordingly

In this example we are in the second case, as the only thing that changed in the output is the version name.
Jump to the next section to see how to properly update snapshots.

Updating snapshots

Let's say we want to update the snapshots to match the validator output for a new version.
That is the purpose of the update_snapshots.sh script, which replaces all snapshots by the current output of the validator for each input file. Note that this implies that all previous snapshots will be overriden, so make sure that all of the new validator outputs are valid before running it!

If you run:

./update_snapshots.sh scl

you should get an output of this kind:

Updating snapshots for the 'scl' format... Done !

=== Updated snapshots: 1 ===
> scl/snapshots/test.out

And now when you run run_tests.sh again:

Running tests for the 'scl' format... Done !

=== Passed tests: 1 ===
> input/test.xml

Congratulations, your snapshots are now up to date!
You will need to update your snapshots at each new validator version, but remember to check the output of run_tests.sh before doing so, to make sure that the new validator output is the expected one.

Testing other formats

If you want to run tests for other formats than SCL, you will need to specify the format to the run_tests.sh and update_snapshots.sh scripts as an argument.
You will also need to place the files you want to test under a *format_name*/input directory in the riseclipse-validators-test repository.
Please refer to the --help option of each script for more information regarding their usage.

Clone this wiki locally