Skip to content

SCL Validator Testing Guide

Jump to bottom
Armel Chausse edited this page Apr 14, 2022 · 11 revisions

Introduction

This guide aims at explaining the testing principles for the SCL validator CLI tool. The tests covered in this document are automatically run by RiseClipse's CI pipeline, but this guide will show you how to run these tests locally and update them if needed.

Requirements

First off, verify that the environment in which you plan to run the tests is UNIX based, and that Bash is installed (this will be needed to run the test shell scripts).
You will then need to have a local copy of the riseclipse-validator-scl2003 and riseclipse-validator-scl2003-test repositories.
You will also need a working java installation, as the tests will run a java -jar command on the SCL validator fatjar.

Speaking of the SCL validator, you will need to have it built locally in order to run the tests.
The resulting RiseClipseValidatorSCL-*version*.jar should be located in the riseclipse-validator-scl2003/fr.centralesupelec.edf.riseclipse.iec61850.scl.validator/target directory.

Special instructions for OS X systems

The scripts described in this documentation use GNU getopt to parse options from the command line.
As OS X getopt does not support long options and has a slightly different syntax compared to GNU getopt, you should install GNU getopt on your system.
See this Stackoverflow answer for detailed instructions to install GNU getopt on OS X.

Tests structure

Below you can find the test directory structure:

riseclipse-validators-test/
├── nsd
│   └── ...
├── ocl
│   └── ...
├── scl
│   ├── input
│   │   ├── *.conf.json
│   │   └── *.xml
│   └── snapshots
│       └── *.out
├── scripts
│   ├── ...
│   ├── update_nsd.sh
│   └── update_ocl.sh
├── run_tests.py
├── update_snapshots.py
└── utils.py

The input folder in the scl directory is where you will place the SCL files that need to be validated (example: myfile.xml) along with their configuration (example: myfile.conf.json), and the snapshots folder will contain the validator output for each input file.

The use for the remaining files of this directory will be detailed in the testing guide.

Testing guide

Snapshot setup

Upon adding a new file to be validated in the scl/input folder, there will be no corresponding snapshot in the scl/snapshots folder.
This is where the Python script run_tests.py comes into play. It has two purposes:

  1. Compare the SCL validator output for each input file (using its config defined in a .conf.json file) with its corresponding snapshot in the scl/snapshots folder
  2. Create a new snapshot based on the validator output for each input file that has no existing snapshot

Now let's say you added a test.xml file in the scl/input folder. Your directory structure should look like this:

riseclipse-validators-test/
├── ...
├── scl
│   ├── input
│   │   └── test.xml
├── run_tests.py
├── update_snapshots.py
└── utils.py

As you can see there is no scl/snapshots/test.out file yet, that is because we need to generate it.
In order to do this, we will use the run_tests.py Python script. In your terminal, run:

python run_tests.py

This will run tests for all SCL files (you can also provide specific paths to specific SCL files as an argument to run_tests.py), and create snapshots for the tests that did not have one yet.
If your installation is working correctly, you should get an output of this kind:

Test command: /path/to/riseclipse-validator-scl2003-test/scripts/run_tests.sh -o -n "/path/to/riseclipse-validator-scl2003-test/scl/input/test.xml"
Running tests... Done !

=== Created snapshots: 1 ===
> /path/to/riseclipse-validator-scl2003-test/scl/snapshots/ICD_test.out

And now your directory tree should look like this:

riseclipse-validators-test/
├── ...
├── scl
│   ├── input
│   │   └── test.xml
│   ├── snapshots
│   │   └── test.out
├── run_tests.py
├── update_snapshots.py
└── utils.py

Congratulations, you just created your first snapshot!
Note that we did not use any configuration file alongside the test.xml files, which means the entirety of the ocl and nsd folders is used to validate the file, which is likely not what you want. Refer to the next section for more information on configuration files.

If you plan on integrating the test you just added to the production test suite, you should commit the input and snapshot files and make a pull request with your changes. Once your changes are integrated, your test will be checked at each CI pipeline run.

Using configuration files

As discussed earlier, it is possible to provide specific configuration for each SCL file.
When you add a new SCL file in the scl/input folder (scl/input/test.xml), add a new JSON file with the same name as the SCL file but replace .xml by .conf.json (scl/input/test.conf.json).

Your directory structure should look like this:

riseclipse-validators-test/ ├── ... ├── scl │ ├── input │ │ ├── test.conf.json │ │ └── test.xml ├── run_tests.py ├── update_snapshots.py └── utils.py

Configuration files format

.conf.json files must follow the following format:

{
    "ocl": "*" | ["dir1", "dir2"],
    "nsd": "*" | ["dir3", "dir4"],
    # All options below are optional and will be passed directly to the SCL validator JAR.
    # Please refer to its help command for more information about these options.
    "verbose": 1,
    "info": 1,
    "warning": 1,
    "error": 1,
    "use-color": 1,
    "make-explicit-links": 1,
    "display-nsd-messages": 1,
    "do-not-display-copyright": 1,
    "help-environment": 1
}

Note that the paths specified in the ocl and nsd fields are relative to the ocl and nsd folder respectively.
If the ocl (resp. nsd) field is omitted, no OCL (resp. NSD) validation will be performed.
To validate using all OCL (resp. NSD) files, provide "*" as value to the ocl (resp. nsd) field. For example, to include all OCL files under the ocl/RequiredAttributes directory, the configuration file would look like this:

{
    "ocl": ["RequiredAttributes"]
}

And after running python run_tests.py you should see something like:

Test command: /path/to/riseclipse-validator-scl2003-test/scripts/run_tests.sh -oRequiredAttributes   "/path/to/riseclipse-validator-scl2003-test/scl/input/test.xml"
Running tests... Done !

=== Created snapshots: 1 ===
> /path/to/riseclipse-validator-scl2003-test/scl/snapshots/test.out

You can see in the Test command being desplayed on the screen that the RequiredAttributes folder has been passed to the underlying Bash script's -o (OCL) option, which means that our config was parsed correctly.

Testing a new validator version

If the validator was modified, chances are that some tests will not produce the same output as before.
This is where snapshot testing comes in handy: if the output for one of the validated files changes, the run_tests.sh script will let you know about it.

Let's say you have a validator version for which all tests pass. When you run the run_tests.sh script, you should get an output of this kind:

Running tests for the 'scl' format... Done !

=== Passed tests: 1 ===
> scl/input/test.xml

Now let's say you just built a new version of the validator, and it breaks one of your tests.
When you run the test script again, you should get an output of this kind:

Running tests for the 'scl' format... Done !

=== Failed tests: 1 ===

> scl/input/test.xml
        Expected:
INFO:    Copyright (c) 2016-2021 CentraleSupélec & EDF.
INFO:    All rights reserved. This program and the accompanying materials
INFO:    are made available under the terms of the Eclipse Public License v2.0
INFO:    which accompanies this distribution, and is available at
INFO:    https://www.eclipse.org/legal/epl-v20.html
INFO:    
INFO:    This tool is part of RiseClipse.
INFO:    Contributors:
INFO:        Computer Science Department, CentraleSupélec
INFO:        EDF R&D
INFO:    Contacts:
INFO:        [email protected]
INFO:        [email protected]
INFO:    Web site:
INFO:        https://riseclipse.github.io/
INFO:    
INFO:    RiseClipseValidatorSCL version: 1.1.0 a24 (15 october 2021)
INFO:    
        Got:
INFO:    Copyright (c) 2016-2021 CentraleSupélec & EDF.
INFO:    All rights reserved. This program and the accompanying materials
INFO:    are made available under the terms of the Eclipse Public License v2.0
INFO:    which accompanies this distribution, and is available at
INFO:    https://www.eclipse.org/legal/epl-v20.html
INFO:    
INFO:    This tool is part of RiseClipse.
INFO:    Contributors:
INFO:        Computer Science Department, CentraleSupélec
INFO:        EDF R&D
INFO:    Contacts:
INFO:        [email protected]
INFO:        [email protected]
INFO:    Web site:
INFO:        https://riseclipse.github.io/
INFO:    
INFO:    RiseClipseValidatorSCL version: 1.1.1 a25 (15 december 2021)
INFO:

When faced with such a snapshot difference upon running the tests, there are two possibilities:

  1. The new validator is malfunctioning, in which case it needs to be fixed and the snapshot needs to remain unchanged
  2. The new validator output is the expected one, and we need to update the snapshot accordingly

In this example we are in the second case, as the only thing that changed in the output is the version name.
Jump to the next section to see how to properly update snapshots.

Updating snapshots

Let's say we want to update the snapshots to match the validator output for a new version.
That is the purpose of the update_snapshots.sh script, which replaces all snapshots by the current output of the validator for each input file. Note that this implies that all previous snapshots will be overriden, so make sure that all of the new validator outputs are valid before running it!

If you run:

./update_snapshots.sh scl

you should get an output of this kind:

Updating snapshots for the 'scl' format... Done !

=== Updated snapshots: 1 ===
> scl/snapshots/test.out

And now when you run run_tests.sh again:

Running tests for the 'scl' format... Done !

=== Passed tests: 1 ===
> input/test.xml

Congratulations, your snapshots are now up to date!
You will need to update your snapshots at each new validator version, but remember to check the output of run_tests.sh before doing so, to make sure that the new validator output is the expected one.

Testing other formats

If you want to run tests for other formats than SCL, you will need to specify the format to the run_tests.sh and update_snapshots.sh scripts as an argument.
You will also need to place the files you want to test under a *format_name*/input directory in the riseclipse-validators-test repository.
Please refer to the --help option of each script for more information regarding their usage.

Clone this wiki locally