Skip to content

Commit 5a6cbdd

Browse files
jobulckejobulcke
authored
feat: processing service (#7)
* feat: small refactor + init of processor service * feat: processing service finished * feat: processing service implemented * feat: bugfixes and cleanup of new service * feat: additional test added * docs: little remark added * fix: invalid logging levels * feat: PR remarks * feat: small improvements * fix: broken ci pipeline
1 parent 8f86901 commit 5a6cbdd

40 files changed

+1159
-397
lines changed

‎README.md

Lines changed: 100 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,74 @@
11
# VSDS TestBed Shacl Validator
22

33
<!-- TOC -->
4-
54
* [VSDS TestBed Shacl Validator](#vsds-testbed-shacl-validator)
6-
* [Introduction](#introduction)
7-
* [Validation service implementation](#validation-service-implementation)
5+
* [Introduction](#introduction)
6+
* [ReplicationProcesssingService](#replicationprocesssingservice)
7+
* [startReplicating](#startreplicating)
8+
* [haltWhenReplicated](#haltwhenreplicated)
9+
* [destroyPipeline](#destroypipeline)
10+
* [ShaclValidationService](#shaclvalidationservice)
11+
* [How to run in Docker](#how-to-run-in-docker)
812
* [Prerequisites](#prerequisites)
9-
* [Building and running](#building-and-running)
10-
* [Live reload for development](#live-reload-for-development)
11-
* [Packaging using Docker](#packaging-using-docker)
12-
13+
* [Steps to use the TestBed Shacl Validator](#steps-to-use-the-testbed-shacl-validator)
14+
* [interact](#interact)
15+
* [call](#call)
16+
* [Building and running the project](#building-and-running-the-project)
17+
* [Live reload for development](#live-reload-for-development)
18+
* [Packaging using Docker](#packaging-using-docker)
1319
<!-- TOC -->
14-
1520
## Introduction
1621

1722
This application implements the [GITB test service APIs](https://www.itb.ec.europa.eu/docs/services/latest/) in a
1823
[Spring Boot](https://spring.io/projects/spring-boot) web application that is meant to support
1924
[GITB TDL test cases](https://www.itb.ec.europa.eu/docs/tdl/latest/) running in the Interoperability Test Bed.
2025

21-
### Validation service implementation
26+
## ReplicationProcesssingService
27+
28+
This service is responsible for creating an LDIO pipeline, checking if the LDES Client is still REPLACTING and
29+
destroying the pipeline afterwards. It is available through the endpoint's
30+
WSDL: http://localhost:8080/services/process?wsdl. This processing service has three operations:
31+
32+
#### startReplicating
33+
34+
This operation is responsible for creating the pipeline, which immediately starts the replication process.
35+
36+
Input parameters:
37+
38+
| Name | Description | Required |
39+
|:--------:|---------------------------------|----------|
40+
| ldes-url | the url of the LDES to validate | true |
41+
42+
#### haltWhenReplicated
43+
44+
This operation is responsible for polling and returning the status of the LDES Client
45+
46+
#### destroyPipeline
47+
48+
This operation is resonsible for deleting the LDIO pipeline, as well the GraphDB repository
49+
50+
More information about processing services can be
51+
found [here](https://www.itb.ec.europa.eu/docs/services/latest/processing/)
52+
53+
## ShaclValidationService
2254

23-
The sample validation service validates a text against an (also provided) expected value. The user of the service can
24-
also select whether he/she wants to have a mismatch reported as an error or a warning. Finally, an information message
25-
is also returned in case values match but when ignoring casing.
55+
This service is responsible for executing the SHACL validation and is accessible through the endpoint's
56+
WSDL: http://localhost:8080/services/validation?wsdl
2657

27-
Once running, the validation endpoint's WDSL is available at http://localhost:8080/services/validation?WSDL. See
28-
[here](https://www.itb.ec.europa.eu/docs/services/latest/validation/) for further information on processing service
29-
implementations.
58+
Input parameters:
59+
60+
| Name | Description | Required |
61+
|:-----------:|---------------------------------------------------------------------|----------|
62+
| shacl-shape | the shacl shape that will be used to validate the server against to | true |
3063

3164
This validation services requires in the validation call two parameters:
3265

3366
1. **ldes-url**: the url of the LDES to validate
3467
2. **shacl-shape**: the shacl shape that will be used to validate the server against to
3568

69+
More information about validation services can be
70+
found [here](https://www.itb.ec.europa.eu/docs/services/latest/validation/)
71+
3672
## How to run in Docker
3773

3874
### Prerequisites
@@ -75,6 +111,8 @@ In this specific tutorial, this would result in the following config:
75111
- SERVER_PORT=8080
76112
```
77113
114+
This service has already been added to the provided `docker-compose.yaml` file.
115+
78116
2. Start up all the services
79117

80118
```shell
@@ -92,10 +130,24 @@ First of all, make a folder that will contain all required files for the test su
92130
```shell
93131
mkdir validate_ldes_to_shacl_shape
94132
cd validate_ldes_to_shacl_shape
133+
```
134+
135+
Secondly, a scriptlet will be added to the test suite, which contains all the required logic to create the necessary
136+
LDIO pipelines, check the LDES Client status and so on.
137+
138+
```shell
139+
mkdir scriptlets
140+
```
141+
142+
In this directory, place the [`validate-ldes.xml`](./docker/scriplets/validate-ldes.xml) file.
143+
144+
Now, the test cases can be written and added to the test suite. First, a new folder must be created.
145+
146+
```shell
95147
mkdir test_cases
96148
```
97149

98-
Secondly, test case xml file can be added to the `test_cases` folder.
150+
Now, `test_case.xml` file can be added to the newly created folder
99151

100152
```xml
101153
@@ -111,20 +163,20 @@ Secondly, test case xml file can be added to the `test_cases` folder.
111163
</actors>
112164
113165
<steps stopOnError="true">
114-
<!-- Prompt the user to enter the required data to run the test -->
115166
<interact id="validationData" desc="Setup parameters to validate the LDES">
116-
<request desc="URL of the LDES to validate" name="ldes"/>
167+
<request desc="URL of the LDES to validate" name="ldesUrl"/>
117168
<request desc="Shacl shape that must be used for validating" name="shaclShape" inputType="UPLOAD"/>
169+
<request desc="Amount of seconds between each polling attempt" name="pollingInterval" inputType="TEXT"/>
118170
</interact>
119-
<log>"Starting shacl validation test"</log>
120-
<!-- Step 3: Check relation timestamp consistency. -->
121-
<!-- Notice here how we refer to the address of the validation service using the domain-level "validationServiceAddress" configuration property. -->
122-
<verify output="validatorOutput" id="shaclValidationStep" desc="validate against shacl"
123-
handler="http://testbed-shacl-validator:8080/services/validation?wsdl">
124-
<input name="ldes-url">$validationData{ldes}</input>
125-
<input name="shacl-shape">$validationData{shaclShape}</input>
126-
</verify>
127-
<log>"shacl verification finished"</log>
171+
<assign to="delayDuration">concat($validationData{pollingInterval}, '000')</assign>
172+
<assign to="addresses{processing}">"http://testbed-shacl-validator:8080/services/process?wsdl"</assign>
173+
<assign to="addresses{validation}">"http://testbed-shacl-validator:8080/services/validation?wsdl"</assign>
174+
<call id="validateLdes" path="scriptlets/validate-ldes.xml">
175+
<input name="ldesUrl">$validationData{ldesUrl}</input>
176+
<input name="shaclShape">$validationData{shaclShape}</input>
177+
<input name="delayDuration">$delayDuration</input>
178+
<input name="addresses">$addresses</input>
179+
</call>
128180
</steps>
129181
130182
<output>
@@ -145,21 +197,27 @@ Some interesting things we see in this test case, are the `interact` block, as w
145197
#### interact
146198

147199
This block is responsible for prompting the user/tester to enter some data that will be used for the test. In this
148-
specific case, it will be the URL of the LDES that must be validated.
200+
specific case, it will be the URL of the LDES that must be validated, a SHACL shape file and a polling interval on which
201+
the LDES Client status must be checked.
202+
203+
| Name of the parameter | Description |
204+
|:---------------------:|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
205+
| ldesUrl | URL of the LDES to be validated |
206+
| SHACL shape | SHACL shape that will be used to validate the LDES |
207+
| pollingInterval | Interval in which the LDES Client status will be checked in seconds. Keep in mind if the LDES to validate contains 1M members, it can take a very long time to replicate. Therefore, it can be better to configure a larger interval (e.g. an hour or even half a day) |
149208

150-
#### verify
209+
#### call
151210

152-
This block is responsible for the verifying the LDES by calling the external service, which is this TestBed Shacl
153-
Validator service. The most important part to notice here is the `handler` attribute, where the external TestBed Shacl
154-
Validator service will be set. In this case, the docker compose service name must be used, followed by the port. After
155-
that, the "fixed" uri is placed, which will call the right validation service, which is in this case
156-
`/services/validation?wsdl`
211+
This block is responsible for executing the scriptlet, which is responsible for creating an LDIO pipeline, checking when
212+
the LDES Client has finished with REPLICATING, performing the SHACL validation and destroying the pipeline afterward.
157213

158-
Another thing that can be noticed here, are the input parameters. These two parameters are required to let the test run.
214+
Another thing that can be noticed here, are the input parameters. These three parameters are required to let the test
215+
run.
159216

160-
1. `ldes-url`: the url of the LDES that must be validated. The value here is fetched from the prompted input
161-
2. `shacl-shape`: the SHACL shape that will be used to validate all the members against to. Here is the value also
217+
1. `ldesUrl`: the url of the LDES that must be validated. The value here is fetched from the prompted input
218+
2. `shaclShape`: the SHACL shape that will be used to validate all the members against to. Here is the value also
162219
fetched from the prompted input
220+
3. `delayDuration`: the amount of seconds on which the LDES Client must be checked
163221

164222
To finish up this step, a test suite itself must be added as well. This can be done by adding `test_suite.xml` to the
165223
`validate_ldes_to_shacl_shape` folder.
@@ -206,11 +264,13 @@ This is how the file should look like:
206264
Most important to notice here, is that de test case declared in the `test_cases` folder, must be referred from this file
207265
by adding the `test case` tag with as attribute the id that has been assigned to the test case in its file.
208266

209-
4. Compress the `validate_ldes_to_shacl_shape` folder to a zip and upload it to TestBed
267+
4. Add the Test Suite to the TestBed instance
268+
269+
Create a ZIP file contains all the items that are in `validate_ldes_to_shacl_shape` folder. This can be uploaded to
270+
TestBed now. This can be done either by the UI, or via the REST API. If you choose to do it via the REST API, the
210271

211-
If everything is set up, the folder containing everything must be zipped. After that, it can be uploaded to TestBed. This can be done either by the UI, or via the REST API. If you choose to do it via the REST API, the
212272
```shell
213-
curl -F updateSpecification=true -F specification=<SPECIFICATION_API_KEY> -F testSuite=@test_shacl_validator.zip --header "ITB_API_KEY: <DOMAIN_API_KEY>" -X POST http://localhost:9000/api/rest/testsuite/deploy;
273+
curl -F updateSpecification=true -F specification=<SPECIFICATION_API_KEY> -F testSuite=@test_shacl_validator.zip --header "ITB_API_KEY: <COMMUNITY_API_KEY>" -X POST http://localhost:9000/api/rest/testsuite/deploy;
214274
```
215275

216276
5. Run the test via the TestBed UI

‎docker/scriplets/validate-ldes.xml

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
<?xml version="1.0" encoding="UTF-8" ?>
2+
<scriptlet id="validate-ldes" xmlns="http://www.gitb.com/tdl/v1/">
3+
<params>
4+
<var name="ldesUrl" type="string"/>
5+
<var name="shaclShape" type="string"/>
6+
<var name="delayDuration" type="number">
7+
<value>60000</value>
8+
</var>
9+
<var name="addresses" type="map" />
10+
</params>
11+
<steps stopOnError="true">
12+
<process output="startupReport" desc="Start Replication" id="startReplicatingProcess"
13+
handler="$addresses{processing}">
14+
<operation>startReplicating</operation>
15+
<input name="ldes-url">$ldesUrl</input>
16+
</process>
17+
<log>"Waiting for the LDES Client status to be available"</log>
18+
<process desc="Wait until client is available" handler="DelayProcessor">
19+
<operation>delay</operation>
20+
<input>5000</input>
21+
</process>
22+
23+
<log>"Start checking LDES Client status"</log>
24+
<assign to="replicationOutput"/>
25+
<repuntil desc="Check if replication has ended">
26+
<do>
27+
<process desc="Wait" handler="DelayProcessor">
28+
<operation>delay</operation>
29+
<input>$delayDuration</input>
30+
</process>
31+
<process output="replicationOutput" handler="$addresses{processing}">
32+
<operation>haltWhenReplicated</operation>
33+
</process>
34+
<log level="DEBUG">$replicationOutput{STATUS}</log>
35+
</do>
36+
<cond>$replicationOutput{STATUS} = 'REPLICATING'</cond>
37+
</repuntil>
38+
<log>"Starting shacl validation"</log>
39+
40+
<verify output="validatorOutput" id="shaclValidationStep" desc="validate against shacl"
41+
handler="$addresses{validation}">
42+
<input name="shacl-shape">$shaclShape</input>
43+
</verify>
44+
<log>"shacl verification finished"</log>
45+
<process desc="Delete pipeline" handler="$addresses{processing}" operation="destroyPipeline"/>
46+
</steps>
47+
<output name="validatorOutput">
48+
$validatorOutput
49+
</output>
50+
</scriptlet>

‎src/main/java/be/vlaanderen/informatievlaanderen/ldes/gitb/ReplicationProcessingService.java

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
package be.vlaanderen.informatievlaanderen.ldes.gitb;
2+
3+
import be.vlaanderen.informatievlaanderen.ldes.gitb.services.replication.ProcessExecutors;
4+
import be.vlaanderen.informatievlaanderen.ldes.gitb.valueobjects.ParameterDefinition;
5+
import be.vlaanderen.informatievlaanderen.ldes.gitb.valueobjects.ProcessParameters;
6+
import be.vlaanderen.informatievlaanderen.ldes.gitb.valueobjects.ProcessResult;
7+
import com.gitb.core.ConfigurationParameters;
8+
import com.gitb.core.Metadata;
9+
import com.gitb.core.TypedParameters;
10+
import com.gitb.ps.Void;
11+
import com.gitb.ps.*;
12+
import org.slf4j.Logger;
13+
import org.slf4j.LoggerFactory;
14+
import org.springframework.stereotype.Component;
15+
16+
@Component
17+
public class ReplicationProcessingService implements ProcessingService {
18+
private static final String SERVICE_NAME = "ReplicationProcessingService";
19+
private static final Logger log = LoggerFactory.getLogger(ReplicationProcessingService.class);
20+
private final ProcessExecutors processExecutors;
21+
22+
public ReplicationProcessingService(ProcessExecutors processExecutors) {
23+
this.processExecutors = processExecutors;
24+
}
25+
26+
@Override
27+
public GetModuleDefinitionResponse getModuleDefinition(Void parameters) {
28+
final ProcessingModule processingModule = new ProcessingModule();
29+
processingModule.setId(SERVICE_NAME);
30+
31+
final Metadata metadata = new Metadata();
32+
metadata.setName(SERVICE_NAME);
33+
processingModule.setMetadata(metadata);
34+
35+
processingModule.setConfigs(new ConfigurationParameters());
36+
37+
processExecutors.getProcessExecutors().stream()
38+
.map(processExecutor -> {
39+
final var processingOperation = new ProcessingOperation();
40+
final var typedParameters = new TypedParameters();
41+
processingOperation.setName(processExecutor.getName());
42+
typedParameters.getParam().addAll(processExecutor
43+
.getParameterDefinitions()
44+
.stream()
45+
.map(ParameterDefinition::convertToTypedParameter)
46+
.toList());
47+
processingOperation.setInputs(typedParameters);
48+
return processingOperation;
49+
})
50+
.forEach(processingModule.getOperation()::add);
51+
52+
final GetModuleDefinitionResponse getModuleDefinitionResponse = new GetModuleDefinitionResponse();
53+
getModuleDefinitionResponse.setModule(processingModule);
54+
return getModuleDefinitionResponse;
55+
}
56+
57+
@Override
58+
public ProcessResponse process(ProcessRequest parameters) {
59+
log.info("Received 'process' command with '{}' operation from test bed for session [{}]", parameters.getOperation(), parameters.getSessionId());
60+
return processExecutors.getProcessExecutor(parameters.getOperation())
61+
.map(processExecutor -> processExecutor.execute(new ProcessParameters(parameters.getSessionId(), parameters.getInput())))
62+
.orElseGet(() -> ProcessResult.invalidOperation(parameters.getOperation()))
63+
.convertToResponse();
64+
}
65+
66+
@Override
67+
public BeginTransactionResponse beginTransaction(BeginTransactionRequest parameters) {
68+
return new BeginTransactionResponse();
69+
}
70+
71+
@Override
72+
public Void endTransaction(BasicRequest parameters) {
73+
return new Void();
74+
}
75+
76+
77+
}

0 commit comments

Comments
 (0)