From 9be94bc49de1a40a3591fa2e8736c737c1136df1 Mon Sep 17 00:00:00 2001 From: Valentino Pinna Date: Thu, 18 Apr 2024 09:20:35 +0200 Subject: [PATCH] Create Guidelines.md --- Guidelines.md | 346 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 346 insertions(+) create mode 100644 Guidelines.md diff --git a/Guidelines.md b/Guidelines.md new file mode 100644 index 0000000..5043ec7 --- /dev/null +++ b/Guidelines.md @@ -0,0 +1,346 @@ +# Contents +1. [Introduction](#introduction) +2. [Error Messages](#error-messages) +3. [Serialization of VTL artifacts](#serialization-of-vtl-artifacts) +4. [VTL web services](#vtl-web-services) + +# Introduction + +## Guidelines version + +These technical guidelines address implementation issues for VTL standard version 2.1. + +These guidelines are meant to facilitate developers of VTL tools (engines, interpreters, +translators...) in order to make implementation of such tools easier and speedier. + +For such reasons, these guidelines may frequently change in order to accomodate various +issues that implementors bring to the attention of the VTL task force. + +# Error messages + +VTL error messages are categorized by one or two numbers, separated by a dash. The first number +assigns the error message to one of the general categories. The second number, when indicated, +assigns the error message to a group inside each category. VTL engine implementations may use +any number of additional levels; the meaning of additional classification levels beyond those +listed in this document is implementation dependent. + +The taxonomy of categories (first level) is as follows: + +1. Syntax errors +2. Semantic errors +3. Runtime errors + +## Error message examples + +This section provides a list of template examples for specific error messages. The templates +can be parameterized in order to include information about an actual error. The list is not +intented to be exhaustive, and additional categories may be added at a later time. + +VTL engine implementations may choose to override any template definition and the textual +representation of any eventual parameter in each template. + +Abbreviations are used in the template to signify specific VTL language elements, as follows: + +| Placeholder | Meaning | +|-------------|--------------------------------------------------------------------------------------| +| `%c` | A component (represented with name, role and valuedomain) | +| `%r` | The role of a component (component, identifier, measure, attribute, viral attribute) | +| `%d` | A valuedomain or valuedomain subset | +| `%s` | A data structure or set of unique components | +| `%n` | A name of a dataset, component, ruleset or user-defined operator | +| `%o` | A VTL operator's name or operator group's name | +| `%v` | A VTL scalar or component runtime value | + +### Syntax errors + +A syntax error happens when the VTL script does not exactly match the VTL grammar. Since the syntax check +of the VTL script is usually delegated to a generated parser tool, these errors are directly provided +by the tool that generates the parser. + +These errors prevent the VTL script from being semantically analyzed and should be not present in any one +VTL script that is going to be committed to be used in production. + +Since the grammar is written using the ANTLR4 parser generator syntax, it has become a common choice to +to use this tool to generate the parser for the target language used by a specific VTL engine implementation. +Any possible syntax error raised by the parser is documented at the official +[ANTLR4 documentation web site](https://github.com/antlr/antlr4/blob/master/doc/index.md). + +### Semantic errors + +A semantic error happens when the VTL script is incompatible with the structures or the datasets +referenced; an example is a value from a domain that is incompatible with the component it is +assigned to. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
GroupDescriptionTemplate examples
1Expected exactly one component of a given role and/or domain
Required exactly one %r of domain %d, but found: %s
+Required exactly one %r, but found: %s
2Components of a given role and/or domain are not present in a structure
Required at least one %r of domain %d, but found: %s
+Required at least one %r, but found: %s
3One or more components were requested but are not present
Component %n not found in %s
+Components %n, %n, %n... not found in %s
4Identifiers must be invariant
%o cannot change identifiers %s
+%o cannot change identifier %n to a %r
+%o cannot change %c to an identifier
+%o cannot change identifiers from %s to %s
5Named components are not present in a structure
Components named %n, %n, ... not found in %s
+Component named %n not found in %s
+Components %s not found in %s
+Component %c not found in %s
6An undefined name is used in an expression
Name %n is not bound to this transformation scheme
+In %o, name %n is not defined
7Operator parameters or results have mismatched domains
Incompatible types in %o: %c and %c
+Incompatible types in %o: %c and %d
+Incompatible types in %o: %d and %c
+Incompatible types in %o: %d and %d
+ +### Runtime errors + +A runtime error happens during the processing of invalid data or when performing an illegal +operation, i.e. division by 0 or casting errors, square root or logarithm of negative numbers. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
GroupDescriptionTemplate examples
1Casting a scalar value or component to a invalid type
Cannot cast %v of %d to valuedomain %d
+Component %n cannot accept a value of %v
2Mismatching user-defined operator parameters and arguments
Argument %v of type %d does not match parameter %n
3Error reading data from a source
Error fetching data contents of %n
4Platform-related error
A system error occurred: message
+ +# Serialization of VTL artifacts + +Sometimes there may be a need to exchange VTL artifacts when operating in H2M or M2M mode. +This section defines a JSON schema that aims to provide an easy implementation of this +serialization mechanism. + +The schema encompasses all VTL artifacts included in the VTL User Manual section "Generic Model +for Variables and Value domains". VTL engines may not implement this specific scheme and/or +they may provide support for alternative or extended serialization schemes. All of the four +sections of the JSON schema are optional if the implementation does not require them. + +## JSON scheme for VTL metadata + +```json +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "description": "VTL Metadata JSON serialization", + "$defs": { + "vtl-id": { + "type": "string", + "pattern": "^[a-zA-Z][a-zA-Z0-9_]*$|^'.*'$" + }, + "set-type": { + "type": "array", + "uniqueItems": true, + "oneOf": [ + { "items": { "oneOf": [ { "type": "string" }, { "type": "null" } ] } }, + { "items": { "oneOf": [ { "type": "number" }, { "type": "null" } ] } } + ] + }, + "identifiable": { + "type": "object", + "properties": { + "name": { "$ref": "#/$defs/vtl-id" }, + "description": { "type": "string" } + }, + "required": [ "name" ] + } + }, + "type": "object", + "properties": { + "datasets": { + "type": "array", + "items": { + "allOf": [ { "$ref": "#/$defs/identifiable" } ], + "properties": { + "source": { "type": "string" }, + "structure": { "$ref": "#/$defs/vtl-id" } + }, + "required": [ "structure" ] + } + }, + "structures": { + "type": "array", + "items": { + "allOf": [ { "$ref": "#/$defs/identifiable" } ], + "properties": { + "components": { + "type": "array", + "items": { + "allOf": [ { "$ref": "#/$defs/identifiable" } ], + "properties": { + "role": { + "type": "string", + "enum": [ "Identifier", "Measure", "Attribute", "Viral Attribute" ] + }, + "subset": { "$ref": "#/$defs/vtl-id" }, + "nullable": { "type": "boolean" }, + "data_type": { + "type": "string", + "enum": [ "String", "Number", "Integer", "Boolean", "Time", "TimePeriod", "Date", "Duration" ] + } + }, + "required": [ "role" ] + } + } + }, + "required": [ "components" ] + } + }, + "variables": { + "type": "array", + "items": { + "allOf": [ { "$ref": "#/$defs/identifiable" } ], + "properties": { + "domain": { "$ref": "#/$defs/vtl-id" } + }, + "required": [ "domain" ] + } + }, + "domains": { + "type": "array", + "items": { + "allOf": [ { "$ref": "#/$defs/identifiable" } ], + "unevaluatedProperties": false, + "oneOf": [ + { + "properties": { + "externalRef": { "type": "string" } + }, + "required": [ "externalRef" ] + }, { + "properties": { + "parent": { "$ref": "#/$defs/vtl-id" } + }, + "required": [ "parent" ], + "oneOf": [{ + "properties": { + "restriction": { "$ref": "#/$defs/set-type" } + }, + "required": [ "restriction" ] + }, { + "properties": { + "enumerated": { "$ref": "#/$defs/set-type" } + }, + "required": [ "enumerated" ] + }, { + "properties": { + "described": { "type": "string" } + }, + "required": [ "described" ] + } + ] + } + ] + } + } + } +} +``` + +# VTL web services + +This chapter describes guidelines for implementors of web service applications providing access +to VTL engines, or VTL engines exposing web services. + +## Architecture + +A VTL engine may expose a set of web services, or an application layer may support interoperability +with a VTL engine instance located elsewhere. In the latter case, the web service may use any +technology supported by the engine instance, such as JSON-RPC or CORBA, to transfer control between +the engine and the application layer. + +A VTL web service implementation should provide ways to load VTL data, perform execution of a scheme +and retrieve the results. + +The payload and the results of calls to a VTL web service (where applicable) should be encoded in UTF-8 +without the byte-order mark (BOM). + +## Storage + +Web services implementations may choose to rely on external storage technology in order to provide +data to the submitted request. Web services implementations may additionally support user storage of +VTL artifacts. In this case any details regarding how to achieve this, and the list of required HTTP +endpoints, are left to each implementation and must be documented. + +Web services implementations should be able to map VTL aliases used in the submitted program to specific +data sources, either internal or external. + +## Security + +Security must be implemented when the web service implementation requires an user context in order to +process the submitted VTL program. In this case, the web application container must be configured to +forward the authentication token to the web service, or the application must support an authentication +and authorization mechanism, such as Basic or Negotiate. HTTP headers should be used to provide those +tokens. + +Encryption of communication between the user and the web service should be enforced by deploying +the implementation behind a reverse proxy or on an application server that is configured to +provide a X509 certificate with the https:// protocol.