✨ Schema validation (#1467)

This is the implementation of the discussion
#1451
and a follow-up on #1441.

The PR adds schema validation to Sphinx-Needs that
supports local validation and network validation.
The Sphinx-Needs internal data type representation is not changed yet,
all types are still strings. This will come shortly after merge.
Users can already try out the new interface and provide feedback.

Differences to PR 1441:
- Aligning with standards in JSON schema for re-using subschemas via
`$defs` and `$ref`
- Fully typed implementation including runtime checks of valid schema
user input
- Auto inject the default string type if not given
- Replace `trigger_schema` with `select` which aligns with query
language terminology
- Replace `trigger_schema_id` with the mentioned `$ref` mechanism
- New schemas root key `validate` with sub-keys `local` and `network`
for the 2 validation types
- Network validation items does not allow the `select` key anymore as
the selection happens by linking target needs.
  This cleans up an ambiguity in the other PR.
- Network validation errors now bubble up to the root json schema and
are displayed to see exactly why the chain fails
- More network rule types for better control over debug schema output
- Rewrite test cases to use a declarative definition as yaml, so all
pieces can be given in one place:
  - conf.py
  - ubproject.toml
  - index.rst
  - schemas.json
  - expected ontology warnings
- Simplified the code logic
- String patterns (regex) are constrained to a basic subset that works
across engines
- Added docs
  - Examples and explanations
- Comparison with `needs_warnings` and `needs_constraints` and migration
path
- Many more test cases
- `items` with `minItems` and `maxItems` and `contains` with
`minContains` and `maxContains` are now semantically equivalent to JSON
schema spec

---------

Co-authored-by: Chris Sewell <[email protected]>

Author: ubmarco

.pre-commit-config.yaml

@@ -29,6 +29,7 @@ repos:
           - types-docutils==0.20.0.20240201
           - types-jsonschema
           - types-requests
+          - typeguard
 
   # TODO this does not work on pre-commit.ci
   # - repo: https://github.com/astral-sh/uv-pre-commit

docs/configuration.rst

@@ -2316,3 +2316,195 @@ needs_debug_filters
 
 If set to ``True``, all calls to :ref:`filter processing <filter>` will be logged to a ``debug_filters.jsonl`` file in the build output directory,
 appending a single-line JSON for each filter call.
+
+.. _`needs_schema_definitions`:
+
+needs_schema_definitions
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 6.0.0
+
+Defines validation schemas for needs using a definition format derived from JSON Schema.
+Schemas can be used to validate need fields, enforce constraints, and ensure data consistency.
+
+Default value: ``{}``
+
+.. code-block:: python
+
+   needs_schema_definitions = {
+       "$defs": {
+           "type-spec": {
+               "properties": {"type": {"const": "spec"}}
+           },
+           "safe-need": {
+               "properties": {"asil": {"enum": ["A", "B", "C", "D"]}},
+               "required": ["asil"]
+           }
+       },
+       "schemas": [
+           {
+               "id": "unique-id-validation",
+               "severity": "warning",
+               "message": "ID must be uppercase",
+               "validate": {
+                   "local": {
+                       "properties": {
+                           "id": {"pattern": "^[A-Z0-9_]+$"}
+                       }
+                   }
+               }
+           },
+           {
+               "id": "safe-spec",
+               "validate": {
+                   "local": {
+                       "allOf": [
+                           {"$ref": "#/$defs/type-spec"},
+                           {"$ref": "#/$defs/safe-need"}
+                       ]
+                   }
+               }
+           }
+       ]
+   }
+
+See :ref:`schema_validation` for detailed documentation.
+
+.. _`needs_schema_definitions_from_json`:
+
+needs_schema_definitions_from_json
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 6.0.0
+
+Path to a JSON file containing schema definitions. This is the recommended approach for
+defining schemas as it provides better IDE support and maintainability.
+
+Default value: ``None``
+
+.. code-block:: python
+
+   needs_schema_definitions_from_json = "schemas.json"
+
+The JSON file should contain the same structure as :ref:`needs_schema_definitions`:
+
+.. _`needs_schema_severity`:
+
+needs_schema_severity
+~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 6.0.0
+
+Minimum severity level for schema validation reporting.
+Extra option and extra link schema errors are always reported as violations.
+For each entry in :ref:`needs_schema_definitions` the severity can be defined by the user.
+
+Available severity levels:
+
+- ``info``: Informational message (default)
+- ``warning``: Warning message
+- ``violation``: Violation message
+
+The levels align with how `SHACL <https://www.w3.org/TR/shacl/#severity>`__ defines severity levels.
+
+Default value: ``"info"``
+
+.. code-block:: python
+
+   needs_schema_severity = "warning"
+
+.. _`needs_schema_debug_active`:
+
+needs_schema_debug_active
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 6.0.0
+
+Activates debug mode for schema validation. When enabled, the validation dumps JSON files,
+schema files, and validation messages to help troubleshoot schema validation issues.
+
+Default value: ``False``
+
+.. code-block:: python
+
+   needs_schema_debug_active = True
+
+Debug files are written to the directory specified by :ref:`needs_schema_debug_path`.
+
+.. _`needs_schema_debug_path`:
+
+needs_schema_debug_path
+~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 6.0.0
+
+Directory path where schema debug files are stored when :ref:`needs_schema_debug_active` is
+enabled.
+
+If the path is relative, it will be resolved relative to the ``conf.py`` directory.
+
+Default value: ``"schema_debug"``
+
+.. code-block:: python
+
+   needs_schema_debug_path = "debug/schemas"
+
+.. _`needs_schema_debug_ignore`:
+
+needs_schema_debug_ignore
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 6.0.0
+
+List of validation scenarios to ignore when dumping debug information. This helps reduce
+noise in debug output by filtering out irrelevant validations.
+
+Default value::
+
+  [
+     "extra_option_success",
+     "extra_link_success",
+     "select_success",
+     "select_fail",
+     "local_success",
+     "network_local_success"
+  ]
+
+.. code-block:: python
+
+   needs_schema_debug_ignore = [
+       "extra_option_success",
+       "local_success",
+       "network_local_success"
+   ]
+
+To write all scenarios, set it to an empty list: ``[]``.
+
+Available scenarios that can be ignored:
+
+- ``cfg_schema_error``: The user provided schema is invalid
+- ``extra_option_type_error``: A need extra option cannot be coerced to the type specified in the schema
+- ``extra_option_success``: Global extra option validation was successful
+- ``extra_option_fail``: Global extra option validation failed
+- ``extra_link_success``: Global extra link validations was successful
+- ``extra_link_fail``: Global extra link validation failed
+- ``select_success``: Successful select validation
+- ``select_fail``: Failed select validation
+- ``local_success``: Successful local validation
+- ``local_fail``: Need local validation failed
+- ``network_missing_target``: An outgoing link target cannot be resolved
+- ``network_contains_too_few``: minContains or minItems validation failed for the given link_schema link type
+- ``network_contains_too_many``: maxContains or maxItems validation failed for the given link_schema link type
+- ``network_items_fail``: items validation failed for the given link_schema link type
+- ``network_local_success``: Successful network local validation
+- ``network_local_fail``: Need does not validate against the local schema in a network context
+- ``network_max_nest_level``: The maximum nesting level of network links was exceeded
+
+The debug information is written to the directory specified by :ref:`needs_schema_debug_path`.
+The ``_success`` scenarios exist to analyze why a validation was successful and how the
+final need and schema looks like.
+
+.. note::
+
+   For large need counts, the debug output can become very large.
+   Writing debug output also affects the validation performance negatively.

docs/index.rst

@@ -208,6 +208,7 @@ Contents
    services/index
    layout_styles
    api
+   schema/index
    utils
 
 .. toctree::