feat!: introduce confidence scores for check facts (#620)

This PR changes the data model and allows specifying confidence scores for check results, which is especially useful when a check reports multiple candidate results. All of these confidence scores are added to the check tables in the database. However, the fact that has the highest confidence is shown in the HTML/JSON report only.

The justifications are no longer required to be added manually to the CheckResultData. Instead, they are curated directly from the results in the table. If a column has specified JustificationType in the column mapping, it will be picked up automatically and rendered as plain text or href depending on the specified type. If a check fails or is skipped, we show a default Not Available. justification. This allows to create HTML/JSON reports from the database reproducibly.

Signed-off-by: behnazh-w <[email protected]>

Author: behnazh-w

docs/source/assets/er-diagram.svg

@@ -1,4 +1,4 @@
-.. Copyright (c) 2023 - 2023, Oracle and/or its affiliates. All rights reserved.
+.. Copyright (c) 2023 - 2024, Oracle and/or its affiliates. All rights reserved.
 .. Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/.
 
 =========================
@@ -11,6 +11,213 @@ To follow the project's code style, see the :doc:`Macaron Style Guide </pages/de
 
 For API reference, see the :doc:`API Reference </pages/developers_guide/apidoc/index>` page.
 
+-------------------
+Writing a New Check
+-------------------
+
+Contributors to Macaron are very likely to need to write a new check or modify an existing one at some point. In this
+section, we will explain how Macaron checks work. We will also show how to develop a new check.
+
++++++++++++++++++
+High-level Design
++++++++++++++++++
+
+Before jumping into coding, it is useful to understand how Macaron as a framework works. Macaron is an extensible
+framework designed to make writing new supply chain security analyses easy. It provides an interface
+that you can leverage to access existing models and abstractions instead of implementing everything from scratch. For
+instance, many security checks require traversing through the code in GitHub Actions configurations. Normally,
+you would need to find the right repository and commit, clone it, find the workflows, and parse them. With Macaron,
+you don't need to do any of that and can simply write your security check by using the parsed shell scripts that are
+triggered in the CI.
+
+Another important aspect of our design is that all the check results are automatically mapped and stored in a local database.
+By performing this mapping, we make it possible to enforce use case-specific policies on the results of the checks. While storing
+the check results in the database happens automatically in Macaron's backend, the developer needs to add a brief specification
+to make that possible as we will see later.
+
+Once you get familiar with writing a basic check, you can explore the check dependency feature in Macaron. The checks
+in our framework can be customized to only run if another check has run and returned a specific
+:class:`result type <macaron.slsa_analyzer.checks.check_result.CheckResultType>`. This feature can be used when checks
+have an ordering and a parent-child relationship, i.e., one check implements a weaker or stronger version of a
+security property in a parent check. Therefore, it might make sense to skip running the check and report a
+:class:`result type <macaron.slsa_analyzer.checks.check_result.CheckResultType>` based on the result of the parent check.
+
++++++++++++++++++++
+The Check Interface
++++++++++++++++++++
+
+Each check needs to be implemented as a Python class in a Python module under ``src/macaron/slsa_analyzer/checks``.
+A check class should subclass the :class:`BaseCheck <macaron.slsa_analyzer.checks.base_check.BaseCheck>` class.
+
+The main logic of a check should be implemented in the :func:`run_check <macaron.slsa_analyzer.checks.base_check.BaseCheck.run_check>` abstract method. It is important to understand the input
+parameters and output objects computed by this method.
+
+.. code-block: python
+    def run_check(self, ctx: AnalyzeContext) -> CheckResultData:
+
+''''''''''''''''
+Input Parameters
+''''''''''''''''
+
+The :func:`run_check <macaron.slsa_analyzer.checks.base_check.BaseCheck.run_check>` method is a callback called by our checker framework. The framework pre-computes a context object,
+:class:`ctx: AnalyzeContext <macaron.slsa_analyzer.analyze_context.AnalyzeContext>` and makes it available as the input
+parameter to the function. The ``ctx`` object contains various intermediate representations and models as the input parameter.
+Most likely, you will need to use the following properties:
+
+* :attr:`component <macaron.slsa_analyzer.analyze_context.AnalyzeContext.component>`
+* :attr:`dynamic_data <macaron.slsa_analyzer.analyze_context.AnalyzeContext.dynamic_data>`
+
+The :attr:`component <macaron.slsa_analyzer.analyze_context.AnalyzeContext.component>`
+object acts as a representation of a software component and contains data, such as it's
+corresponding :class:`Repository <macaron.database.table_definitions.Repository>` and
+:data:`dependencies <macaron.database.table_definitions.components_association_table>`.
+Note that :attr:`component <macaron.slsa_analyzer.analyze_context.AnalyzeContext.component>` will also be stored
+in the database and its attributes, such as :attr:`repository <macaron.database.table_definitions.Component.repository>`
+are established as database relationships. You can see the existing tables and their relationships
+in our :mod:`data model <macaron.database.table_definitions>`.
+
+The :attr:`dynamic_data <macaron.slsa_analyzer.analyze_context.AnalyzeContext.dynamic_data>` property would be particularly useful as it contains
+data about the CI service, artifact registry, and build tool used for building the software component.
+Note that this object is a shared state among checks. If a check runs before another check, it can
+make changes to this object, which will be accessible to the checks run subsequently.
+
+''''''
+Output
+''''''
+
+The :func:`run_check <macaron.slsa_analyzer.checks.base_check.BaseCheck.run_check>` method returns a :class:`CheckResultData <macaron.slsa_analyzer.checks.check_result.CheckResultData>` object.
+This object consists of :attr:`result_tables <macaron.slsa_analyzer.checks.check_result.CheckResultData.result_tables>` and
+:attr:`result_type <macaron.slsa_analyzer.checks.check_result.CheckResultData.result_type>`.
+The :attr:`result_tables <macaron.slsa_analyzer.checks.check_result.CheckResultData.result_tables>` object is the list of facts generated from the check. The :attr:`result_type <macaron.slsa_analyzer.checks.check_result.CheckResultData.result_type>`
+value shows the final result type of the check.
+
++++++++
+Example
++++++++
+
+In this example, we show how to add a check to determine if a software component has a source-code repository.
+Note that this is a simple example to just demonstrate how to add a check from scratch.
+Feel free to explore other existing checks under ``src/macaron/slsa_analyzer/checks`` for more examples.
+
+As discussed earlier, each check needs to be implemented as a Python class in a Python module under ``src/macaron/slsa_analyzer/checks``.
+A check class should subclass the :class:`BaseCheck <macaron.slsa_analyzer.checks.base_check.BaseCheck>` class.
+
+'''''''''''''''
+Create a module
+'''''''''''''''
+First create a module called ``repo_check.py`` under ``src/macaron/slsa_analyzer/checks``.
+
+
+''''''''''''''''''''''''''''
+Add a class for the database
+''''''''''''''''''''''''''''
+
+* Add a class that subclasses :class:`CheckFacts <macaron.database.table_definitions.CheckFacts>` to map your outputs to a table in the database. The class name should follow the ``<MyCheck>Facts`` pattern.
+* Specify the table name in the ``__tablename__`` class variable. Note that the table name should start with ``_`` and it should not have been used by other checks.
+* Add the ``id`` column as the primary key where the foreign key is ``_check_facts.id``.
+* Add columns for the check outputs that you would like to store in the database. If a column needs to appear as a justification in the HTML/JSON report, pass ``info={"justification": JustificationType.<TEXT or HREF>}`` to the column mapper.
+* Add ``__mapper_args__`` class variable and set ``"polymorphic_identity"`` key to the table name.
+
+.. code-block:: python
+
+   # Add this line at the top of the file to create the logger object if you plan to use it.
+   logger: logging.Logger = logging.getLogger(__name__)
+
+
+   class RepoCheckFacts(CheckFacts):
+       """The ORM mapping for justifications in the check repository check."""
+
+       __tablename__ = "_repo_check"
+
+       #: The primary key.
+       id: Mapped[int] = mapped_column(ForeignKey("_check_facts.id"), primary_key=True)
+
+       #: The Git repository path.
+       git_repo: Mapped[str] = mapped_column(String, nullable=True, info={"justification": JustificationType.HREF})
+
+       __mapper_args__ = {
+           "polymorphic_identity": "_repo_check",
+       }
+
+'''''''''''''''''''
+Add the check class
+'''''''''''''''''''
+
+Add a class for your check that subclasses :class:`BaseCheck <macaron.slsa_analyzer.checks.base_check.BaseCheck>`,
+provide the check details in the initializer method, and implement the logic of the check in
+:func:`run_check <macaron.slsa_analyzer.checks.base_check.BaseCheck.run_check>`.
+
+A ``check_id`` should match the ``^mcn_([a-z]+_)+([0-9]+)$`` regular expression, which means it should meet the following requirements:
+
+    - The general format: ``mcn_<name>_<digits>``.
+    - Use lowercase alphabetical letters in ``name``. If ``name`` contains multiple words, they must be separated by underscores.
+
+You can set the ``depends_on`` attribute in the initializer method to declare such dependencies. In this example, we leave this list empty.
+
+.. code-block:: python
+
+   class RepoCheck(BaseCheck):
+       """This Check checks whether the target software component has a source-code repository."""
+
+       def __init__(self) -> None:
+           """Initialize instance."""
+           check_id = "mcn_repo_exists_1"
+           description = "Check whether the target software component has a source-code repository."
+           depends_on: list[tuple[str, CheckResultType]] = []  # This check doesn't depend on any other checks.
+           eval_reqs = [
+               ReqName.VCS
+           ]  # Choose a SLSA requirement that roughly matches this check from the ReqName enum class.
+           super().__init__(check_id=check_id, description=description, depends_on=depends_on, eval_reqs=eval_reqs)
+
+       def run_check(self, ctx: AnalyzeContext) -> CheckResultData:
+           """Implement the check in this method.
+
+           Parameters
+           ----------
+           ctx : AnalyzeContext
+                 The object containing processed data for the target software component.
+
+           Returns
+           -------
+           CheckResultData
+                 The result of the check.
+           """
+           if not ctx.component.repository:
+               logger.info("Unable to find a Git repository for %s", ctx.component.purl)
+               # We do not store any results in the database if a check fails. So, just leave result_tables empty.
+               return CheckResultData(result_tables=[], result_type=CheckResultType.FAILED)
+
+           return CheckResultData(
+               result_tables=[RepoCheckFacts(git_repo=ctx.component.repository.remote_path, confidence=Confidence.HIGH)],
+               result_type=CheckResultType.PASSED,
+           )
+
+As you can see, the result of the check is returned via the :class:`CheckResultData <macaron.slsa_analyzer.checks.check_result.CheckResultData>` object.
+You should specify a :class:`Confidence <macaron.slsa_analyzer.checks.check_result.Confidence>`
+score choosing one of the :class:`Confidence <macaron.slsa_analyzer.checks.check_result.Confidence>` enum values,
+e.g., :class:`Confidence.HIGH <macaron.slsa_analyzer.checks.check_result.Confidence.HIGH>` and pass it via keyword
+argument :attr:`confidence <macaron.database.table_definitions.CheckFacts.confidence>`. You should choose a suitable
+confidence score based on the accuracy of your check analysis.
+
+'''''''''''''''''''
+Register your check
+'''''''''''''''''''
+
+Finally, you need to register your check by adding it to the :mod:`registry module <macaron.slsa_analyzer.registry>` at the end of your check module:
+
+.. code-block:: python
+
+   registry.register(RepoCheck())
+
+
+'''''''''''''''
+Test your check
+'''''''''''''''
+
+Finally, you can add tests for you check by adding ``tests/slsa_analyzer/checks/test_repo_check.py`` module. Macaron
+uses `pytest <https://docs.pytest.org>`_ and `hypothesis <https://hypothesis.readthedocs.io>`_ for testing. Take a look
+at other tests for inspiration!
+
 .. toctree::
    :maxdepth: 1
 

docs/source/pages/developers_guide/index.rst

@@ -1,4 +1,4 @@
-# Copyright (c) 2023 - 2023, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2023 - 2024, Oracle and/or its affiliates. All rights reserved.
 # Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/.
 
 """
@@ -10,23 +10,31 @@
 
 For table associated with a check see the check module.
 """
-import hashlib
 import logging
 import os
 import string
 from datetime import datetime
 from pathlib import Path
-from typing import Any, Self
+from typing import Any
 
 from packageurl import PackageURL
-from sqlalchemy import Boolean, Column, Enum, ForeignKey, Integer, String, Table, UniqueConstraint
+from sqlalchemy import (
+    Boolean,
+    CheckConstraint,
+    Column,
+    Enum,
+    Float,
+    ForeignKey,
+    Integer,
+    String,
+    Table,
+    UniqueConstraint,
+)
 from sqlalchemy.orm import Mapped, mapped_column, relationship
 
 from macaron.database.database_manager import ORMBase
 from macaron.database.rfc3339_datetime import RFC3339DateTime
-from macaron.errors import CUEExpectationError, CUERuntimeError, InvalidPURLError
-from macaron.slsa_analyzer.provenance.expectations.cue import cue_validator
-from macaron.slsa_analyzer.provenance.expectations.expectation import Expectation
+from macaron.errors import InvalidPURLError
 from macaron.slsa_analyzer.slsa_req import ReqName
 
 logger: logging.Logger = logging.getLogger(__name__)
@@ -415,6 +423,16 @@ class CheckFacts(ORMBase):
     #: The primary key.
     id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)  # noqa: A003
 
+    #: The confidence score to estimate the accuracy of the check fact. This value should be in the range [0.0, 1.0] with
+    #: a lower value depicting a lower confidence. Because some analyses used in checks may use
+    #: heuristics, the results can be inaccurate in certain cases.
+    #: We use the confidence score to enable the check designer to assign a confidence estimate.
+    #: This confidence is stored in the database to be used by the policy. This confidence score is
+    #: also used to decide which evidence should be shown to the user in the HTML/JSON report.
+    confidence: Mapped[float] = mapped_column(
+        Float, CheckConstraint("confidence>=0.0 AND confidence<=1.0"), nullable=False
+    )
+
     #: The foreign key to the software component.
     component_id: Mapped[int] = mapped_column(Integer, ForeignKey("_component.id"), nullable=False)
 
@@ -437,62 +455,6 @@ class CheckFacts(ORMBase):
     }
 
 
-class CUEExpectation(Expectation, CheckFacts):
-    """ORM Class for an expectation."""
-
-    # TODO: provenance content check should store the expectation, its evaluation result,
-    # and which PROVENANCE it was applied to rather than only linking to the repository.
-
-    __tablename__ = "_expectation"
-
-    #: The primary key, which is also a foreign key to the base check table.
-    id: Mapped[int] = mapped_column(ForeignKey("_check_facts.id"), primary_key=True)  # noqa: A003
-
-    #: The polymorphic inheritance configuration.
-    __mapper_args__ = {
-        "polymorphic_identity": "_expectation",
-    }
-
-    @classmethod
-    def make_expectation(cls, expectation_path: str) -> Self | None:
-        """Construct a CUE expectation from a CUE file.
-
-        Note: we require the CUE expectation file to have a "target" field.
-
-        Parameters
-        ----------
-        expectation_path: str
-            The path to the expectation file.
-
-        Returns
-        -------
-        Self
-            The instantiated expectation object.
-        """
-        logger.info("Generating an expectation from file %s", expectation_path)
-        expectation: CUEExpectation = CUEExpectation(
-            description="CUE expectation",
-            path=expectation_path,
-            target="",
-            expectation_type="CUE",
-        )
-
-        try:
-            with open(expectation_path, encoding="utf-8") as expectation_file:
-                expectation.text = expectation_file.read()
-                expectation.sha = str(hashlib.sha256(expectation.text.encode("utf-8")).hexdigest())
-                expectation.target = cue_validator.get_target(expectation.text)
-                expectation._validator = (  # pylint: disable=protected-access
-                    lambda provenance: cue_validator.validate_expectation(expectation.text, provenance)
-                )
-        except (OSError, CUERuntimeError, CUEExpectationError) as error:
-            logger.error("CUE expectation error: %s", error)
-            return None
-
-        # TODO remove type ignore once mypy adds support for Self.
-        return expectation  # type: ignore
-
-
 class Provenance(ORMBase):
     """ORM class for a provenance document."""
 

src/macaron/database/table_definitions.py

@@ -1,12 +1,12 @@
-# Copyright (c) 2023 - 2023, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2023 - 2024, Oracle and/or its affiliates. All rights reserved.
 # Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/.
 
 """Generate souffle datalog for policy prelude."""
 
 import logging
 import os
 
-from sqlalchemy import Column, MetaData, Table
+from sqlalchemy import Column, Float, MetaData, Table
 from sqlalchemy.sql.sqltypes import Boolean, Integer, String, Text
 
 logger: logging.Logger = logging.getLogger(__name__)
@@ -81,6 +81,8 @@ def column_to_souffle_type(column: Column) -> str:
         souffle_type = "symbol"
     elif isinstance(sql_type, Integer):
         souffle_type = "number"
+    elif isinstance(sql_type, Float):
+        souffle_type = "number"
     elif isinstance(sql_type, Text):
         souffle_type = "symbol"
     elif isinstance(sql_type, Boolean):