WIP for Maxi :)

Author: sbernauer

_TEST/CURLS.sh

@@ -0,0 +1,13 @@
+curl 'localhost:9477/metadata/database?system=trino&instance=kuttl-secure-hermit-wf84/my-trino&database=tpch' | jq
+curl 'localhost:9477/metadata/schema?system=trino&instance=kuttl-secure-hermit-wf84/my-trino&database=tpch&schema=sf1' | jq
+curl 'localhost:9477/metadata/table?system=trino&instance=kuttl-secure-hermit-wf84/my-trino&database=tpch&schema=sf1&table=customer' | jq
+curl 'localhost:9477/metadata/stream?system=kafka&instance=kuttl-secure-hermit-wf84/test-kafka&queue=orders' | jq
+curl 'localhost:9477/metadata/dashboard?system=superset&instance=kuttl-secure-hermit-wf84/my-superset&id=1' | jq
+curl 'localhost:9477/metadata/chart?system=superset&instance=kuttl-secure-hermit-wf84/my-superset&id=1' | jq
+
+curl 'localhost:9477/metadata/rawIdentifier?identifier=urn:li:container:792ce6aace288712a1ef036fbca2bb37' | jq
+curl 'localhost:9477/metadata/rawIdentifier?identifier=urn:li:container:ae5d10cb199e26e8df70c7563e6d1c58' | jq
+curl 'localhost:9477/metadata/rawIdentifier?identifier=urn:li:dataset:(urn:li:dataPlatform:trino,kuttl-secure-hermit-wf84/my-trino.tpch.sf1.customer,PROD)' | jq
+curl 'localhost:9477/metadata/rawIdentifier?identifier=urn:li:dataset:(urn:li:dataPlatform:kafka,kuttl-secure-hermit-wf84/test-kafka.orders,PROD)' | jq
+curl 'localhost:9477/metadata/rawIdentifier?identifier=urn:li:dashboard:(superset,kuttl-secure-hermit-wf84/my-superset.1)' | jq
+curl 'localhost:9477/metadata/rawIdentifier?identifier=urn:li:chart:(superset,kuttl-secure-hermit-wf84/my-superset.1)' | jq

docs/modules/opa/pages/usage-guide/resource-info-fetcher.adoc

@@ -2,7 +2,7 @@
 :description: Resource Info Fetcher for OPA retrieves data from backends like DataHub. Integrate extra resource details into Rego rules for enhanced policy management.
 
 The _Resource info fetcher_ allows for additional information to be obtained from the configured backend (for example, DataHub).
-You can then write Rego rules for OpenPolicyAgent which make an HTTP request to the Resource info fetcher and make use of the additional information returned for the resource, such as Trino tables or Kafka topic.
+You can then write Rego rules for OpenPolicyAgent which make an HTTP request to the Resource info fetcher and make use of the additional information returned for the resource, such as Trino tables or Apache Kafka topics.
 
 You can enable the Resource info fetcher sidecar as follows:
 
@@ -45,7 +45,7 @@ stringData:
 <1> Enable the `resource-info-fetcher` sidecar
 <2> Enable TLS verification using the CA from the `tls` SecretClass.
 <3> Authenticate to DataHub with a Personal Access Token (PAT) read from the specified Secret. The Secret must have a `token` entry. See the DataHub documentation on https://docs.datahub.com/docs/authentication/personal-access-tokens[Personal Access Tokens] for how to create one, and make sure https://docs.datahub.com/docs/authentication/introducing-metadata-service-authentication[Metadata Service Authentication] is enabled on your DataHub.
-<4> The DataHub environment (fabric) to query, e.g. `PROD` or `DEV`. Defaults to `PROD`.
+<4> The DataHub environment (fabric) to query, e.g. `PROD` or `DEV`. Defaults to `PROD`, which is also what DataHub's ingestion sources default to.
 
 Currently the following backends are supported:
 
@@ -65,7 +65,33 @@ DataHub is currently only supported backend, so the general documentation from a
 
 == Resource info fetcher API
 
-Resource information can be retrieved from regorules using the functions `TODO` in `data.stackable.opa.resourceinfo.v1`.
+Resource information can be retrieved from regorules using the functions in `data.stackable.opa.resourceinfo.v1`.
+There is one function per kind of resource:
+
+[source,rego]
+----
+databaseResourceInfo(system, instance, database)
+schemaResourceInfo(system, instance, database, schema)
+tableResourceInfo(system, instance, database, schema, table)
+streamResourceInfo(system, instance, queue)
+dashboardResourceInfo(system, instance, id)
+chartResourceInfo(system, instance, id)
+
+rawIdentifierResourceInfo(identifier)
+----
+
+The naming is intentionally product-agnostic, so that one function serves the equivalent resource of every product.
+`databaseResourceInfo` addresses what Trino calls a catalog, and `streamResourceInfo` what Apache Kafka calls a topic.
+
+`rawIdentifierResourceInfo` is the escape hatch for resources the functions above do not cover: it passes the identifier to the backend as-is.
+For DataHub that is a URN, such as `urn:li:chart:(superset,my-namespace/my-superset.1)`.
+
+The first two arguments are the same everywhere:
+
+* `system` is the kind of product the resource lives in, for example `trino`, `kafka` or `superset`. DataHub calls this the _data platform_.
+* `instance` identifies _which_ deployment of that product, for example `my-namespace/my-trino`. DataHub calls this the _platform instance_, and the value must match the `platform_instance` of the ingestion source that produced the metadata.
+
+The DataHub environment (fabric) is deliberately *not* an argument: it describes how the catalog was populated rather than the resource being authorized, so it is configured once on the OpaCluster (see `env` above) instead of being passed in by every Rego rule.
 
 An example of the returned structure:
 
@@ -99,22 +125,41 @@ An example of the returned structure:
 
 === Debug request
 
-To debug the resource-info-fetcher you can `curl` it's API for a given resource.
+To debug the resource-info-fetcher you can `curl` its API for a given resource.
+Every Rego function above maps to a `GET /metadata/<resource type>` endpoint that takes its arguments as query parameters.
 To achieve this shell into the `opa` container and execute
 
 [source,bash]
 ----
-curl --header 'Content-Type: application/json' -d '{"stacklet": "trino", "trinoSchema": {"catalog": "lakehouse", "schema": "customer_analytics"}}' localhost:9477/resource | jq
+curl 'localhost:9477/metadata/schema?system=trino&instance=my-namespace/my-trino&database=lakehouse&schema=customer_analytics' | jq
+
+curl 'localhost:9477/metadata/chart?system=superset&instance=my-namespace/my-superset&id=1' | jq
 
-curl --header 'Content-Type: application/json' -d '{"stacklet": "", "dataHubUrn": "urn:li:chart:(superset,1)"}' localhost:9477/resource | jq
+curl 'localhost:9477/metadata/rawIdentifier?identifier=urn:li:chart:(superset,my-namespace/my-superset.1)' | jq
 ----
 
 === Rego rule library
 
 The HTTP API exposed by the resource-info-fetcher can be called directly using the rego function `http.send`.
 However, we provide a convenience rego rule library, which we ship with `OpaClusters` by default.
 
+For example, the following rule allows access to tables tagged as `public`:
+
 [source,rego]
 ----
-TODO
+package test
+
+import data.stackable.opa.resourceinfo.v1 as resourceinfo
+
+default allow := false
+
+allow if {
+    table := resourceinfo.tableResourceInfo("trino", "my-namespace/my-trino", input.catalog, input.schema, input.table)
+    some tag in table.tags
+    tag.urn == "urn:li:tag:public"
+}
 ----
+
+A resource the backend does not know about is not reported as an error: the resource-info-fetcher returns a record with empty `tags`, `owners` and `dataProducts` and a `null` `domain`.
+Prefer rules that require a positive signal, like the one above, which denies access in that case.
+A rule that merely excludes a tag would instead grant access to every resource missing from the backend.

extra/crds.yaml

@@ -59,7 +59,7 @@ spec:
                       catalog.
 
                       Data catalog could e.g. be DataHub and resources could be Trino catalogs, schemas,
-                      tables or Kafka topic etc.
+                      tables or Kafka topics etc.
                     nullable: true
                     properties:
                       backend:
@@ -83,7 +83,41 @@ spec:
                                 type: string
                               env:
                                 default: PROD
-                                description: The env in DataHub, defaults to `PROD`
+                                description: |-
+                                  The DataHub environment (in DataHub terms: the fabric) the resources live in.
+
+                                  This must match the `env` of the DataHub ingestion source that produced the metadata,
+                                  because `env` is part of the dataset URN (and of the container key that database and
+                                  schema URNs are hashed from). A mismatch is not reported as an error: the constructed URN
+                                  simply does not resolve and the resource-info-fetcher returns empty metadata.
+
+                                  This is configured here rather than passed in by the Rego rule on purpose. `env`
+                                  describes how the catalog was populated, not the resource that is being authorized, and
+                                  the Rego rule has no way of knowing it. It is also DataHub-specific: no other metadata
+                                  catalog we know of has this concept.
+
+                                  `env` is largely superseded by DataHub's platform instance, which the Rego rules already
+                                  pass as `instance` — within a single DataHub, a platform instance belongs to exactly one
+                                  fabric. Should a single resource-info-fetcher ever need to serve multiple fabrics, we
+                                  would add per-instance `envOverrides` here instead of moving `env` back into the API.
+                                enum:
+                                - DEV
+                                - TEST
+                                - QA
+                                - UAT
+                                - EI
+                                - PRE
+                                - STG
+                                - NON_PROD
+                                - PROD
+                                - CORP
+                                - RVW
+                                - PRD
+                                - TST
+                                - SIT
+                                - SBX
+                                - SANDBOX
+                                - CERT
                                 type: string
                               hostname:
                                 description: Hostname of DataHub
@@ -2256,7 +2290,7 @@ spec:
                       catalog.
 
                       Data catalog could e.g. be DataHub and resources could be Trino catalogs, schemas,
-                      tables or Kafka topic etc.
+                      tables or Kafka topics etc.
                     nullable: true
                     properties:
                       backend:
@@ -2280,7 +2314,41 @@ spec:
                                 type: string
                               env:
                                 default: PROD
-                                description: The env in DataHub, defaults to `PROD`
+                                description: |-
+                                  The DataHub environment (in DataHub terms: the fabric) the resources live in.
+
+                                  This must match the `env` of the DataHub ingestion source that produced the metadata,
+                                  because `env` is part of the dataset URN (and of the container key that database and
+                                  schema URNs are hashed from). A mismatch is not reported as an error: the constructed URN
+                                  simply does not resolve and the resource-info-fetcher returns empty metadata.
+
+                                  This is configured here rather than passed in by the Rego rule on purpose. `env`
+                                  describes how the catalog was populated, not the resource that is being authorized, and
+                                  the Rego rule has no way of knowing it. It is also DataHub-specific: no other metadata
+                                  catalog we know of has this concept.
+
+                                  `env` is largely superseded by DataHub's platform instance, which the Rego rules already
+                                  pass as `instance` — within a single DataHub, a platform instance belongs to exactly one
+                                  fabric. Should a single resource-info-fetcher ever need to serve multiple fabrics, we
+                                  would add per-instance `envOverrides` here instead of moving `env` back into the API.
+                                enum:
+                                - DEV
+                                - TEST
+                                - QA
+                                - UAT
+                                - EI
+                                - PRE
+                                - STG
+                                - NON_PROD
+                                - PROD
+                                - CORP
+                                - RVW
+                                - PRD
+                                - TST
+                                - SIT
+                                - SBX
+                                - SANDBOX
+                                - CERT
                                 type: string
                               hostname:
                                 description: Hostname of DataHub

rust/operator-binary/src/controller/build/resource/daemonset/resource_info_fetcher.rs

@@ -17,9 +17,8 @@ use crate::{
             self,
             resource::daemonset::{
                 CONFIG_DIR, CONFIG_VOLUME_NAME, RESOURCE_INFO_FETCHER_CREDENTIALS_DIR,
-                RESOURCE_INFO_FETCHER_CREDENTIALS_VOLUME_NAME,
-                add_stackable_rust_cli_env_vars, container_name, sidecar_container_log_level,
-                sidecar_resource_requirements,
+                RESOURCE_INFO_FETCHER_CREDENTIALS_VOLUME_NAME, add_stackable_rust_cli_env_vars,
+                container_name, sidecar_container_log_level, sidecar_resource_requirements,
             },
         },
     },

rust/operator-binary/src/crd/resource_info_fetcher/mod.rs

@@ -46,5 +46,91 @@ pub mod versioned {
         ///
         /// Must contain the field `token`.
         pub credentials_secret_name: SecretName,
+
+        /// The DataHub environment (in DataHub terms: the fabric) the resources live in.
+        ///
+        /// This must match the `env` of the DataHub ingestion source that produced the metadata,
+        /// because `env` is part of the dataset URN (and of the container key that database and
+        /// schema URNs are hashed from). A mismatch is not reported as an error: the constructed URN
+        /// simply does not resolve and the resource-info-fetcher returns empty metadata.
+        ///
+        /// This is configured here rather than passed in by the Rego rule on purpose. `env`
+        /// describes how the catalog was populated, not the resource that is being authorized, and
+        /// the Rego rule has no way of knowing it. It is also DataHub-specific: no other metadata
+        /// catalog we know of has this concept.
+        ///
+        /// `env` is largely superseded by DataHub's platform instance, which the Rego rules already
+        /// pass as `instance` — within a single DataHub, a platform instance belongs to exactly one
+        /// fabric. Should a single resource-info-fetcher ever need to serve multiple fabrics, we
+        /// would add per-instance `envOverrides` here instead of moving `env` back into the API.
+        #[serde(default)]
+        pub env: FabricType,
+    }
+
+    /// DataHub's `FabricType`: the environments (in DataHub terms: fabrics) a resource can live in.
+    ///
+    /// The variants serialize exactly the way DataHub spells them, so a value can be copied from an
+    /// ingestion recipe or the DataHub UI as-is.
+    ///
+    /// See the [`FabricType` definition] in DataHub's metadata model.
+    ///
+    /// [`FabricType` definition]: https://github.com/datahub-project/datahub/blob/master/li-utils/src/main/pegasus/com/linkedin/common/FabricType.pdl
+    #[derive(
+        Clone, Debug, Default, Deserialize, Eq, JsonSchema, PartialEq, Serialize, strum::Display,
+    )]
+    #[serde(rename_all = "SCREAMING_SNAKE_CASE")]
+    #[strum(serialize_all = "SCREAMING_SNAKE_CASE")]
+    pub enum FabricType {
+        /// Development fabrics.
+        Dev,
+
+        /// Testing fabrics.
+        Test,
+
+        /// Quality assurance fabrics.
+        Qa,
+
+        /// User acceptance testing fabrics.
+        Uat,
+
+        /// Early-integration fabrics.
+        Ei,
+
+        /// Pre-production fabrics.
+        Pre,
+
+        /// Staging fabrics.
+        Stg,
+
+        /// Non-production fabrics.
+        NonProd,
+
+        /// Production fabrics. DataHub's ingestion sources default to this fabric, so we do as well.
+        #[default]
+        Prod,
+
+        /// Corporation fabrics.
+        Corp,
+
+        /// Review fabrics.
+        Rvw,
+
+        /// Alternative spelling of [`Self::Prod`], which DataHub accepts as well.
+        Prd,
+
+        /// Alternative spelling of [`Self::Test`], which DataHub accepts as well.
+        Tst,
+
+        /// System integration testing fabrics.
+        Sit,
+
+        /// Alternative spelling of [`Self::Sandbox`], which DataHub accepts as well.
+        Sbx,
+
+        /// Sandbox fabrics.
+        Sandbox,
+
+        /// Certification fabrics.
+        Cert,
     }
 }