From c2b8c5506aa15c49c55aa502ba03c71d5e7140df Mon Sep 17 00:00:00 2001 From: Aditya Thebe Date: Mon, 19 Feb 2024 19:46:52 +0545 Subject: [PATCH 1/9] transform relationships & kubernetes relationship example --- .editorconfig | 6 ++ .gitignore | 4 + .vscode/settings.json | 2 - canary-checker/docs/index.md | 95 ------------------- mission-control/Makefile | 2 +- .../docs/config-db/concepts/transform.md | 70 +++++++++++--- .../examples/kubernetes-relationship.md | 29 ++++++ mission-control/sidebars.js | 11 +++ 8 files changed, 107 insertions(+), 112 deletions(-) create mode 100644 .editorconfig delete mode 100644 .vscode/settings.json delete mode 100644 canary-checker/docs/index.md create mode 100644 mission-control/docs/config-db/examples/kubernetes-relationship.md diff --git a/.editorconfig b/.editorconfig new file mode 100644 index 00000000..69a0a2d3 --- /dev/null +++ b/.editorconfig @@ -0,0 +1,6 @@ +[*.{md,js}] +quote_type = single +indent_size = 2 +end_of_line = lf +insert_final_newline = true +trim_trailing_whitespace = true \ No newline at end of file diff --git a/.gitignore b/.gitignore index 4b190ca6..9ac110fe 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,9 @@ +.envrc +default.nix + .DS_Store site/ **/build/ .docusaurus node_modules +.vscode diff --git a/.vscode/settings.json b/.vscode/settings.json deleted file mode 100644 index 7a73a41b..00000000 --- a/.vscode/settings.json +++ /dev/null @@ -1,2 +0,0 @@ -{ -} \ No newline at end of file diff --git a/canary-checker/docs/index.md b/canary-checker/docs/index.md deleted file mode 100644 index af5562f1..00000000 --- a/canary-checker/docs/index.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -slug: / -title: Overview -hide_title: true -hide_table_of_contents: true -pagination_next: null -pagination_prev: null ---- - -![Canary Checker](./images/canary-checker.svg#gh-light-mode-only) -![Canary Checker](./images/canary-checker-white.svg#gh-dark-mode-only) - -Canary checker is a kubernetes-native platform for monitoring health across application and infrastructure using both passive and active (synthetic) mechanisms. - -## Features - -* **Batteries Included** - 35+ built-in check types -* **Kubernetes Native** - Health checks (or canaries) are CRD's that reflect health via the `status` field, making them compatible with GitOps, [Flux Health Checks](https://fluxcd.io/flux/components/kustomize/kustomization/#health-checks), Argo, Helm, etc.. -* **Secret Management** - Leverage K8S secrets and configmaps for authentication and connection details -* **Prometheus** - Prometheus compatible metrics are exposed at `/metrics`. A Grafana Dashboard is also available. -* **Dependency Free** - Runs an embedded postgres instance by default, can also be configured to use an external database. -* **JUnit Export (CI/CD)** - Export health check results to JUnit format for integration into CI/CD pipelines -* **JUnit Import (k6/newman/puppeter/etc)** - Use any container that creates JUnit test results -* **Scriptable** - Go templates, Javascript and [Expr](https://github.com/antonmedv/expr) can be used to: - * Evaluate whether a check is passing and severity to use when failing - * Extract a user friendly error message - * Transform and filter check responses into individual check results -* **Multi-Modal** - While designed as a Kubernetes Operator, canary checker can also run as a CLI and a server without K8s - -Canary Checker can collect health about systems in few different ways: - -* **Active** **Application** health checks involve sending periodic requests to the service or application and checking the response to ensure that it is working correctly, Active health checks are proactive and can detect issues quickly, but they can also introduce some load on the system being monitored. -* **Active Infrastructure** health checks are similar to application health checks, but instead of sending a request to the application it sends a request to the infrastructure to deploy a new application or infrastructure component e.g. a new Kubernetes pod or EC2 instance. -* **Passive** health checks rely on monitoring the activity in the system, analysing it, and detecting anomalies or errors. Passive health checks are less intrusive than active health checks, but they may not detect issues as quickly. - -Health checks can be defined in 3 different ways: - -1. **UI**: Navigate to Settings --> Health --> Click on the :material-plus-circle: button -1. **[GitOps](./concepts/gitops.md)** canary-checker is fully Gitops enabled using Kubernetes Custom Resource Definitions (CRD) -1. **[CLI](./tutorials/run.md)** For rapid development and feedback, canary-checker can be run as a normal CLI application by specifying the health check definition in a config file. - -## License - -Canary Checker core (the code in this repository) is licensed under [Apache 2.0](https://raw.githubusercontent.com/flanksource/canary-checker/main/LICENSE) and accepts contributions via GitHub pull requests after signing a CLA. - -The UI (Dashboard) is free to use with canary checker under a license exception of [Flanksource UI](https://github.com/flanksource/flanksource-ui/blob/main/LICENSE#L7) - -## Check Types - -| Protocol | Status | Checks | -| ----------------------------------- | ------------------ | ---- | -| [HTTP(s)](reference/http) | GA | Response body, headers and duration | -| [DNS](reference/dns) | GA | Response and duration | -| [Ping/ICMP](reference/icmp) | GA | Duration and packet loss | -| [TCP](reference/tcp) | GA | Port is open and connectable | -| **Data Sources** | | | -| SQL ([MySQL](reference/mysql), [Postgres](reference/postgres), [SQL Server](reference/mssql)) | GA | Ability to login, results, duration, health exposed via stored procedures | -| [LDAP](reference/ldap) | GA | Ability to login, response time | -| [ElasticSearch / Opensearch](reference/elasticsearch) | GA | Ability to login, response time, size of search results | -| [Mongo](reference/mongo) | Beta | Ability to login, results, duration, | -| [Redis](reference/redis) | GA | Ability to login, results, duration, | -| [Prometheus](reference/prometheus) | GA | Ability to login, results, duration, | -| **Alerts** | | Prometheus | -| [Prometheus Alert Manager](reference/alert-manager) | GA | Pending and firing alerts | -| [AWS Cloudwatch Alarms](reference/cloudwatch) | GA | Pending and firing alarms | -| [Dynatrace Problems](reference/dynatrace) | Beta | Problems deteced | -| **DevOps** | | | -| [Git](reference/git) | GA | Query Git and Github repositories via SQL | -| [Azure Devops](reference/azure-devops) | | | -| **Integration Testing** | | | -| [JMeter](reference/jmeter) | Beta | Runs and checks the result of a JMeter test | -| [JUnit](reference/junit) | Beta | Run a container/pod that saves Junit test results | -| [K6](reference/k6) | Beta | Runs K6 tests that export JUnit via a container | -| [Newman](reference/newman) | Beta | Runs Newman / Postman tests that export JUnit via a container | -| [Playwright](reference/Playwright) | Beta | Runs Playwright tests that export JUnit via a container | -| **File Systems / Batch** | | | -| [Local Disk / NFS](reference/folder) | GA | Check folders for files that are: too few/many, too old/new, too small/large | -| [S3](reference/s3-bucket) | GA | Check contents of AWS S3 Buckets | -| [GCS](reference/gcs-bucket) | GA | Check contents of Google Cloud Storage Buckets | -| [SFTP](reference/sftp) | GA | Check contents of folders over SFTP | -| [SMB / CIFS](reference/smb) | GA | Check contents of folders over SMB/CIFS | -| **Config** | | | -| [AWS Config](reference/aws-config) | GA | Query AWS config using SQL | -| [AWS Config Rule](reference/aws-config-rule) | GA | AWS Config Rules that are firing, Custom AWS Config queries | -| [Config DB](reference/configdb) | GA | Custom config queries for Mission Control Config D | -| [Kubernetes Resources](reference/kubernetes) | GA | Kubernetes resources that are missing or are in a non-ready state | -| **Backups** | | | -| [GCP Databases](reference/gcs-database-backup) | GA | Backup freshness | -| [Restic](reference/restic) | Beta | Backup freshness and integrity | -| **Infrastructure** | | | -| [EC2](reference/ec2) | GA | Ability to launch new EC2 instances | -| [Kubernetes Ingress](reference/pod) | GA | Ability to schedule and then route traffic via an ingress to a pod | -| [Docker/Containerd](reference/containerd) | Deprecated | Ability to push and pull containers via docker/containerd | -| [Helm](reference/helm) | Deprecated | Ability to push and pull helm charts | -| [S3 Protocol](reference/s3-protocol) | GA | Ability to read/write/list objects on an S3 compatible object store | diff --git a/mission-control/Makefile b/mission-control/Makefile index 869f12ec..500d98e4 100644 --- a/mission-control/Makefile +++ b/mission-control/Makefile @@ -1,6 +1,6 @@ .PHONY: sync: - rm docs/canary-checker/index.md + rm -f docs/canary-checker/index.md watch: sync npm exec -c 'docusaurus start' diff --git a/mission-control/docs/config-db/concepts/transform.md b/mission-control/docs/config-db/concepts/transform.md index cd86607d..cedea494 100644 --- a/mission-control/docs/config-db/concepts/transform.md +++ b/mission-control/docs/config-db/concepts/transform.md @@ -6,22 +6,66 @@ Transformation can be performed after the configs have been scraped from the tar - transform the scraped configuration using CEL - remove certain fields from the scraped configuration -| Field | Description | Scheme | Required | -| --------------- | ---------------------------------------------------------------------------------------- | --------------------------------------- | -------- | -| `gotemplate` | Specify Go template for use in script | `string` | | -| `javascript` | Specify javascript syntax for script | `string` | | -| `jsonpath` | Specify JSONPath | `string` | | -| `expr` | Specify Cel expression | `string` | | -| `change` | Apply transformaion on the scraped changes | [`[]Changes`](#changes) | | -| `exclude` | Fields to remove from the config, useful for removing sensitive data and fields | [`[]Exclude`](#exclude) | | -| | that change often without a material impact i.e. Last Scraped Time | | | -| [`mask`](#mask) | Specify configurations to replace sensitive fields with hash functions or static string. | [`[]Mask`](./mask.md) | | +| Field | Description | Scheme | Required | +| --------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------- | -------- | +| `gotemplate` | Specify Go template for use in script | `string` | | +| `javascript` | Specify javascript syntax for script | `string` | | +| `jsonpath` | Specify JSONPath | `string` | | +| `expr` | Specify Cel expression | `string` | | +| `changes` | Apply transformation on the scraped changes | [`[]Changes`](#changes) | | +| `exclude` | Fields to remove from the config, useful for removing sensitive data and fields | [`[]Exclude`](#exclude) | | +| | that change often without a material impact i.e. Last Scraped Time | | | +| [`mask`](#mask) | Specify configurations to replace sensitive fields with hash functions or static string. | [`[]Mask`](./mask.md) | | +| `relationship` | form relationships between config items using selectors | [`[]RelationshipConfig`](#relationshipconfig) | | :::note Unlike other transformation functions, scripts (gotemplate, javascript, jsonpath & expr) are ran before the attributes _(id, name, type, ...)_ are extracted. So please make sure your transformation scripts are inline with the JSONPath selectors for the attributes. ::: +## RelationshipConfig + +This transformation function allows you to dynamically form relationships between two different config items using selectors. + +Example: You can link a kubernetes deployment with the corresponding pods, or you can link AWS EC2 instances with the AWS Account. It's even possible to link two configs scraped by different scrape configs like: linking a Kubernetes Node in an EKS cluster to the EC2 instance. + +| Field | Description | Scheme | Required | +| -------- | ------------------------------------------------------------------------------------- | ------------------------------------------- | -------- | +| `filter` | Specify the config item with which relationship should be formed | `string` | `true` | +| `expr` | cel-expression that returns a list of [relationship selector](#relationshipselector). | `string` | | +| `id` | id of the config to link to | [`RelationshipLookup`](#relationshiplookup) | | +| `name` | id of the config to link to | [`RelationshipLookup`](#relationshiplookup) | | +| `type` | id of the config to link to | [`RelationshipLookup`](#relationshiplookup) | | +| `agent` | id of the config to link to | [`RelationshipLookup`](#relationshiplookup) | | +| `labels` | Labels of the config to link to | [`RelationshipLookup`](#relationshiplookup) | | + +:::info +`expr` is an alternative, more flexible, way to define the selectors. Either use `expr` or the other selector fields (`id`, `name`, `type`, `agent`, `labels`) but not both. +[**See example**](../examples/kubernetes-relationship). +::: + +### RelationshipSelector + +| Field | Description | Scheme | Required | +| -------- | ------------------------------- | -------- | -------- | +| `id` | id of the config to link to | `string` | | +| `name` | id of the config to link to | `string` | | +| `type` | id of the config to link to | `string` | | +| `agent` | id of the config to link to | `string` | | +| `labels` | Labels of the config to link to | `string` | | + +### RelationshipLookup + +RelationshipLookup offers different ways to specify a lookup value + +| Field | Description | Scheme | Required | +| ------- | ---------------------------------- | -------- | -------- | +| `expr` | Use an expression to get the value | `string` | | +| `value` | Specify a static value | `string` | | +| `label` | Get the value from a label | `string` | | + +## Script + ### JavaScript You can supply a JavaScript code to transform the scraped configuration. Your JS code will have access to the special `config` variable which will contain the scraped config. Your script is expected to return a stringified JSON object which will be the new configuration. @@ -173,13 +217,12 @@ At the moment, only `md5sum` is supported. More hash functions will be added in ## Changes -| Field | Description | Scheme | Required | -| --------- | ----------------------------------------------------------------- | ---------- | -------- | +| Field | Description | Scheme | Required | +| --------- | ------------------------------------------------------ | ---------- | -------- | | `exclude` | A list of CEL expressions that excludes a given change | `[]string` | | The scraped changes can be accessed using the `details` field. - ```yaml title="exclude-canary-pass-fail.yaml" spec: kubernetes: @@ -198,7 +241,6 @@ This feature allows you to specify custom creation and deletion times for config You'll be making use of the `createFields` and `deleteFields` fields that are supported by all the scrapers. They are both a list of [JSONPath expression](../concepts/templating.md#jsonpath) and are used to extract the created/deleted time of the config item from the scraped configuration. If multiple fields are specified, the first non-empty value will be used. - Consider the following configuration file ```yaml diff --git a/mission-control/docs/config-db/examples/kubernetes-relationship.md b/mission-control/docs/config-db/examples/kubernetes-relationship.md new file mode 100644 index 00000000..63c69450 --- /dev/null +++ b/mission-control/docs/config-db/examples/kubernetes-relationship.md @@ -0,0 +1,29 @@ +This example demonstrates 2 different ways you can form relationships between config items. + +The first relationship is formed between a Kubernetes service and its corresponding deployment using the inline relationship selector _(type & name)_ while the second relationship is formed between Pods and PVCs using the `expr` way. + +```yaml title="kubernetes-scraper.yaml" +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: kubernetes-scraper +spec: + kubernetes: + - clusterName: local-kind-cluster + transform: + relationship: + # Link a service to a deployment (adjust the label selector accordingly) + - filter: config_type == "Kubernetes::Service" + type: + value: 'Kubernetes::Deployment' + name: + expr: | + has(config.spec.selector) && has(config.spec.selector.name) ? config.spec.selector.name : '' + # Link Pods to PVCs + - filter: config_type == 'Kubernetes::Pod' + expr: | + config.spec.volumes. + filter(item, has(item.persistentVolumeClaim)). + map(item, {"type": "Kubernetes::PersistentVolumeClaim", "name": item.persistentVolumeClaim.claimName}). + toJSON() +``` diff --git a/mission-control/sidebars.js b/mission-control/sidebars.js index 0c6b293d..7a08febd 100644 --- a/mission-control/sidebars.js +++ b/mission-control/sidebars.js @@ -452,6 +452,17 @@ const sidebars = { } ] }, + { + type: 'category', + label: 'Examples', + items: [ + { + type: 'doc', + id: 'config-db/examples/kubernetes-relationship', + label: 'Forming relationships' + }, + ] + }, ], playbooksSidebar: [ { From 9edc3b13610fe5cb31f119aac91992965f5fd58e Mon Sep 17 00:00:00 2001 From: Aditya Thebe Date: Mon, 19 Feb 2024 19:59:09 +0545 Subject: [PATCH 2/9] update field exclusion --- .../docs/config-db/concepts/transform.md | 20 +++++-------------- .../docs/config-db/examples/exclude-fields.md | 19 ++++++++++++++++++ mission-control/sidebars.js | 5 +++++ 3 files changed, 29 insertions(+), 15 deletions(-) create mode 100644 mission-control/docs/config-db/examples/exclude-fields.md diff --git a/mission-control/docs/config-db/concepts/transform.md b/mission-control/docs/config-db/concepts/transform.md index cedea494..21c7f7da 100644 --- a/mission-control/docs/config-db/concepts/transform.md +++ b/mission-control/docs/config-db/concepts/transform.md @@ -146,22 +146,12 @@ The above transformation will result in the following config This transformation function allows you to remove certain fields from the scraped configuration. This is useful when you want to remove sensitive or just useless data from the scraped configuration. -### Filter +| Field | Description | Scheme | Required | +| ---------- | -------------------------------------------------------------------------- | ---------- | -------- | +| `jsonpath` | Specify JSONPath expression for the fields | `string` | `true` | +| `types` | specify the config types from which the JSONPath fields need to be removed | `[]string` | | -| Field | Description | Scheme | Required | -| ---------- | ------------------------------------------ | -------- | -------- | -| `jsonpath` | Specify JSONPath expression for the fields | `string` | `true` | - -_Example_: With the following `Config DB` configuration for AWS, the transformation will delete the `tags` and `privateDnsNameOptionsOnLaunch` fields from the scraped configuration. - -```yaml -aws: - - type: AWS - transform: - exclude: - - jsonpath: $.tags - - jsonpath: $.privateDnsNameOptionsOnLaunch -``` +The `types` field is optional and if left empty, the filter will apply to all config items. [See Example](../examples/exclude-fields) ## Mask diff --git a/mission-control/docs/config-db/examples/exclude-fields.md b/mission-control/docs/config-db/examples/exclude-fields.md new file mode 100644 index 00000000..5a2ae829 --- /dev/null +++ b/mission-control/docs/config-db/examples/exclude-fields.md @@ -0,0 +1,19 @@ +# Exclude fields + +In the following scrape config for Kubernetes, the transformation will delete the `.metadata.ownerReferences` field from all the scraped items and `.metadata.generateName` field will be removed only from Pods. + +```yaml title="kubernetes.yaml" +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: kubernetes-scraper +spec: + kubernetes: + - clusterName: local-kind-cluster + transform: + exclude: + - jsonpath: '.metadata.ownerReferences' + - types: + - Kubernetes::Pod + jsonpath: '.metadata.generateName' +``` diff --git a/mission-control/sidebars.js b/mission-control/sidebars.js index 7a08febd..11af18f7 100644 --- a/mission-control/sidebars.js +++ b/mission-control/sidebars.js @@ -461,6 +461,11 @@ const sidebars = { id: 'config-db/examples/kubernetes-relationship', label: 'Forming relationships' }, + { + type: 'doc', + id: 'config-db/examples/exclude-fields', + label: 'Exclude superflous fields' + }, ] }, ], From c0202ed9be34691061e8e257dabc2457026aa543 Mon Sep 17 00:00:00 2001 From: Aditya Thebe Date: Mon, 19 Feb 2024 20:16:43 +0545 Subject: [PATCH 3/9] properties --- .../docs/config-db/scrapers/aws.md | 143 +++++++++++------- mission-control/docs/reference/property.md | 31 ++++ 2 files changed, 119 insertions(+), 55 deletions(-) create mode 100644 mission-control/docs/reference/property.md diff --git a/mission-control/docs/config-db/scrapers/aws.md b/mission-control/docs/config-db/scrapers/aws.md index 161fc2f0..901c6014 100644 --- a/mission-control/docs/config-db/scrapers/aws.md +++ b/mission-control/docs/config-db/scrapers/aws.md @@ -2,41 +2,73 @@ This config type is used to scrape information about your AWS infrastructure. -```yaml -logLevel: "" -schedule: "@every 15m" -retention: - changes: - - name: CreateRole - age: 30d # Any change older than 30 days is removed - count: 50 # Only 50 last changes will be retained -aws: - - region: - - eu-west-2 - - us-east-1 - - af-south-1 - - ap-south-1 - - eu-central-1 -compliance: true -patch_states: false -trusted_advisor_check: false -patch_details: false -cost_reporting: - s3_bucket_path: s3://flanksource-cost-reports/query-results - database: athenacurcfn_flanksource_report - table: flanksource_report - region: af-south-1 -inventory: true -exclude: - - Amazon EC2 Reserved Instances Optimization - - Savings Plan -transform: - exclude: - - jsonpath: $.tags - - jsonpath: $.privateDnsNameOptionsOnLaunch - - jsonpath: outpostArn - - jsonpath: mapCustomerOwnedIpOnLaunch - - jsonpath: subnetArn +```yaml title="aws-scraper.yaml" +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: aws-scraper +spec: + aws: + - region: + - eu-west-2 + - us-east-1 + - af-south-1 + - ap-south-1 + - eu-central-1 + properties: + - name: AWS Link + filter: 'config_type == AWS::IAM::Role' + icon: aws-iam + links: + - text: AWS Link + url: 'https://us-east-1.console.aws.amazon.com/iamv2/home#/roles/details/{{.name}}?section=permissions' + compliance: true + patch_states: false + trusted_advisor_check: false + patch_details: false + cost_reporting: + s3_bucket_path: s3://flanksource-cost-reports/query-results + database: athenacurcfn_flanksource_report + table: flanksource_report + region: af-south-1 + inventory: true + exclude: + - Amazon EC2 Reserved Instances Optimization + - Savings Plan + transform: + relationship: + # EKS Cluster to Kubernetes Cluster & Kubernetes Node + - filter: config_type == 'AWS::EKS::Cluster' + expr: | + [ + {"type": "Kubernetes::Cluster","labels": {"aws/account-id": tags['account'],"eks-cluster-name": tags["alpha.eksctl.io/cluster-name"]}}, + {"type": "Kubernetes::Node","labels": {"aws/account-id": tags['account'],"alpha.eksctl.io/cluster-name": tags["alpha.eksctl.io/cluster-name"]}} + ].toJSON() + # EC2 Instance to kubernetes node + - filter: config_type == 'AWS::EC2:Instance' + expr: | + [{"type": "Kubernetes::Node", "labels": {"alpha.eksctl.io/instance-id": config["instance_id"]}}].toJSON() + # IAM Role to Kubernetes Node + - filter: config_type == 'AWS::IAM::Role' + expr: | + [{"type": "Kubernetes::Node", "labels": {"aws/iam-role": config["Arn"]}}].toJSON() + # AvailabilityZone to Zone ID & Kubernetes Node + - filter: config_type == 'AWS::AvailabilityZone' + expr: | + [ + {"type": "Kubernetes::Node", "labels": {"aws/account-id": tags['account'], "topology.kubernetes.io/zone": name}}, + {"type": "AWS::AvailabilityZoneID", "name": config["ZoneId"]} + ].toJSON() + # Region to ZoneID + - filter: config_type == 'AWS::Region' + expr: | + [{"type": "AWS::AvailabilityZoneID", "labels": {"region": name}}].toJSON() + exclude: + - jsonpath: $.tags + - jsonpath: $.privateDnsNameOptionsOnLaunch + - jsonpath: outpostArn + - jsonpath: mapCustomerOwnedIpOnLaunch + - jsonpath: subnetArn ``` ### Scraper @@ -51,26 +83,27 @@ transform: ### AWS -| Field | Description | Scheme | Required | -| -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | -------- | -| `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | -| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` | -| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` | -| `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | -| `transform` | Specify field to transform result | [`Transform`](../concepts/transform.md) | `false` | -| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | `false` | -| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` | -| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `compliance` | Toggle scraping of compliance metadata | _bool_ | | -| `patch_states` | Set boolean value for scraping of patch state status, and report | _bool_ | | -| `trusted_advisor_check` | Set boolean value for enabling/disabling trusted advisor | _bool_ | | -| `patch_details` | Set boolean value to enable/disable scraping of patch details | _bool_ | | -| [`cost_reporting`](#cost-reporting) | Specify cost reporting for scraping of data | [_CostReporting_](#cost-reporting) | | -| [`cloud_trail`](#cloudtrail-cloudtrail-cloudtrail) | Set CloudTrail specifications | [_CloudTrail_](#cloudtrail-cloudtrail) | | -| `include` | Specify AWS resources to include for scraping | _\[\]string_ | | -| `exclude` | Specify AWS resources to exclude from scraping | _\[\]string_ | | -| `inventory` | Toggle scrape of _metadata_ for AWS resources | _bool_ | | +| Field | Description | Scheme | Required | +| -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | -------- | +| `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | +| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` | +| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` | +| `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | +| `transform` | Specify field to transform result | [`Transform`](../concepts/transform.md) | `false` | +| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | `false` | +| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` | +| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | +| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | +| `compliance` | Toggle scraping of compliance metadata | _bool_ | | +| `patch_states` | Set boolean value for scraping of patch state status, and report | _bool_ | | +| `trusted_advisor_check` | Set boolean value for enabling/disabling trusted advisor | _bool_ | | +| `patch_details` | Set boolean value to enable/disable scraping of patch details | _bool_ | | +| [`cost_reporting`](#cost-reporting) | Specify cost reporting for scraping of data | [_CostReporting_](#cost-reporting) | | +| [`cloud_trail`](#cloudtrail-cloudtrail-cloudtrail) | Set CloudTrail specifications | [_CloudTrail_](#cloudtrail-cloudtrail) | | +| `include` | Specify AWS resources to include for scraping | _\[\]string_ | | +| `exclude` | Specify AWS resources to exclude from scraping | _\[\]string_ | | +| `inventory` | Toggle scrape of _metadata_ for AWS resources | _bool_ | | +| `properties` | Custom templatable properties for the scraped config items. | [`[]ConfigProperty`](../../reference/property) | | ### CloudTrail (`cloudtrail`) diff --git a/mission-control/docs/reference/property.md b/mission-control/docs/reference/property.md new file mode 100644 index 00000000..6085c8e1 --- /dev/null +++ b/mission-control/docs/reference/property.md @@ -0,0 +1,31 @@ +# Property + +| Field | Description | Schema | Required | +| ---------------- | ----------------------------------- | -------- | -------- | +| `label` | The label of the property. | `string` | | +| `name` | The name of the property. | `string` | | +| `tooltip` | The tooltip of the property. | `string` | | +| `icon` | The icon of the property. | `string` | | +| `type` | The type of the property. | `string` | | +| `color` | The color of the property. | `string` | | +| `order` | The order of the property. | `int` | | +| `headline` | The headline of the property. | `bool` | | +| `text` | The text of the property. | `string` | | +| `value` | The value of the property. | `int` | | +| `unit` | The unit of the property. | `string` | | +| `max` | The max of the property. | `int` | | +| `min` | The min of the property. | `int` | | +| `status` | The status of the property. | `string` | | +| `lastTransition` | The lastTransition of the property. | `string` | | +| `links` | The links of the property. | `[]Link` | | + +## Link + +| Field | Description | Schema | Required | +| --------- | ------------------------ | -------- | -------- | +| `type` | The type of the link. | `string` | | +| `url` | The url of the link. | `string` | | +| `tooltip` | The tooltip of the link. | `string` | | +| `icon` | The icon of the link. | `string` | | +| `text` | The text of the link. | `string` | | +| `label` | The label of the link. | `string` | | From 9cb10cdf9328671441aa1ae66fe703a3947fdf05 Mon Sep 17 00:00:00 2001 From: Aditya Thebe Date: Tue, 20 Feb 2024 10:16:56 +0545 Subject: [PATCH 4/9] masking examples --- .../docs/config-db/concepts/transform.md | 36 ++++------------- .../docs/config-db/examples/exclude-fields.md | 19 ++++++++- .../examples/kubernetes-relationship.md | 2 + .../docs/config-db/examples/masking-fields.md | 40 +++++++++++++++++++ mission-control/sidebars.js | 7 +++- 5 files changed, 73 insertions(+), 31 deletions(-) create mode 100644 mission-control/docs/config-db/examples/masking-fields.md diff --git a/mission-control/docs/config-db/concepts/transform.md b/mission-control/docs/config-db/concepts/transform.md index 21c7f7da..f0cc4e03 100644 --- a/mission-control/docs/config-db/concepts/transform.md +++ b/mission-control/docs/config-db/concepts/transform.md @@ -156,7 +156,8 @@ The `types` field is optional and if left empty, the filter will apply to all co ## Mask Mask allows replacing sensitive fields with hash of that field or with a static string. -_Example_: You could set the `value` to `***` and all the fields that match the `jsonPath` will be replaced with `***`. +The field to mask is specified by the `jsonPath` config and `value` field defines the hash function or the static value. +[See example](../examples/masking-fields). | Field | Description | Scheme | Required | | ---------- | ------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -------- | @@ -164,28 +165,13 @@ _Example_: You could set the `value` to `***` and all the fields that match the | `jsonpath` | Specify JSONPath expression for the fields | `string` | `true` | | `value` | Value can be a name of a hash function or just a string. See [supported hash functions](#supported-hash-functions). | `string` | `true` | -As an example let's take the following configuration file for `Config DB` +### MaskSelector -```yaml -file: - - type: Config - id: $.id - name: $.name - transform: - mask: - - selector: - type: Config - jsonpath: $.password - value: md5sum - - selector: - type: Config - jsonpath: $.secret - value: '***' - paths: - - fixtures/data/single-config.json -``` +| Field | Description | Scheme | Required | +| ------ | ----------------------------------------- | -------- | -------- | +| `type` | Type is the config type to apply the mask | `string` | `true` | -This configuration specifies 2 different masks. The first one will replace the value of the field `password` with the md5sum of the value. The second one will replace the value of the field `secret` with `***`. +`Config DB` allows selectively applying masks to certain types of configs. So you could apply a mask to all the `Config` types and another mask to all the `Secret` types. :::info Masks are applied in the order they are specified in the configuration file. @@ -197,14 +183,6 @@ At the moment, only `md5sum` is supported. More hash functions will be added in - md5sum -### MaskSelector - -| Field | Description | Scheme | Required | -| ------ | ----------------------------------------- | -------- | -------- | -| `type` | Type is the config type to apply the mask | `string` | `true` | - -`Config DB` allows selectively applying masks to certain types of configs. So you could apply a mask to all the `Config` types and another mask to all the `Secret` types. - ## Changes | Field | Description | Scheme | Required | diff --git a/mission-control/docs/config-db/examples/exclude-fields.md b/mission-control/docs/config-db/examples/exclude-fields.md index 5a2ae829..a38728e8 100644 --- a/mission-control/docs/config-db/examples/exclude-fields.md +++ b/mission-control/docs/config-db/examples/exclude-fields.md @@ -2,7 +2,7 @@ In the following scrape config for Kubernetes, the transformation will delete the `.metadata.ownerReferences` field from all the scraped items and `.metadata.generateName` field will be removed only from Pods. -```yaml title="kubernetes.yaml" +```yaml title="kubernetes-exclude-superfluous-fields.yaml" apiVersion: configs.flanksource.com/v1 kind: ScrapeConfig metadata: @@ -17,3 +17,20 @@ spec: - Kubernetes::Pod jsonpath: '.metadata.generateName' ``` + +Field exclusions are also helpful when you want to exclude sensitive fields from the scraped data. + +```yaml title="kubernetes-exclude-sensitive-fields.yaml" +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: kubernetes-scraper +spec: + kubernetes: + - clusterName: local-kind-cluster + transform: + exclude: + - types: + - Kubernetes::Secret + jsonpath: '.data' +``` diff --git a/mission-control/docs/config-db/examples/kubernetes-relationship.md b/mission-control/docs/config-db/examples/kubernetes-relationship.md index 63c69450..73072fb4 100644 --- a/mission-control/docs/config-db/examples/kubernetes-relationship.md +++ b/mission-control/docs/config-db/examples/kubernetes-relationship.md @@ -1,3 +1,5 @@ +# Kubernetes Relationship + This example demonstrates 2 different ways you can form relationships between config items. The first relationship is formed between a Kubernetes service and its corresponding deployment using the inline relationship selector _(type & name)_ while the second relationship is formed between Pods and PVCs using the `expr` way. diff --git a/mission-control/docs/config-db/examples/masking-fields.md b/mission-control/docs/config-db/examples/masking-fields.md new file mode 100644 index 00000000..e668054c --- /dev/null +++ b/mission-control/docs/config-db/examples/masking-fields.md @@ -0,0 +1,40 @@ +# Masking sensitive fields + +```yaml title="file-mask-scraper.yaml" +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: file-mask-scraper +spec: + file: + - type: Config + id: $.id + name: $.name + transform: + mask: + - selector: config.name == 'Config1' + jsonpath: $.password + value: md5sum + - selector: config.name == 'Config1' + jsonpath: $.secret + value: '***' + paths: + - fixtures/data/single-config.json +``` + +This configuration specifies 2 different masks. The first one will replace the value of the field `password` with the md5sum of the value. The second one will replace the value of the field `secret` with `***`. + +```yaml title="kubernetes-mask-secrets.yaml" +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: kubernetes-scraper +spec: + kubernetes: + - clusterName: local-kind-cluster + transform: + mask: + - selector: config.type == 'Kubernetes::Secret' + jsonpath: .data + value: md5sum +``` diff --git a/mission-control/sidebars.js b/mission-control/sidebars.js index 11af18f7..e1452048 100644 --- a/mission-control/sidebars.js +++ b/mission-control/sidebars.js @@ -464,7 +464,12 @@ const sidebars = { { type: 'doc', id: 'config-db/examples/exclude-fields', - label: 'Exclude superflous fields' + label: 'Exclude superflous/sensitive fields' + }, + { + type: 'doc', + id: 'config-db/examples/masking-fields', + label: 'Masking sensitive fields' }, ] }, From 336dd19c5d56e32f385fbbc3cd3b725c0b1f5af7 Mon Sep 17 00:00:00 2001 From: Aditya Thebe Date: Tue, 20 Feb 2024 10:27:09 +0545 Subject: [PATCH 5/9] convert examples to CRD specs --- .../docs/config-db/concepts/extraction.md | 18 ++- .../docs/config-db/concepts/transform.md | 137 +++++++++++------- .../docs/config-db/scrapers/azure-devops.md | 31 ++-- .../docs/config-db/scrapers/file.md | 19 ++- .../config-db/scrapers/kubernetes-file.md | 9 +- .../docs/config-db/scrapers/kubernetes.md | 59 ++++---- .../docs/config-db/scrapers/sql.md | 117 ++++++++------- .../docs/config-db/scrapers/trivy.md | 15 +- 8 files changed, 233 insertions(+), 172 deletions(-) diff --git a/mission-control/docs/config-db/concepts/extraction.md b/mission-control/docs/config-db/concepts/extraction.md index f68b8a8d..4f7d954f 100644 --- a/mission-control/docs/config-db/concepts/extraction.md +++ b/mission-control/docs/config-db/concepts/extraction.md @@ -1,4 +1,5 @@ # Extraction + `Config DB` needs to extract few important pieces of information from the config. For example: to know the id of a config item it needs to extract the id from the scraped config. For this, it makes heavy use of JSONPath expression. ## JSONPath @@ -9,12 +10,17 @@ A JSONPath expression, similar to `XPath` for XML, is used to extract data from Below is an example of the JSONPath expression in use for the [File scraper](../scrapers/file.md) -```yaml -file: - - type: $.Config.InstanceType - id: $.Config.InstanceId - path: - - my-config.json +```yaml title="file-scraper.yaml" +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: file-scraper +spec: + file: + - type: $.Config.InstanceType + id: $.Config.InstanceId + path: + - my-config.json ``` Suppose that `my-config.json` file referenced in the path above contains the following JSON structure diff --git a/mission-control/docs/config-db/concepts/transform.md b/mission-control/docs/config-db/concepts/transform.md index f0cc4e03..a04aa051 100644 --- a/mission-control/docs/config-db/concepts/transform.md +++ b/mission-control/docs/config-db/concepts/transform.md @@ -72,20 +72,25 @@ You can supply a JavaScript code to transform the scraped configuration. Your JS _Example_: The following `Config DB` configuration specifies a transformation that'll add a new field `"hello"` with the value `"world"` to all the scraped configurations. -```yaml -file: - - type: Config - id: $[0].id - name: $[0].name - transform: - script: - javascript: |+ - for (var i = 0; i < config.length; i++) { - config[i].hello = "world" - } - JSON.stringify(config) - paths: - - fixtures/data/multiple-configs.json +```yaml title="file-scraper.yaml" +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: file-scraper +spec: + file: + - type: Config + id: $[0].id + name: $[0].name + transform: + script: + javascript: |+ + for (var i = 0; i < config.length; i++) { + config[i].hello = "world" + } + JSON.stringify(config) + paths: + - fixtures/data/multiple-configs.json ``` Considering that the `fixtures/data/multiple-configs.json` file contains the following configuration @@ -118,22 +123,27 @@ The JS transformation will result in two new config items Go template is another powerful way to transform the scraped configuration. Just as you provide a javascript code, you can also provide a Go template. The Go template will have access to the special `config` variable which will contain the scraped config. -```yaml -file: - - type: Config - id: '$.id' - name: 'scraped' - transform: - script: - gotemplate: |+ - [{ - {{range .config}} - "name-{{.id}}": "hi {{.name}}", - {{end}} - "id": "what" - }] - paths: - - fixtures/data/multiple-configs.json +```yaml title="file-scraper.yaml" +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: file-scraper +spec: + file: + - type: Config + id: '$.id' + name: 'scraped' + transform: + script: + gotemplate: |+ + [{ + {{range .config}} + "name-{{.id}}": "hi {{.name}}", + {{end}} + "id": "what" + }] + paths: + - fixtures/data/multiple-configs.json ``` The above transformation will result in the following config @@ -185,22 +195,27 @@ At the moment, only `md5sum` is supported. More hash functions will be added in ## Changes +Changes define how the config changes should be transformed. At the moment, only change exclusion is supported which lets you selectively discard changes that are not relevant. + | Field | Description | Scheme | Required | | --------- | ------------------------------------------------------ | ---------- | -------- | | `exclude` | A list of CEL expressions that excludes a given change | `[]string` | | The scraped changes can be accessed using the `details` field. -```yaml title="exclude-canary-pass-fail.yaml" +```yaml title="kubernetes-scraper.yaml" +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: kubernetes-scraper spec: kubernetes: - clusterName: local-kind-cluster transform: - change: + changes: exclude: - # Canary checker events are handled natively, no need to import the K8S events - - 'details.source.component == "canary-checker" && details.reason == "Failed"' - - 'details.source.component == "canary-checker" && details.reason == "Succeeded"' + - 'config_type == "Kubernetes::Node" && details.message == "status.images"' + - 'details.source.component == "canary-checker" && (change_type == "Failed" || change_type == "Pass")' ``` ## Date Mapping @@ -211,31 +226,41 @@ You'll be making use of the `createFields` and `deleteFields` fields that are su Consider the following configuration file -```yaml -file: - - type: $.aws[0].region - id: $.aws[0].region - createFields: - - $.aws[0].made_at - - $.aws[0].created_at - deleteFields: - - '$.aws[0].removed_at' - - '$.aws[0].deleted_at' - paths: - - fixtures/data/test.yaml +```yaml title="kubernetes-scraper.yaml" +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: kubernetes-scraper +spec: + file: + - type: $.aws[0].region + id: $.aws[0].region + createFields: + - $.aws[0].made_at + - $.aws[0].created_at + deleteFields: + - '$.aws[0].removed_at' + - '$.aws[0].deleted_at' + paths: + - fixtures/data/test.yaml ``` where `fixtures/data/test.yaml` is -```yaml -aws: - - region: eu-west-1 - compliance: true - patch_states: true - patch_details: true - inventory: true - made_at: '2017-03-06T21:04:11Z' - deleted_at: '2017-04-04T15:04:05Z' +```yaml title="aws-scraper.yaml" +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: aws-scraper +spec: + aws: + - region: eu-west-1 + compliance: true + patch_states: true + patch_details: true + inventory: true + made_at: '2017-03-06T21:04:11Z' + deleted_at: '2017-04-04T15:04:05Z' ``` When the scraped configuration is saved in the database, the created date will be `2017-03-06T21:04:11Z` instead of being the current time and the deleted date will be `2017-04-04T15:04:05Z` instead of being empty. diff --git a/mission-control/docs/config-db/scrapers/azure-devops.md b/mission-control/docs/config-db/scrapers/azure-devops.md index 8223c8b6..3fecc990 100644 --- a/mission-control/docs/config-db/scrapers/azure-devops.md +++ b/mission-control/docs/config-db/scrapers/azure-devops.md @@ -4,19 +4,24 @@ The Azure Devops scanner will create a new configuration item for each unique pi Each time the pipeline is run it will create a change for that configuration item. -```yaml -azureDevops: - - organization: - personalAccessToken: - valueFrom: - secretKeyRef: - name: ado-credentials - key: TOKEN - projects: - - # leave empty to select all projects - pipelines: - - # leave empty to select all pipelines - type: Release # the change type the pipeline produces +```yaml title="azure-devops-scraper.yaml" +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: azure-devops +spec: + azureDevops: + - organization: + personalAccessToken: + valueFrom: + secretKeyRef: + name: ado-credentials + key: TOKEN + projects: + - # leave empty to select all projects + pipelines: + - # leave empty to select all pipelines + type: Release # the change type the pipeline produces ``` ## Scraper diff --git a/mission-control/docs/config-db/scrapers/file.md b/mission-control/docs/config-db/scrapers/file.md index 4d0cc5db..7679892b 100644 --- a/mission-control/docs/config-db/scrapers/file.md +++ b/mission-control/docs/config-db/scrapers/file.md @@ -2,13 +2,18 @@ The file config type is set to scrape configurations or configurations in common with the related elements that can be specified in the fields; `type`, and `id`. The paths to the configuration(s) to be scraped is set with the field `path` as a list. -```yaml -file: - - type: $.Config.InstanceType - id: $.Config.InstanceId - path: - - config*.json - - test*.json +```yaml title='file-scraper.yaml' +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: file-scraper +spec: + file: + - type: $.Config.InstanceType + id: $.Config.InstanceId + path: + - config*.json + - test*.json ``` For more examples of configuration file, please check the [GitHub repo](https://github.com/flanksource/config-db/tree/main/fixtures) diff --git a/mission-control/docs/config-db/scrapers/kubernetes-file.md b/mission-control/docs/config-db/scrapers/kubernetes-file.md index 9d191bac..4cd1ae53 100644 --- a/mission-control/docs/config-db/scrapers/kubernetes-file.md +++ b/mission-control/docs/config-db/scrapers/kubernetes-file.md @@ -2,7 +2,12 @@ The KubernetesFile config type is used to scrape configurations contained in your specified resource e.g Pod. -```yaml +```yaml title='k8s-file-scraper.yaml' +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: k8s-file-scraper +spec: kubernetesFile: - selector: namespace: demo @@ -39,7 +44,7 @@ The KubernetesFile config type is used to scrape configurations contained in you | `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | | `selector` | Specify Kubernetes resource for configuration based on `namespace`, `kind`, `name` and more. | [`ResourceSelector`](./resourceselector) | yes | | `container` | Set container name | `string` | | -| `files` | Specify path to file contained in Pod | `[]File` | | +| `files` | Specify path to file contained in Pod | `[]File` | | ### File diff --git a/mission-control/docs/config-db/scrapers/kubernetes.md b/mission-control/docs/config-db/scrapers/kubernetes.md index 51968166..8287f39c 100644 --- a/mission-control/docs/config-db/scrapers/kubernetes.md +++ b/mission-control/docs/config-db/scrapers/kubernetes.md @@ -2,34 +2,39 @@ The `kubernetes` config type scrapes the configurations of your Kubernetes resources as specified with the fields; `namespace`, `selector`, `fieldSelector` and more. -```yaml -kubernetes: - - clusterName: local-kind-cluster - exclusions: - - Secret - - ReplicaSet - - APIService - - endpoints.discovery.k8s.io - - endpointslices.discovery.k8s.io - - leases.coordination.k8s.io - - podmetrics.metrics.k8s.io - - nodemetrics.metrics.k8s.io - - customresourcedefinition - - controllerrevision - - certificaterequest - - orders.acme.cert-manager.io - event: +```yaml title='kubernetes-scraper.yaml' +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: kubernetes-scraper +spec: + kubernetes: + - clusterName: local-kind-cluster exclusions: - - SuccessfulCreate - - Created - - DNSConfigForming - severityKeywords: - error: - - failed - - error - warn: - - backoff - - nodeoutofmemory + - Secret + - ReplicaSet + - APIService + - endpoints.discovery.k8s.io + - endpointslices.discovery.k8s.io + - leases.coordination.k8s.io + - podmetrics.metrics.k8s.io + - nodemetrics.metrics.k8s.io + - customresourcedefinition + - controllerrevision + - certificaterequest + - orders.acme.cert-manager.io + event: + exclusions: + - SuccessfulCreate + - Created + - DNSConfigForming + severityKeywords: + error: + - failed + - error + warn: + - backoff + - nodeoutofmemory ``` ## Scraper diff --git a/mission-control/docs/config-db/scrapers/sql.md b/mission-control/docs/config-db/scrapers/sql.md index 8abc220d..12b1161f 100644 --- a/mission-control/docs/config-db/scrapers/sql.md +++ b/mission-control/docs/config-db/scrapers/sql.md @@ -4,66 +4,71 @@ The SQL configuration scraper will execute a SQL query and then create a configu The example below creates a new `MSSQL::Database` configuration for each database on the sql server, and then creates a roles object container the SQL Server login to database role mapping. With change detection this will highlight when new users are added / modified / removed on an individual databases. -```yaml -sql: - - connection: "sqlserver://localhost:1433?database=master" - auth: - username: - value: sa - password: - value: password - type: MSSQL::Database - id: $.name - - transform: - full: true # transform the entire configuration item, and not just the configuration data (row) - script: - javascript: |+ - var dbs = {} - for (var i = 0; i < config.rows.length; i++) { - var db = config.rows[i] - var name = db.DB - if (dbs[db.DB] == null) { - { - config: dbs[db.DB] = { - name: name, - roles: {} - }, - changes: { - - }, - analysis: { +```yaml title='sql-scraper.yaml' +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: sql-scraper +spec: + sql: + - connection: 'sqlserver://localhost:1433?database=master' + auth: + username: + value: sa + password: + value: password + type: MSSQL::Database + id: $.name + + transform: + full: true # transform the entire configuration item, and not just the configuration data (row) + script: + javascript: |+ + var dbs = {} + for (var i = 0; i < config.rows.length; i++) { + var db = config.rows[i] + var name = db.DB + if (dbs[db.DB] == null) { + { + config: dbs[db.DB] = { + name: name, + roles: {} + }, + changes: { + + }, + analysis: { + + } } - } + dbs[name].roles[db.role] = db.name } - dbs[name].roles[db.role] = db.name - } - JSON.stringify(_.values(dbs)) - - query: | - declare @mytable table ( - [DB] [nvarchar](128) NULL, - [name] [nvarchar](255) NOT NULL, - [role] [nvarchar](255) NOT NULL - ) - - - DECLARE @command varchar(1000) - SELECT @command = - 'USE ?; SELECT DB_NAME() as DB, DP1.name AS [user], - isnull (DP2.name, ''No members'') AS [role] - FROM sys.database_role_members AS DRM - RIGHT OUTER JOIN sys.database_principals AS DP1 - ON DRM.role_principal_id = DP1.principal_id - LEFT OUTER JOIN sys.database_principals AS DP2 - ON DRM.member_principal_id = DP2.principal_id - WHERE DP1.type = ''R'' and DP2.name is not null' - - insert into @mytable EXEC sp_MSforeachdb @command - - select * from @mytable + JSON.stringify(_.values(dbs)) + + query: | + declare @mytable table ( + [DB] [nvarchar](128) NULL, + [name] [nvarchar](255) NOT NULL, + [role] [nvarchar](255) NOT NULL + ) + + + DECLARE @command varchar(1000) + SELECT @command = + 'USE ?; SELECT DB_NAME() as DB, DP1.name AS [user], + isnull (DP2.name, ''No members'') AS [role] + FROM sys.database_role_members AS DRM + RIGHT OUTER JOIN sys.database_principals AS DP1 + ON DRM.role_principal_id = DP1.principal_id + LEFT OUTER JOIN sys.database_principals AS DP2 + ON DRM.member_principal_id = DP2.principal_id + WHERE DP1.type = ''R'' and DP2.name is not null' + + insert into @mytable EXEC sp_MSforeachdb @command + + select * from @mytable ``` ## Scraper diff --git a/mission-control/docs/config-db/scrapers/trivy.md b/mission-control/docs/config-db/scrapers/trivy.md index 643ead04..fb9ff922 100644 --- a/mission-control/docs/config-db/scrapers/trivy.md +++ b/mission-control/docs/config-db/scrapers/trivy.md @@ -2,11 +2,16 @@ The `trivy` scraper uses [Trivy](https://trivy.dev/) to scan for security vulnerabilities & misconfigurations in your configuration. At the moment, there's only support for scanning Kubernetes objects. -```yaml -trivy: - - version: "0.40.0" - kubernetes: - namespace: production +```yaml title="trivy-scraper.yaml" +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: trivy-scraper +spec: + trivy: + - version: '0.40.0' + kubernetes: + namespace: production ``` Unlike other scrapers, this one does not scape new configs but rather look for security vulnerabilities in the existing configs. This scrapper, if configured to scan a kubernetes cluster, will map all the found vulnerabilities to the corresponding config item. From df13233c1003ee488f79b43cb09c08539256a77d Mon Sep 17 00:00:00 2001 From: Aditya Thebe Date: Tue, 20 Feb 2024 10:40:21 +0545 Subject: [PATCH 6/9] chore: improve relationship examples. added reference to Relationship lookup --- .../docs/config-db/concepts/transform.md | 28 ++---- .../examples/forming-relationship.md | 53 ++++++++++ .../examples/kubernetes-relationship.md | 53 +++++----- .../references/relationship-lookup.md | 9 ++ .../docs/config-db/scrapers/kubernetes.md | 98 ++++++------------- mission-control/sidebars.js | 20 +++- 6 files changed, 145 insertions(+), 116 deletions(-) create mode 100644 mission-control/docs/config-db/examples/forming-relationship.md create mode 100644 mission-control/docs/config-db/references/relationship-lookup.md diff --git a/mission-control/docs/config-db/concepts/transform.md b/mission-control/docs/config-db/concepts/transform.md index a04aa051..d79fc57a 100644 --- a/mission-control/docs/config-db/concepts/transform.md +++ b/mission-control/docs/config-db/concepts/transform.md @@ -29,15 +29,15 @@ This transformation function allows you to dynamically form relationships betwee Example: You can link a kubernetes deployment with the corresponding pods, or you can link AWS EC2 instances with the AWS Account. It's even possible to link two configs scraped by different scrape configs like: linking a Kubernetes Node in an EKS cluster to the EC2 instance. -| Field | Description | Scheme | Required | -| -------- | ------------------------------------------------------------------------------------- | ------------------------------------------- | -------- | -| `filter` | Specify the config item with which relationship should be formed | `string` | `true` | -| `expr` | cel-expression that returns a list of [relationship selector](#relationshipselector). | `string` | | -| `id` | id of the config to link to | [`RelationshipLookup`](#relationshiplookup) | | -| `name` | id of the config to link to | [`RelationshipLookup`](#relationshiplookup) | | -| `type` | id of the config to link to | [`RelationshipLookup`](#relationshiplookup) | | -| `agent` | id of the config to link to | [`RelationshipLookup`](#relationshiplookup) | | -| `labels` | Labels of the config to link to | [`RelationshipLookup`](#relationshiplookup) | | +| Field | Description | Scheme | Required | +| -------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------- | -------- | +| `filter` | Specify the config item with which relationship should be formed | `string` | `true` | +| `expr` | cel-expression that returns a list of [relationship selector](#relationshipselector). | `string` | | +| `id` | id of the config to link to | [`RelationshipLookup`](../references/relationship-lookup) | | +| `name` | id of the config to link to | [`RelationshipLookup`](../references/relationship-lookup) | | +| `type` | id of the config to link to | [`RelationshipLookup`](../references/relationship-lookup) | | +| `agent` | id of the config to link to | [`RelationshipLookup`](../references/relationship-lookup) | | +| `labels` | Labels of the config to link to | [`RelationshipLookup`](../references/relationship-lookup) | | :::info `expr` is an alternative, more flexible, way to define the selectors. Either use `expr` or the other selector fields (`id`, `name`, `type`, `agent`, `labels`) but not both. @@ -54,16 +54,6 @@ Example: You can link a kubernetes deployment with the corresponding pods, or yo | `agent` | id of the config to link to | `string` | | | `labels` | Labels of the config to link to | `string` | | -### RelationshipLookup - -RelationshipLookup offers different ways to specify a lookup value - -| Field | Description | Scheme | Required | -| ------- | ---------------------------------- | -------- | -------- | -| `expr` | Use an expression to get the value | `string` | | -| `value` | Specify a static value | `string` | | -| `label` | Get the value from a label | `string` | | - ## Script ### JavaScript diff --git a/mission-control/docs/config-db/examples/forming-relationship.md b/mission-control/docs/config-db/examples/forming-relationship.md new file mode 100644 index 00000000..e3cd42d4 --- /dev/null +++ b/mission-control/docs/config-db/examples/forming-relationship.md @@ -0,0 +1,53 @@ +# Forming Relationships + +## Kubernetes + +This example demonstrates 2 different ways you can form relationships between config items. + +The first relationship is formed between a Kubernetes service and its corresponding deployment using the inline relationship selector _(type & name)_ while the second relationship is formed between Pods and PVCs using the `expr` way. + +```yaml title="kubernetes-scraper.yaml" +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: kubernetes-scraper +spec: + kubernetes: + - clusterName: local-kind-cluster + transform: + relationship: + # Link a service to a deployment (adjust the label selector accordingly) + - filter: config_type == "Kubernetes::Service" + type: + value: 'Kubernetes::Deployment' + name: + expr: | + has(config.spec.selector) && has(config.spec.selector.name) ? config.spec.selector.name : '' + # Link Pods to PVCs + - filter: config_type == 'Kubernetes::Pod' + expr: | + config.spec.volumes. + filter(item, has(item.persistentVolumeClaim)). + map(item, {"type": "Kubernetes::PersistentVolumeClaim", "name": item.persistentVolumeClaim.claimName}). + toJSON() +``` + +## AWS + +```yaml title='aws-scraper.yaml' +apiVersion: configs.flanksource.com/v1 +kind: ScrapeConfig +metadata: + name: aws-scraper +spec: + aws: + - region: + - eu-west-2 + - us-east-1 + transform: + relationship: + # Region to ZoneID + - filter: config_type == 'AWS::Region' + expr: | + [{"type": "AWS::AvailabilityZoneID", "labels": {"region": name}}].toJSON() +``` diff --git a/mission-control/docs/config-db/examples/kubernetes-relationship.md b/mission-control/docs/config-db/examples/kubernetes-relationship.md index 73072fb4..3198a78f 100644 --- a/mission-control/docs/config-db/examples/kubernetes-relationship.md +++ b/mission-control/docs/config-db/examples/kubernetes-relationship.md @@ -1,31 +1,32 @@ # Kubernetes Relationship -This example demonstrates 2 different ways you can form relationships between config items. +Kubernetes scraper offers a more tailored relationship selector in addition to the [general relationship selector](../concepts/transform.md#relationshipconfig). -The first relationship is formed between a Kubernetes service and its corresponding deployment using the inline relationship selector _(type & name)_ while the second relationship is formed between Pods and PVCs using the `expr` way. +```yaml title="kubernetes-relationship.yaml" +kubernetes: + - clusterName: 'eks' + relationships: + # If object has spec.claimRef field, use its kind, name and namespace + - kind: + expr: "has(spec.claimRef) ? spec.claimRef.kind : ''" + name: + expr: "has(spec.claimRef) ? spec.claimRef.name : ''" + namespace: + expr: "has(spec.claimRef) ? spec.claimRef.namespace : ''" -```yaml title="kubernetes-scraper.yaml" -apiVersion: configs.flanksource.com/v1 -kind: ScrapeConfig -metadata: - name: kubernetes-scraper -spec: - kubernetes: - - clusterName: local-kind-cluster - transform: - relationship: - # Link a service to a deployment (adjust the label selector accordingly) - - filter: config_type == "Kubernetes::Service" - type: - value: 'Kubernetes::Deployment' - name: - expr: | - has(config.spec.selector) && has(config.spec.selector.name) ? config.spec.selector.name : '' - # Link Pods to PVCs - - filter: config_type == 'Kubernetes::Pod' - expr: | - config.spec.volumes. - filter(item, has(item.persistentVolumeClaim)). - map(item, {"type": "Kubernetes::PersistentVolumeClaim", "name": item.persistentVolumeClaim.claimName}). - toJSON() + # If object flux kustomize labels, link it to the parent Kustomization object + - kind: + value: Kustomization + name: + label: kustomize.toolkit.fluxcd.io/name + namespace: + label: kustomize.toolkit.fluxcd.io/namespace + + # If object helm kustomize labels, link it to the parent HelmRelease object + - kind: + value: HelmRelease + name: + label: helm.toolkit.fluxcd.io/name + namespace: + label: helm.toolkit.fluxcd.io/namespace ``` diff --git a/mission-control/docs/config-db/references/relationship-lookup.md b/mission-control/docs/config-db/references/relationship-lookup.md new file mode 100644 index 00000000..9aa64c8b --- /dev/null +++ b/mission-control/docs/config-db/references/relationship-lookup.md @@ -0,0 +1,9 @@ +# Relationship Lookup + +RelationshipLookup offers different ways to specify a lookup value + +| Field | Description | Scheme | Required | +| ------- | ---------------------------------- | -------- | -------- | +| `expr` | Use an expression to get the value | `string` | | +| `value` | Specify a static value | `string` | | +| `label` | Get the value from a label | `string` | | diff --git a/mission-control/docs/config-db/scrapers/kubernetes.md b/mission-control/docs/config-db/scrapers/kubernetes.md index 8287f39c..949bc64a 100644 --- a/mission-control/docs/config-db/scrapers/kubernetes.md +++ b/mission-control/docs/config-db/scrapers/kubernetes.md @@ -49,28 +49,28 @@ spec: ### Kubernetes -| Field | Description | Scheme | Required | -| ----------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -------- | -| `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | -| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` | -| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` | -| `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | -| `transform` | Specify field to transform result | [`Transform`](../concepts/transform.md) | `false` | -| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | `false` | -| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` | -| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `clusterName` | Specify cluster name | `string` | | -| `namespace` | Specify namespace for scraping of Kubernetes resources | `string` | | -| `useCache` | Specify boolean value to toggle fetching results from Kube-apiserver or fetch response from etcd | `bool` | | -| `scope` | Specify scope for scrape. e.g `cluster` for scraping at Cluster level | `string` | | -| `since` | Set time constraint for scraping resources within the set period | `string` | | -| `selector` | Specify Kubernetes resource to scrape based on selector. e.g `matchLabels` | `string` | | -| `fieldSelector` | Specify Kubernetes resource based on value of resource fields. e.g `status.Phase=Running` | `string` | | -| `exclusions` | Specify Kubernetes resources to be excluded from scraping | `[]string` | | -| **`kubeconfig`** | Specify kubeconfig for access to your Kubernetes Cluster | [`kommons.EnvVar`](https://pkg.go.dev/github.com/flanksource/kommons#EnvVar) | yes | -| `event` | Specify configuration to handle Kubernetes events. | [`Event`](#sevent) | yes | -| `relationships` | Create relationships between kubernetes objects. | [`[]Relationships`](#srelationships) | `false` | +| Field | Description | Scheme | Required | +| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | -------- | +| `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | +| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` | +| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` | +| `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | +| `transform` | Specify field to transform result | [`Transform`](../concepts/transform.md) | `false` | +| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | `false` | +| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` | +| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | +| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | +| `clusterName` | Specify cluster name | `string` | | +| `namespace` | Specify namespace for scraping of Kubernetes resources | `string` | | +| `useCache` | Specify boolean value to toggle fetching results from Kube-apiserver or fetch response from etcd | `bool` | | +| `scope` | Specify scope for scrape. e.g `cluster` for scraping at Cluster level | `string` | | +| `since` | Set time constraint for scraping resources within the set period | `string` | | +| `selector` | Specify Kubernetes resource to scrape based on selector. e.g `matchLabels` | `string` | | +| `fieldSelector` | Specify Kubernetes resource based on value of resource fields. e.g `status.Phase=Running` | `string` | | +| `exclusions` | Specify Kubernetes resources to be excluded from scraping | `[]string` | | +| **`kubeconfig`** | Specify kubeconfig for access to your Kubernetes Cluster | [`kommons.EnvVar`](https://pkg.go.dev/github.com/flanksource/kommons#EnvVar) | yes | +| `event` | Specify configuration to handle Kubernetes events. | [`Event`](#sevent) | yes | +| `relationships` | Create relationships between kubernetes objects. | [`[]Relationship`](#kubernetes-relationship) | `false` | ### Events @@ -90,55 +90,15 @@ In addition, you can also specify keywords used to identify the severity of the | `warn` | A list of keywords used to identify a warning severity from the reason. It could also be a match pattern: example "\*" to match all or "!badword" to exclude "badword" | `[]string` | `false` | | `error` | Same as `warn` but used to map to error severity. | `[]string` | `false` | -### KubernetesRelationships +### Kubernetes Relationship You can create relationships between kubernetes objects on the basis of kind, name and labels. While relationships between node and pod, deployment and pod, namespace and deployment -are created automatically, there are cases where we want to link objects on the basis of metadata (like linking resources created by a flux object to it). +are created automatically, there are cases where we want to link objects on the basis of metadata (like linking resources created by a flux object to it). [See Example](../examples/kubernetes-relationship). We support static values, expressions and label lookups to find the parent -| Field | Description | Scheme | Required | -| ----------- | -------------------------------- | --------------------------------------------------------------- | -------- | -| `kind` | `kind` of Kubernetes Object | [`RelationshipLookup`](#relationshiplookup) | `true` | -| `name` | `name` of Kubernetes Object | [`RelationshipLookup`](#relationshiplookup) | `true` | -| `namespace` | `namespace` of Kubernetes Object | [`RelationshipLookup`](#relationshiplookup) | `true` | - -#### RelationshipLookup - -| Field | Description | Scheme | Required | -| ------- | -------------------------------------------------- | -------- | -------- | -| `value` | Static string value of the resource | `string` | | -| `expr` | CEL Expression to evaluate | `string` | | -| `label` | Label key containing the value of the the resource | `string` | | - -#### - -```yaml title="relationship-example.yaml" -kubernetes: - - clusterName: 'eks' - ... - relationships: - # If object has spec.claimRef field, use its kind, name and namespace - - kind: - expr: "has(spec.claimRef) ? spec.claimRef.kind : ''" - name: - expr: "has(spec.claimRef) ? spec.claimRef.name : ''" - namespace: - expr: "has(spec.claimRef) ? spec.claimRef.namespace : ''" - - # If object flux kustomize labels, link it to the parent Kustomization object - - kind: - value: Kustomization - name: - label: kustomize.toolkit.fluxcd.io/name - namespace: - label: kustomize.toolkit.fluxcd.io/namespace - - # If object helm kustomize labels, link it to the parent HelmRelease object - - kind: - value: HelmRelease - name: - label: helm.toolkit.fluxcd.io/name - namespace: - label: helm.toolkit.fluxcd.io/namespace -``` +| Field | Description | Scheme | Required | +| ----------- | -------------------------------- | --------------------------------------------------------- | -------- | +| `kind` | `kind` of Kubernetes Object | [`RelationshipLookup`](../references/relationship-lookup) | `true` | +| `name` | `name` of Kubernetes Object | [`RelationshipLookup`](../references/relationship-lookup) | `true` | +| `namespace` | `namespace` of Kubernetes Object | [`RelationshipLookup`](../references/relationship-lookup) | `true` | diff --git a/mission-control/sidebars.js b/mission-control/sidebars.js index e1452048..f870e8f9 100644 --- a/mission-control/sidebars.js +++ b/mission-control/sidebars.js @@ -452,14 +452,19 @@ const sidebars = { } ] }, - { + { type: 'category', label: 'Examples', items: [ { type: 'doc', - id: 'config-db/examples/kubernetes-relationship', + id: 'config-db/examples/forming-relationship', label: 'Forming relationships' + }, + { + type: 'doc', + id: 'config-db/examples/kubernetes-relationship', + label: 'Kubernetes relationships' }, { type: 'doc', @@ -473,6 +478,17 @@ const sidebars = { }, ] }, + { + type: 'category', + label: 'References', + items: [ + { + type: 'doc', + id: 'config-db/references/relationship-lookup', + label: 'Relationship Lookup' + }, + ] + }, ], playbooksSidebar: [ { From 3f3c62b81b4a954acfc5cca234cc2b76a5a8a73c Mon Sep 17 00:00:00 2001 From: Aditya Thebe Date: Tue, 20 Feb 2024 10:44:37 +0545 Subject: [PATCH 7/9] fix: common link --- docs/apm-hub/backends/kubernetes.md | 2 +- .../docs/config-db/scrapers/azure-devops.md | 30 +++++++++---------- .../docs/config-db/scrapers/kubernetes.md | 2 +- mission-control/sidebars.js | 5 ++++ 4 files changed, 22 insertions(+), 17 deletions(-) diff --git a/docs/apm-hub/backends/kubernetes.md b/docs/apm-hub/backends/kubernetes.md index 012cb748..fa56fb78 100644 --- a/docs/apm-hub/backends/kubernetes.md +++ b/docs/apm-hub/backends/kubernetes.md @@ -25,5 +25,5 @@ spec: | ------------ | ------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | ---------- | | `routes` | Specify routes that would match this backend.
_(Read more [Routing](../concepts/routing.md))_ | [`[]Route`](../concepts/routing.md#route) | `true` | | `labels` | A set of key value pairs that'll be attached to individual items in the search result | `map[string]string` | `optional` | -| `kubeconfig` | Specify configuration for Kubernetes connection.
empty kubeconfig means the current kubeconfig will be used for connection. | [`kommons.EnvVar`](https://pkg.go.dev/github.com/flanksource/kommons#EnvVar) | `true` | +| `kubeconfig` | Specify configuration for Kubernetes connection.
empty kubeconfig means the current kubeconfig will be used for connection. | []_EnvVar_ | `true` | | `namespace` | Specify the namespace for the kubeconfig. | `string` | `true` | diff --git a/mission-control/docs/config-db/scrapers/azure-devops.md b/mission-control/docs/config-db/scrapers/azure-devops.md index 3fecc990..0452ce2f 100644 --- a/mission-control/docs/config-db/scrapers/azure-devops.md +++ b/mission-control/docs/config-db/scrapers/azure-devops.md @@ -36,18 +36,18 @@ spec: ### AzureDevops -| Field | Description | Scheme | Required | -| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | -------- | -| `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | -| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` | -| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` | -| `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | -| `transform` | Specify field to transform result | [`Transform`](../concepts/transform.md) | `false` | -| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | `false` | -| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` | -| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `organization` | Specifies the name of the Azure DevOps organization to scrape | `string` | `true` | -| `personalAccessToken` | Specifies the personal access token to authenticate with Azure DevOps | [`kommons.EnvVar`](https://pkg.go.dev/github.com/flanksource/kommons#EnvVar) | `true` | -| `projects` | Specifies the Azure DevOps projects to scrape | `[]string` | `false` | -| `pipelines` | Specifies the Azure DevOps pipelines to scrape | `[]string` | `false` | +| Field | Description | Scheme | Required | +| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | -------- | +| `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | +| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` | +| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` | +| `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | +| `transform` | Specify field to transform result | [`Transform`](../concepts/transform.md) | `false` | +| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | `false` | +| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` | +| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | +| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | +| `organization` | Specifies the name of the Azure DevOps organization to scrape | `string` | `true` | +| `personalAccessToken` | Specifies the personal access token to authenticate with Azure DevOps | []_EnvVar_ | `true` | +| `projects` | Specifies the Azure DevOps projects to scrape | `[]string` | `false` | +| `pipelines` | Specifies the Azure DevOps pipelines to scrape | `[]string` | `false` | diff --git a/mission-control/docs/config-db/scrapers/kubernetes.md b/mission-control/docs/config-db/scrapers/kubernetes.md index 949bc64a..76af4b50 100644 --- a/mission-control/docs/config-db/scrapers/kubernetes.md +++ b/mission-control/docs/config-db/scrapers/kubernetes.md @@ -68,7 +68,7 @@ spec: | `selector` | Specify Kubernetes resource to scrape based on selector. e.g `matchLabels` | `string` | | | `fieldSelector` | Specify Kubernetes resource based on value of resource fields. e.g `status.Phase=Running` | `string` | | | `exclusions` | Specify Kubernetes resources to be excluded from scraping | `[]string` | | -| **`kubeconfig`** | Specify kubeconfig for access to your Kubernetes Cluster | [`kommons.EnvVar`](https://pkg.go.dev/github.com/flanksource/kommons#EnvVar) | yes | +| **`kubeconfig`** | Specify kubeconfig for access to your Kubernetes Cluster | []_EnvVar_ | yes | | `event` | Specify configuration to handle Kubernetes events. | [`Event`](#sevent) | yes | | `relationships` | Create relationships between kubernetes objects. | [`[]Relationship`](#kubernetes-relationship) | `false` | diff --git a/mission-control/sidebars.js b/mission-control/sidebars.js index f870e8f9..f8647eb2 100644 --- a/mission-control/sidebars.js +++ b/mission-control/sidebars.js @@ -922,6 +922,11 @@ const sidebars = { id: 'reference/secret-management', label: 'Secret Management', }, + { + type: 'doc', + id: 'reference/property', + label: 'Property', + }, { type: 'doc', id: 'reference/connection', From 440cd5e24a8706ace00f9759768154fd9412193e Mon Sep 17 00:00:00 2001 From: Aditya Thebe Date: Tue, 20 Feb 2024 15:27:49 +0545 Subject: [PATCH 8/9] chore: inline relationship lookup c# Please enter the commit message for your changes. Lines starting --- .../docs/config-db/concepts/transform.md | 28 ++++++--- .../references/relationship-lookup.md | 9 --- .../docs/config-db/scrapers/kubernetes.md | 62 +++++++++++-------- mission-control/sidebars.js | 11 ---- 4 files changed, 55 insertions(+), 55 deletions(-) delete mode 100644 mission-control/docs/config-db/references/relationship-lookup.md diff --git a/mission-control/docs/config-db/concepts/transform.md b/mission-control/docs/config-db/concepts/transform.md index d79fc57a..1cd217e1 100644 --- a/mission-control/docs/config-db/concepts/transform.md +++ b/mission-control/docs/config-db/concepts/transform.md @@ -29,15 +29,15 @@ This transformation function allows you to dynamically form relationships betwee Example: You can link a kubernetes deployment with the corresponding pods, or you can link AWS EC2 instances with the AWS Account. It's even possible to link two configs scraped by different scrape configs like: linking a Kubernetes Node in an EKS cluster to the EC2 instance. -| Field | Description | Scheme | Required | -| -------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------- | -------- | -| `filter` | Specify the config item with which relationship should be formed | `string` | `true` | -| `expr` | cel-expression that returns a list of [relationship selector](#relationshipselector). | `string` | | -| `id` | id of the config to link to | [`RelationshipLookup`](../references/relationship-lookup) | | -| `name` | id of the config to link to | [`RelationshipLookup`](../references/relationship-lookup) | | -| `type` | id of the config to link to | [`RelationshipLookup`](../references/relationship-lookup) | | -| `agent` | id of the config to link to | [`RelationshipLookup`](../references/relationship-lookup) | | -| `labels` | Labels of the config to link to | [`RelationshipLookup`](../references/relationship-lookup) | | +| Field | Description | Scheme | Required | +| -------- | ------------------------------------------------------------------------------------- | -------------------------------------------- | -------- | +| `filter` | Specify the config item with which relationship should be formed | `string` | `true` | +| `expr` | cel-expression that returns a list of [relationship selector](#relationshipselector). | `string` | | +| `id` | id of the config to link to | [`RelationshipLookup`](#relationship-lookup) | | +| `name` | id of the config to link to | [`RelationshipLookup`](#relationship-lookup) | | +| `type` | id of the config to link to | [`RelationshipLookup`](#relationship-lookup) | | +| `agent` | id of the config to link to | [`RelationshipLookup`](#relationship-lookup) | | +| `labels` | Labels of the config to link to | [`RelationshipLookup`](#relationship-lookup) | | :::info `expr` is an alternative, more flexible, way to define the selectors. Either use `expr` or the other selector fields (`id`, `name`, `type`, `agent`, `labels`) but not both. @@ -54,6 +54,16 @@ Example: You can link a kubernetes deployment with the corresponding pods, or yo | `agent` | id of the config to link to | `string` | | | `labels` | Labels of the config to link to | `string` | | +### Relationship Lookup + +RelationshipLookup offers different ways to specify a lookup value + +| Field | Description | Scheme | Required | +| ------- | ---------------------------------- | -------- | -------- | +| `expr` | Use an expression to get the value | `string` | | +| `value` | Specify a static value | `string` | | +| `label` | Get the value from a label | `string` | | + ## Script ### JavaScript diff --git a/mission-control/docs/config-db/references/relationship-lookup.md b/mission-control/docs/config-db/references/relationship-lookup.md deleted file mode 100644 index 9aa64c8b..00000000 --- a/mission-control/docs/config-db/references/relationship-lookup.md +++ /dev/null @@ -1,9 +0,0 @@ -# Relationship Lookup - -RelationshipLookup offers different ways to specify a lookup value - -| Field | Description | Scheme | Required | -| ------- | ---------------------------------- | -------- | -------- | -| `expr` | Use an expression to get the value | `string` | | -| `value` | Specify a static value | `string` | | -| `label` | Get the value from a label | `string` | | diff --git a/mission-control/docs/config-db/scrapers/kubernetes.md b/mission-control/docs/config-db/scrapers/kubernetes.md index 76af4b50..e3d32793 100644 --- a/mission-control/docs/config-db/scrapers/kubernetes.md +++ b/mission-control/docs/config-db/scrapers/kubernetes.md @@ -49,28 +49,28 @@ spec: ### Kubernetes -| Field | Description | Scheme | Required | -| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | -------- | -| `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | -| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` | -| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` | -| `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | -| `transform` | Specify field to transform result | [`Transform`](../concepts/transform.md) | `false` | -| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | `false` | -| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` | -| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `clusterName` | Specify cluster name | `string` | | -| `namespace` | Specify namespace for scraping of Kubernetes resources | `string` | | -| `useCache` | Specify boolean value to toggle fetching results from Kube-apiserver or fetch response from etcd | `bool` | | -| `scope` | Specify scope for scrape. e.g `cluster` for scraping at Cluster level | `string` | | -| `since` | Set time constraint for scraping resources within the set period | `string` | | -| `selector` | Specify Kubernetes resource to scrape based on selector. e.g `matchLabels` | `string` | | -| `fieldSelector` | Specify Kubernetes resource based on value of resource fields. e.g `status.Phase=Running` | `string` | | -| `exclusions` | Specify Kubernetes resources to be excluded from scraping | `[]string` | | +| Field | Description | Scheme | Required | +| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | -------- | +| `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | +| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` | +| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` | +| `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | +| `transform` | Specify field to transform result | [`Transform`](../concepts/transform.md) | `false` | +| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | `false` | +| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` | +| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | +| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | +| `clusterName` | Specify cluster name | `string` | | +| `namespace` | Specify namespace for scraping of Kubernetes resources | `string` | | +| `useCache` | Specify boolean value to toggle fetching results from Kube-apiserver or fetch response from etcd | `bool` | | +| `scope` | Specify scope for scrape. e.g `cluster` for scraping at Cluster level | `string` | | +| `since` | Set time constraint for scraping resources within the set period | `string` | | +| `selector` | Specify Kubernetes resource to scrape based on selector. e.g `matchLabels` | `string` | | +| `fieldSelector` | Specify Kubernetes resource based on value of resource fields. e.g `status.Phase=Running` | `string` | | +| `exclusions` | Specify Kubernetes resources to be excluded from scraping | `[]string` | | | **`kubeconfig`** | Specify kubeconfig for access to your Kubernetes Cluster | []_EnvVar_ | yes | -| `event` | Specify configuration to handle Kubernetes events. | [`Event`](#sevent) | yes | -| `relationships` | Create relationships between kubernetes objects. | [`[]Relationship`](#kubernetes-relationship) | `false` | +| `event` | Specify configuration to handle Kubernetes events. | [`Event`](#sevent) | yes | +| `relationships` | Create relationships between kubernetes objects. | [`[]Relationship`](#kubernetes-relationship) | `false` | ### Events @@ -97,8 +97,18 @@ are created automatically, there are cases where we want to link objects on the We support static values, expressions and label lookups to find the parent -| Field | Description | Scheme | Required | -| ----------- | -------------------------------- | --------------------------------------------------------- | -------- | -| `kind` | `kind` of Kubernetes Object | [`RelationshipLookup`](../references/relationship-lookup) | `true` | -| `name` | `name` of Kubernetes Object | [`RelationshipLookup`](../references/relationship-lookup) | `true` | -| `namespace` | `namespace` of Kubernetes Object | [`RelationshipLookup`](../references/relationship-lookup) | `true` | +| Field | Description | Scheme | Required | +| ----------- | -------------------------------- | -------------------------------------------- | -------- | +| `kind` | `kind` of Kubernetes Object | [`RelationshipLookup`](#relationship-lookup) | `true` | +| `name` | `name` of Kubernetes Object | [`RelationshipLookup`](#relationship-lookup) | `true` | +| `namespace` | `namespace` of Kubernetes Object | [`RelationshipLookup`](#relationship-lookup) | `true` | + +#### Relationship Lookup + +RelationshipLookup offers different ways to specify a lookup value + +| Field | Description | Scheme | Required | +| ------- | ---------------------------------- | -------- | -------- | +| `expr` | Use an expression to get the value | `string` | | +| `value` | Specify a static value | `string` | | +| `label` | Get the value from a label | `string` | | diff --git a/mission-control/sidebars.js b/mission-control/sidebars.js index f8647eb2..5ce00c65 100644 --- a/mission-control/sidebars.js +++ b/mission-control/sidebars.js @@ -478,17 +478,6 @@ const sidebars = { }, ] }, - { - type: 'category', - label: 'References', - items: [ - { - type: 'doc', - id: 'config-db/references/relationship-lookup', - label: 'Relationship Lookup' - }, - ] - }, ], playbooksSidebar: [ { From 7eb080ee40f5caf38134e691c9ad0acf4e0d1db9 Mon Sep 17 00:00:00 2001 From: Aditya Thebe Date: Tue, 20 Feb 2024 16:11:57 +0545 Subject: [PATCH 9/9] chore: update structure descriptions --- .../docs/config-db/scrapers/aws.md | 70 +++++++++---------- .../docs/config-db/scrapers/azure-devops.md | 26 +++---- .../docs/config-db/scrapers/file.md | 27 +++---- .../config-db/scrapers/kubernetes-file.md | 46 +++++------- .../docs/config-db/scrapers/kubernetes.md | 40 +++++------ .../docs/config-db/scrapers/sql.md | 24 +++---- .../docs/config-db/scrapers/trivy.md | 38 +++++----- 7 files changed, 131 insertions(+), 140 deletions(-) diff --git a/mission-control/docs/config-db/scrapers/aws.md b/mission-control/docs/config-db/scrapers/aws.md index 901c6014..ffbf1fb6 100644 --- a/mission-control/docs/config-db/scrapers/aws.md +++ b/mission-control/docs/config-db/scrapers/aws.md @@ -75,51 +75,51 @@ spec: | Field | Description | Scheme | Required | | ----------- | ---------------------------------------------------------------------------------- | -------------------------------------------- | -------- | -| `logLevel` | Specify the level of logging. | `string` | `false` | -| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | `false` | -| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | `false` | +| `logLevel` | Specify the level of logging. | `string` | | +| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | | +| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | | | `retention` | Settings for retaining changes, analysis and scraped items | [`Retention`](/config-db/concepts/retention) | | -| `aws` | Specifies the list of AWS configurations to scrape. | [`[]AWS`](#aws-1) | `false` | +| `aws` | Specifies the list of AWS configurations to scrape. | [`[]AWS`](#aws-1) | | ### AWS -| Field | Description | Scheme | Required | -| -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | -------- | -| `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | -| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` | -| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` | -| `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | -| `transform` | Specify field to transform result | [`Transform`](../concepts/transform.md) | `false` | -| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | `false` | -| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` | -| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `compliance` | Toggle scraping of compliance metadata | _bool_ | | -| `patch_states` | Set boolean value for scraping of patch state status, and report | _bool_ | | -| `trusted_advisor_check` | Set boolean value for enabling/disabling trusted advisor | _bool_ | | -| `patch_details` | Set boolean value to enable/disable scraping of patch details | _bool_ | | -| [`cost_reporting`](#cost-reporting) | Specify cost reporting for scraping of data | [_CostReporting_](#cost-reporting) | | -| [`cloud_trail`](#cloudtrail-cloudtrail-cloudtrail) | Set CloudTrail specifications | [_CloudTrail_](#cloudtrail-cloudtrail) | | -| `include` | Specify AWS resources to include for scraping | _\[\]string_ | | -| `exclude` | Specify AWS resources to exclude from scraping | _\[\]string_ | | -| `inventory` | Toggle scrape of _metadata_ for AWS resources | _bool_ | | -| `properties` | Custom templatable properties for the scraped config items. | [`[]ConfigProperty`](../../reference/property) | | +| Field | Description | Scheme | Required | +| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------- | -------- | +| `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | +| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | | +| `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | +| `cloud_trail` | Set CloudTrail specifications | [`CloudTrail`](#cloudtrail-cloudtrail) | | +| `compliance` | Toggle scraping of compliance metadata | `bool` | | +| `cost_reporting` | Specify cost reporting for scraping of data | [`CostReporting`](#cost-reporting) | | +| `createFields` | List of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used. | `[]string` | | +| `deleteFields` | List of JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used. | `[]string` | | +| `exclude` | AWS resources to exclude from scraping | `[]string` | | +| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | | +| `include` | AWS resources to include for scraping | `[]string` | | +| `inventory` | Toggle scrape of _metadata_ for AWS resources | `bool` | | +| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | | +| `patch_details` | Enable/disable scraping of patch details | `bool` | | +| `patch_states` | Scrape patch state status, and report | `bool` | | +| `properties` | Custom templatable properties for the scraped config items. | [`[]ConfigProperty`](../../reference/property) | | +| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | | +| `transform` | Field to transform result | [`Transform`](../concepts/transform.md) | | +| `trusted_advisor_check` | Enable/Disable scraping analyses from Trusted Advisor | `bool` | | -### CloudTrail (`cloudtrail`) +### CloudTrail -| Field | Description | Scheme | Required | -| --------- | --------------------------------------- | ------------ | -------- | -| `exclude` | Set events to be excluded from scraping | _\[\]string_ | | -| `max_age` | Set maximum age of events for scraping | _string_ | | +| Field | Description | Scheme | Required | +| --------- | --------------------------------------- | ---------- | -------- | +| `exclude` | Set events to be excluded from scraping | `[]string` | | +| `max_age` | Set maximum age of events for scraping | `string` | | -### Cost Reporting (`cost_reporting`) +### Cost Reporting | Field | Description | Scheme | Required | | ---------------- | -------------------------------------------------------------- | -------- | -------- | -| `s3_bucket_path` | Set path for S3 bucket to scrape published AWS billing reports | _string_ | | -| `table` | Specify table containing cost and usage data | _string_ | | -| `database` | Specify database containing cost and usage data | _string_ | | -| `region` | Specify region for S3 bucket | _string_ | | +| `s3_bucket_path` | Set path for S3 bucket to scrape published AWS billing reports | `string` | | +| `table` | Specify table containing cost and usage data | `string` | | +| `database` | Specify database containing cost and usage data | `string` | | +| `region` | Specify region for S3 bucket | `string` | | ### Supported Resources diff --git a/mission-control/docs/config-db/scrapers/azure-devops.md b/mission-control/docs/config-db/scrapers/azure-devops.md index 0452ce2f..a404916e 100644 --- a/mission-control/docs/config-db/scrapers/azure-devops.md +++ b/mission-control/docs/config-db/scrapers/azure-devops.md @@ -28,26 +28,26 @@ spec: | Field | Description | Scheme | Required | | ------------- | ---------------------------------------------------------------------------------- | -------------------------------------------- | -------- | -| `logLevel` | Specify the level of logging. | `string` | `false` | -| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | `false` | -| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | `false` | +| `logLevel` | Specify the level of logging. | `string` | | +| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | | +| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | | | `retention` | Settings for retaining changes, analysis and scraped items | [`Retention`](/config-db/concepts/retention) | | -| `azureDevops` | Specifies the list of Azure DevOps configurations to scrape. | [`[]AzureDevops`](#azuredevops-1) | `false` | +| `azureDevops` | Specifies the list of Azure DevOps configurations to scrape. | [`[]AzureDevops`](#azuredevops-1) | | ### AzureDevops | Field | Description | Scheme | Required | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | -------- | | `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | -| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` | -| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` | +| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | | +| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | | | `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | -| `transform` | Specify field to transform result | [`Transform`](../concepts/transform.md) | `false` | -| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | `false` | -| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` | -| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | +| `transform` | Specify field to transform result | [`Transform`](../concepts/transform.md) | | +| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | | +| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | | +| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | | +| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | | | `organization` | Specifies the name of the Azure DevOps organization to scrape | `string` | `true` | | `personalAccessToken` | Specifies the personal access token to authenticate with Azure DevOps | []_EnvVar_ | `true` | -| `projects` | Specifies the Azure DevOps projects to scrape | `[]string` | `false` | -| `pipelines` | Specifies the Azure DevOps pipelines to scrape | `[]string` | `false` | +| `projects` | Specifies the Azure DevOps projects to scrape | `[]string` | | +| `pipelines` | Specifies the Azure DevOps pipelines to scrape | `[]string` | | diff --git a/mission-control/docs/config-db/scrapers/file.md b/mission-control/docs/config-db/scrapers/file.md index 7679892b..5b858e8f 100644 --- a/mission-control/docs/config-db/scrapers/file.md +++ b/mission-control/docs/config-db/scrapers/file.md @@ -22,25 +22,26 @@ For more examples of configuration file, please check the [GitHub repo](https:// | Field | Description | Scheme | Required | | ----------- | ---------------------------------------------------------------------------------- | -------------------------------------------- | -------- | -| `logLevel` | Specify the level of logging. | `string` | `false` | -| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | `false` | -| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | `false` | +| `logLevel` | Specify the level of logging. | `string` | | +| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | | +| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | | | `retention` | Settings for retaining changes, analysis and scraped items | [`Retention`](/config-db/concepts/retention) | | -| `file` | Specifies the list of File configurations to scrape. | [`[]File`](#file-1) | `false` | +| `file` | Specifies the list of File configurations to scrape. | [`[]File`](#file-1) | | ### File | Field | Description | Scheme | Required | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | -------- | | `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | -| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` | -| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` | +| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | | +| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | | | `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | -| `transform` | Specify field to transform result | [`Transform`](../concepts/transform) | `false` | -| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | `false` | -| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` | -| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `url` | Specify URL e.g github repository containing the configuration(s) | `string` | `false` | +| `class` | A static value or JSONPath expression to use as the config class. _(Defaults to `type`)_ | `string` | | +| `transform` | Specify field to transform result | [`Transform`](../concepts/transform) | | +| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | | +| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | | +| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | | +| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | | +| `url` | Specify URL e.g github repository containing the configuration(s) | `string` | | | `paths` | Specify paths to configuration(s) for scraping | `[]string` | `true` | -| `ignore` | Set configurations to ignore | `[]string` | `false` | +| `ignore` | Set configurations to ignore | `[]string` | | diff --git a/mission-control/docs/config-db/scrapers/kubernetes-file.md b/mission-control/docs/config-db/scrapers/kubernetes-file.md index 4cd1ae53..a4bda063 100644 --- a/mission-control/docs/config-db/scrapers/kubernetes-file.md +++ b/mission-control/docs/config-db/scrapers/kubernetes-file.md @@ -23,28 +23,28 @@ spec: | Field | Description | Scheme | Required | | ---------------- | ---------------------------------------------------------------------------------- | -------------------------------------------- | -------- | -| `logLevel` | Specify the level of logging. | `string` | `false` | -| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | `false` | -| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | `false` | +| `logLevel` | Specify the level of logging. | `string` | | +| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | | +| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | | | `retention` | Settings for retaining changes, analysis and scraped items | [`Retention`](/config-db/concepts/retention) | | -| `kubernetesFile` | Specifies the list of Kubernetes File configurations to scrape. | [`[]KubernetesFile`](#kubernetesfile-1) | `false` | +| `kubernetesFile` | Specifies the list of Kubernetes File configurations to scrape. | [`[]KubernetesFile`](#kubernetesfile-1) | | ### KubernetesFile -| Field | Description | Scheme | Required | -| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | -------- | -| `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | -| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` | -| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` | -| `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | -| `transform` | Specify field to transform result | [`Transform`](../concepts/transform.md) | `false` | -| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | `false` | -| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` | -| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `selector` | Specify Kubernetes resource for configuration based on `namespace`, `kind`, `name` and more. | [`ResourceSelector`](./resourceselector) | yes | -| `container` | Set container name | `string` | | -| `files` | Specify path to file contained in Pod | `[]File` | | +| Field | Description | Scheme | Required | +| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | -------- | +| `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | +| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | | +| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | | +| `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | +| `transform` | Specify field to transform result | [`Transform`](../concepts/transform.md) | | +| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | | +| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | | +| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | | +| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | | +| `selector` | Specify Kubernetes resource for configuration based on `namespace`, `kind`, `name` and more. | [`ResourceSelector`](../../reference/resource_selector) | `true` | +| `container` | Set container name | `string` | | +| `files` | Specify path to file contained in Pod | `[]File` | | ### File @@ -52,13 +52,3 @@ spec: | -------- | --------------------------------- | ---------- | -------- | | `path` | Set path to file contained in Pod | `[]string` | | | `format` | Specify format of file | `string` | | - -### ResourceSelector - -| Field | Description | Scheme | Required | -| --------------- | ----------------------------------------------------------------------------------------- | -------- | -------- | -| `namespace` | Specify Namespace for Kubernetes resource | `string` | | -| `kind` | Specify Kind of Kubernetes resource | `string` | | -| `name` | Specify name of Kubernetes resource | `string` | | -| `labelSelector` | Specify labels attached to Kubernetes resource | `string` | | -| `fieldSelector` | Specify Kubernetes resource based on value of resource fields. e.g `status.Phase=Running` | `string` | | diff --git a/mission-control/docs/config-db/scrapers/kubernetes.md b/mission-control/docs/config-db/scrapers/kubernetes.md index e3d32793..848e0c5e 100644 --- a/mission-control/docs/config-db/scrapers/kubernetes.md +++ b/mission-control/docs/config-db/scrapers/kubernetes.md @@ -41,25 +41,25 @@ spec: | Field | Description | Scheme | Required | | ------------ | ---------------------------------------------------------------------------------- | -------------------------------------------- | -------- | -| `logLevel` | Specify the level of logging. | `string` | `false` | -| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | `false` | -| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | `false` | +| `logLevel` | Specify the level of logging. | `string` | | +| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | | +| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | | | `retention` | Settings for retaining changes, analysis and scraped items | [`Retention`](/config-db/concepts/retention) | | -| `kubernetes` | Specifies the list of Kubernetes configurations to scrape. | [`[]Kubernetes`](#kubernetes-1) | `false` | +| `kubernetes` | Specifies the list of Kubernetes configurations to scrape. | [`[]Kubernetes`](#kubernetes-1) | | ### Kubernetes | Field | Description | Scheme | Required | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | -------- | -| `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | -| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` | -| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` | -| `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | -| `transform` | Specify field to transform result | [`Transform`](../concepts/transform.md) | `false` | -| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | `false` | -| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` | -| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | +| `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | | +| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | | +| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | | +| `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | | +| `transform` | Specify field to transform result | [`Transform`](../concepts/transform.md) | | +| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | | +| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | | +| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | | +| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | | | `clusterName` | Specify cluster name | `string` | | | `namespace` | Specify namespace for scraping of Kubernetes resources | `string` | | | `useCache` | Specify boolean value to toggle fetching results from Kube-apiserver or fetch response from etcd | `bool` | | @@ -68,9 +68,9 @@ spec: | `selector` | Specify Kubernetes resource to scrape based on selector. e.g `matchLabels` | `string` | | | `fieldSelector` | Specify Kubernetes resource based on value of resource fields. e.g `status.Phase=Running` | `string` | | | `exclusions` | Specify Kubernetes resources to be excluded from scraping | `[]string` | | -| **`kubeconfig`** | Specify kubeconfig for access to your Kubernetes Cluster | []_EnvVar_ | yes | -| `event` | Specify configuration to handle Kubernetes events. | [`Event`](#sevent) | yes | -| `relationships` | Create relationships between kubernetes objects. | [`[]Relationship`](#kubernetes-relationship) | `false` | +| **`kubeconfig`** | Specify kubeconfig for access to your Kubernetes Cluster | []_EnvVar_ | | +| `event` | Specify configuration to handle Kubernetes events. | [`Event`](#sevent) | | +| `relationships` | Create relationships between kubernetes objects. | [`[]Relationship`](#kubernetes-relationship) | | ### Events @@ -80,15 +80,15 @@ In addition, you can also specify keywords used to identify the severity of the | Field | Description | Scheme | Required | | ------------------ | ------------------------------------------------------------------------------------------ | --------------------------------------- | -------- | -| `exclusions` | A list of keywords used to exclude event objects based on the reason | `[]string` | `false` | -| `severityKeywords` | Specify keywords used to identify the severity of the Kubernetes Event based on the reason | [`SeverityKeywords`](#severitykeywords) | `false` | +| `exclusions` | A list of keywords used to exclude event objects based on the reason | `[]string` | | +| `severityKeywords` | Specify keywords used to identify the severity of the Kubernetes Event based on the reason | [`SeverityKeywords`](#severitykeywords) | | ### SeverityKeywords | Field | Description | Scheme | Required | | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | -------- | -| `warn` | A list of keywords used to identify a warning severity from the reason. It could also be a match pattern: example "\*" to match all or "!badword" to exclude "badword" | `[]string` | `false` | -| `error` | Same as `warn` but used to map to error severity. | `[]string` | `false` | +| `warn` | A list of keywords used to identify a warning severity from the reason. It could also be a match pattern: example "\*" to match all or "!badword" to exclude "badword" | `[]string` | | +| `error` | Same as `warn` but used to map to error severity. | `[]string` | | ### Kubernetes Relationship diff --git a/mission-control/docs/config-db/scrapers/sql.md b/mission-control/docs/config-db/scrapers/sql.md index 12b1161f..88aabe03 100644 --- a/mission-control/docs/config-db/scrapers/sql.md +++ b/mission-control/docs/config-db/scrapers/sql.md @@ -75,25 +75,25 @@ spec: | Field | Description | Scheme | Required | | ----------- | ---------------------------------------------------------------------------------- | -------------------------------------------- | -------- | -| `logLevel` | Specify the level of logging. | `string` | `false` | -| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | `false` | -| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | `false` | +| `logLevel` | Specify the level of logging. | `string` | | +| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | | +| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | | | `retention` | Settings for retaining changes, analysis and scraped items | [`Retention`](/config-db/concepts/retention) | | -| `sql` | Specifies the list of SQL configurations to scrape. | [`[]SQL`](#sql-1) | `false` | +| `sql` | Specifies the list of SQL configurations to scrape. | [`[]SQL`](#sql-1) | | ## SQL | Field | Description | Scheme | Required | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | -------- | | `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | -| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` | -| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` | +| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | | +| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | | | `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | -| `transform` | Specify field to transform result | [`Transform`](../concepts/transform) | `false` | -| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | `false` | -| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` | -| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | -| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` | +| `transform` | Specify field to transform result | [`Transform`](../concepts/transform) | | +| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | | +| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | | +| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | | +| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | | | - | Specify connection details to the database | [Connection](#connection) | | -| `driver` | Specify the name of the driver to use for connecting to the database | `string` | `false` | +| `driver` | Specify the name of the driver to use for connecting to the database | `string` | | | `query` | Specify the SQL query to execute | `string` | `true` | diff --git a/mission-control/docs/config-db/scrapers/trivy.md b/mission-control/docs/config-db/scrapers/trivy.md index fb9ff922..3a210bae 100644 --- a/mission-control/docs/config-db/scrapers/trivy.md +++ b/mission-control/docs/config-db/scrapers/trivy.md @@ -26,30 +26,30 @@ _Fig: A detailed view of the analysis on the postgres container_ | Field | Description | Scheme | Required | | ----------- | ---------------------------------------------------------------------------------- | -------------------------------------------- | -------- | -| `logLevel` | Specify the level of logging. | `string` | `false` | -| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | `false` | -| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | `false` | +| `logLevel` | Specify the level of logging. | `string` | | +| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | | +| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | | | `retention` | Settings for retaining changes, analysis and scraped items | [`Retention`](/config-db/concepts/retention) | | -| `trivy` | Specifies the list of Trivy configurations to scrape. | [`[]Trivy`](#trivy-1) | `false` | +| `trivy` | Specifies the list of Trivy configurations to scrape. | [`[]Trivy`](#trivy-1) | | ### Trivy | Field | Description | Scheme | Required | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | -------- | | `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` | -| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` | -| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` | +| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | | +| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | | | `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` | -| `transform` | Specify field to transform result. | [`Transform`](../concepts/transform.md) | `false` | -| `format` | Format of config item, defaults to JSON, available options are JSON. | `string` | `false` | -| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` | -| `version` | Specify the Trivy version to use. (default 0.40.0) | `string` | `false` | -| `compliance` | compliance report to generate(k8s-nsa, k8s-cis, k8s-pss-baseline, k8s-pss-restricted). | `string` | `false` | -| `ignoredLicenses` | specify a list of license to ignore. | `[]string` | `false` | -| `ignoreUnfixed` | display only fixed vulnerabilities. | `bool` | `false` | -| `licenseFull` | eagerly look for licenses in source code headers and license files. | `bool` | `false` | -| `severity` | severities of security issues to be displayed (comma separated)_(default "UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL")_. | `string` | `false` | -| `vulnType` | comma-separated list of vulnerability types (comma separated)_(default "os,library")_. | `string` | `false` | +| `transform` | Specify field to transform result. | [`Transform`](../concepts/transform.md) | | +| `format` | Format of config item, defaults to JSON, available options are JSON. | `string` | | +| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | | +| `version` | Specify the Trivy version to use. (default 0.40.0) | `string` | | +| `compliance` | compliance report to generate(k8s-nsa, k8s-cis, k8s-pss-baseline, k8s-pss-restricted). | `string` | | +| `ignoredLicenses` | specify a list of license to ignore. | `[]string` | | +| `ignoreUnfixed` | display only fixed vulnerabilities. | `bool` | | +| `licenseFull` | eagerly look for licenses in source code headers and license files. | `bool` | | +| `severity` | severities of security issues to be displayed (comma separated)_(default "UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL")_. | `string` | | +| `vulnType` | comma-separated list of vulnerability types (comma separated)_(default "os,library")_. | `string` | | | `kubernetes` | Specify the trivy option to scan kubernetes objects. | [`K8sOptions`](#k8soptions) | `true` | ### K8sOptions @@ -58,7 +58,7 @@ Trivy Options consist of selected few flags that are passed on to trivy. | Field | Description | Scheme | Required | | ------------ | -------------------------------------------------------------------------------------- | ---------- | -------- | -| `components` | Specify which components to scan*(default workload, infra).* | `[]string` | `false` | -| `kubeconfig` | Specify the kubeconfig file path to use as a static value or as a JSONPath expression. | `string` | `false` | +| `components` | Specify which components to scan*(default workload, infra).* | `[]string` | | +| `kubeconfig` | Specify the kubeconfig file path to use as a static value or as a JSONPath expression. | `string` | | | `namespace` | Specify a namespace to scan. | `string` | `true` | -| `context` | Specify a context to scan. | `string` | `false` | +| `context` | Specify a context to scan. | `string` | |