Release 5.13.0 (#95)

* ED (#59)

* update gunicorn version

* update req

* Update schedules

* remove metrics bucket and use thread local storage instead of envs

* Remove old ed rulesets completely

* Remove ed rulesets from cli

* update sync: store rulesets when synced

* do not cache rulesets

* Resolve rulesets from local minio

* Update job executor

* Warnings

* Fix inconsistent sync status

* Update changelog, version and modular-sdk version

* FIx test

* Update helm charts

* Update changelog

* Update changelog

* Add some minor fixes

* Update sre-init.sh backup a bit (#61)

* Send additional metadata with reports (#63)

* Update version of c7n and some other requirements' versions

* TTL for snapshots

* Add lifecycle

* Add some minor improvements

* [WIP] resources

* update resource report handler to new resources representation

* Update all reports to new resources representation

* Fix resources iterator

* Add .env.example

* Minor changes

* Update previous metrics to new reports model

* Resolve regions inside handler

* FIx dojo severity

* Patch that tags files as Type: DataSnapchot (#64)

* Add patch for tagging snapshots

* Update logs and docstring
Add readme

* Fix env variables assertion

* Update readme

---------

Co-authored-by: Dmytro Afanasiev <[email protected]>

* Add errors to shard parts

* Add report visitor

* update

* Update active_regions to activated_regions

* Add minor fixes

* [WIP] implement encode_into

* Update encode_into

* Dojo reports batches (#65)

* Add dojo batching
Fix url in swagger handler
Fix chronicle logs

* Fix import_scan return

* Reuse batches_with_critic for chronicle

* Support second type of report

---------

Co-authored-by: Dmytro Afanasiev <[email protected]>

* Fix typing

* Delete expired metrics (#66)

* Add dojo batching
Fix url in swagger handler
Fix chronicle logs

* Fix import_scan return

* Add delete_expired_metrics task
Add cutomer_end_index to ReportMeetrics
Add query_all_by_customer function to ReportMetricService
Move env variable for dojo client to CASSEnv

* romove logs in dojo import scan

* fix typing in dojo import scan

* Remove duplicated function

* Adjust limit slightly

* Remove excessive decoding of response

* Make expiration not required, add it to helm

* Replace float with int

* Add remover task to celery beat_schedule, use batch delete

---------

Co-authored-by: Dmytro Afanasiev <[email protected]>

* Update changelog

---------

Co-authored-by: Ihor Tkhoruk <[email protected]>

* update uv

* Improve report fields (#67)

* Remove assert from metrics

* Update versions sorting

* Improve report fields resolving, resolve name from tags

* Fix bug with empty resources report

* Fix misspelling

* Fix metrics

* Fix metrics

* Fix finops

* Fix dojo describ

* Fix bugs

* Fix issues with update

* add thread to patch

* Tenant validation during integrations activation (#70)

* added tenant validation during integrations activation

* Use get instead of query if number of names is small

* Add ticket number

---------

Co-authored-by: = <=>
Co-authored-by: Dmytro Afanasiev <[email protected]>

* Fix error when describe regions permission is not available (#68)

* Fix error when describe regions permission is not available

* Make job failed if no policies are succeeded

* Update changelog

* Add editorconfig

* Update versions (#72)

* Update modular-sdk version

* Updated version of custodian, vauld, and minio

* Update some req

* Update changelog

---------

Co-authored-by: = <=>

* Normalized versions (#71)

* added normalized version handling to rule, ruleset, and rule_source

* added patch for normalizing versions
remove normalization from rule_source

* added test for rule deduplication

* Update ruleset create

* Update changelog

---------

Co-authored-by: = <=>
Co-authored-by: Dmytro Afanasiev <[email protected]>

* Check artifacts digest before downloading (#73)

* Check artifacts digest before downloading

* Format sre-init

* Add self update for update command

* Update sre-inti

* Fix some bugs

* Update modularsdk

* Add fixes

* Add tenants validation to update tenant

* [cf-template] Update CF template due to AWS MP requirements (#75)

* Separate application type for DEFECT_DOJO and RABBIT_MQ applications (#77)

* changed DEFECT_DOJO and RABBITMQ types to CUSTODIAN_*

* updated changelog

* added more info in policies failed warning

* update dependencies for modular_sdk

* update modular-sdk

* Add fixes

---------

Co-authored-by: = <=>
Co-authored-by: Dmytro Afanasiev <[email protected]>

* fix azure permission denied

* Add additional guard to to_normalized_version(

* Fix bug with dojo

* Update modular-sdk

* Add generic related filter (#80)

* Add generic related filter

* Move aws res

* Move azure filters

* Add gcp resources and filters

* Add fields to operational rules report (#82)

* added resource_type, service, and severity to operational rules report

* remove unnessesary slot

* add comma to make tuple

* move additional fields creation to metrics collector

* fix averagestatisticsitem
fix test data

* return _operational_rules_custom method to previous shape

* Add remediation to OP resources, deprecations, finops, attacks

* fix resource_type
add new fields in test data

* fix test data for maestro_reports

* Format

* Minors

* [WIP] change operational overview

* Finish new overview

* Update versions

* Fix bug with rules

* Fix bugs

* Fix bug with duplicates again

* Remove logging from plugin

* FIx

* Update changelog

---------

Co-authored-by: Dmytro Afanasiev <[email protected]>

* Operational resources new schema (#84)

* added resource_type, service, and severity to operational rules report

* remove unnessesary slot

* add comma to make tuple

* move additional fields creation to metrics collector

* fix averagestatisticsitem
fix test data

* return _operational_rules_custom method to previous shape

* Add remediation to OP resources, deprecations, finops, attacks

* fix resource_type
add new fields in test data

* fix test data for maestro_reports

* Format

* Minors

* [WIP] change operational overview

* Finish new overview

* Update versions

* Fix bug with rules

* Fix bugs

* Fix bug with duplicates again

* changed schema of operational resources report

* final fixes

* added sre:date field
changed violations_data to violations

* fix freeze issue

* Minors

---------

Co-authored-by: iothor <[email protected]>

* Fix metrics

* Resource Managment using Cloud Custodian (#81)

* added custodian resource collector
added model for resource and compound index

* added unit tests
changed method to static
remove region resource type from gcp policies

* added resources endpoint

* fix pagination bug for resources
fix resource model bug

* move resource saving to _process_policy
fix create Resource bug

* fix ci/cd test fail

* fix ci/cd test fail 2

* second iteration

* added endpoint GET /resources/arn/{arn}
small fixes

* fix unit test

* fixed arn issue
small improvements

* remove useless method

* some notes

* added script to test collection locally

* added abstract class for resource collectors

* added AzureGraph resource collector

* small fixes

* delete test script

* move resource collector alternatives to it's own file

* [WIP] update minors

* Update collector

* Fix test

---------

Co-authored-by: Dmytro Afanasiev <[email protected]>

* added handling of deprecated rules (#85)

* Upddate changelog

* set limits (#87)

* Change vault chart a bit

* Update resource types

* Gunicorn workers

* update c7n

* lazy init mogno

* update limits

* Update helm charts

* Fix bug with operational rules severity

* Add compliance to overview

* Maybe fix resources collector

* Remove checks (#89)

* Report fixes (#90)

* Fix a bug with small number of google executed policies

* Update vault version for rule engine chart

* Fix inconsistent shard parts

* Private minio url

* Fix tests and downgrade version to 5.12.1 (hotfix)

* Add id to rules report

* Fix cli set excluded rules

* Update helm charts

* Updated HELM chart values for modular-related environments (#92)

* Updated HELM chart values for modular-related environments

* Fix typos

* Rename DB and Collections (#88)

* add patch to change database name and collection names
changed collection names in models

* move patch to 5.13

* Renamed CAASEnv, CustodianEndpoint, and CustodianException
Updated changelog

* rename lambdas folders

* added crud endpoints for resources exception and model with service for storing them

* fix resources tests

* added exception data to operational reports
added resources count to overview report

* Rename all Custodian inside cli

* update some more

* added total scanned to operational rules reports
added tests for total count
small fixes

* rename some more

* Update helm charts

* Fixed bug with lm expiration date (#93)

* fixed bug with lm expiration date

* added documentation for resources endpoints
fixed bug with iterables in query parameters
small fixes

* fix unit test

* Update charts

* Fix aws lambda signing config filter

---------

Co-authored-by: Dmytro Afanasiev <[email protected]>

* Fix patch

---------

Co-authored-by: Ihor Tkhoruk <[email protected]>
Co-authored-by: Bohdan Syniuk <[email protected]>
Co-authored-by: Ihor Tkhoruk <[email protected]>

Author: 3 people

CHANGELOG.md

@@ -5,6 +5,13 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [5.13.0] - 2025-08-05
+- added endpoints for interacting with resource exceptions: `GET /resources/exceptions`, `GET /resources/exceptions/{id}`, `POST /resources/exceptions`, `PUT /resources/exceptions/{id}`, `DELETE /resources/exceptions/{id}`.
+- changed operational overview, resources, deprecations, finops, and attacks reports to add exceptions information.
+- changed operational overview report to include total resources scanned.
+- renamed database, collections and other occurrences from `custodian_as_a_service` to `syndicate_rule_engine`.
+- update modular-related environment variables in the HELM chart
+
 ## [5.12.2] - 2025-08-19 (hotfix)
 - moved cron configurations outside
 - disabled scan resources job by default

Makefile

@@ -16,7 +16,7 @@ EXECUTOR_IMAGE_TAG := latest
 SERVER_IMAGE_NAME := public.ecr.aws/x4s4z8e1/syndicate/rule-engine
 SERVER_IMAGE_TAG ?= $(shell PYTHONPATH=./src python -B -c "from src.helpers.__version__ import __version__; print(__version__)")
 
-DOCKERFILE_NAME := Dockerfile
+DOCKERFILE_NAME := Dockerfile-opensource-uv
 ADDITIONAL_BUILD_PARAMS ?=
 
 SYNDICATE_EXECUTABLE_PATH ?= $(shell which syndicate)
@@ -116,7 +116,6 @@ syndicate-update-step-functions: check-syndicate
 	SDCT_CONF=$(SYNDICATE_CONFIG_PATH) $(SYNDICATE_EXECUTABLE_PATH) deploy --deploy_only_types step_functions --replace_output --bundle_name $(SYNDICATE_BUNDLE_NAME)
 
 
-# images with fork which is default for now. Use src/onprem/Dockerfile-opensource for c7n from open source
 #make image-arm64
 #make image-amd64
 #make push-arm64

cli/CHANGELOG.md

@@ -4,6 +4,9 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [5.8.0] - 2025-08-15
+- renamed `Custodian` to `Syndicate Rule Engine (SRE)`
+
 ## [5.7.1] - 2025-08-13
 - fixed `--empty` flag for `sre tenant set_excluded_rules` & `sre customer set_excluded_rules` commands