From e805ed594a9e7ea8bec8157e2b0501ff6fb84f2a Mon Sep 17 00:00:00 2001 From: Joseph Beck Date: Fri, 15 Nov 2024 16:29:02 -0800 Subject: [PATCH 01/40] Add WIP HIP for a proposal to add a sequencing featureset to Helm installations and upgrades. Signed-off-by: Joseph Beck --- hips/hip-00xx-sequencing.md | 60 +++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100644 hips/hip-00xx-sequencing.md diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md new file mode 100644 index 00000000..4c0ef9a9 --- /dev/null +++ b/hips/hip-00xx-sequencing.md @@ -0,0 +1,60 @@ +--- +hip: 9999 +title: "Better Support for Resource Creation Sequencing" +authors: [ "Joe Beck " ] +created: "2024-11-15" +type: "feature" +status: "draft" +--- + +## Abstract + +When an applicaiton requires dependencies to be spun up first before the application can start. Writing this sequencing into a helm chart today is very tedious using hooks and weights to sequence the resources, or some built-in application safety checks at runtime to achieve similar sequencing. I'd like to propose helm has a better sequencing mechanism that allows chart developers to specify dependency chains between resources, that doesn't leverage the current hooks and weights system. + +I'm not sure what exactly the best implementation of this would be, but my first naive approach would be a "depends on" system where a chart developer can specify something such as "this deployment `foo` depends on job `bar` completing before it is created". Then during installation/upgrades, helm would resolve the entire dependency chain and add resources to the cluster in the chain order. + +## Motivation + +Today, to accomplish any kind of sequencing you have two options. The first is leveraging hooks and weights, the second is building the sequencing into the application itself. + +Using hooks and weights can be a tedious process, as for example, running one pre-install job at weight 0, may require other resources such as config maps, secrets, and service accounts, to be created at weight "-1" to ensure the job can run. In a lot of cases, those same resources that exist at weight "-1" will need to be duplicated and exist outside of the context of the hook, which adds bloat to the chart installation. Additionally, for mor complex use cases, the weight system on hooks can become very tedious to maintain and get right on the chart developer side. + +Building the sequencing into the application may reduce complexity in the chart itself, but it significantly increases complexity of the application when safety checks or the equivalent of `while(database not ready) { thread.sleep() }` need to be added just to satisfy the helm deployment use case, when that may be one of many ways to deploy an application. + +By providing a mechanism to the chart developer that allows them to explicitly determine when resources are deployed to a cluster, we work around both of these issues, by reducing applicaiton complexity, and reducing the need to leverage hooks. + +## Rationale + +TBD upon deciding on a design. + +## Specification + +TBD upon deciding on a design. + +## Backwards compatibility + +TBD upon deciding on a design. + +## Security implications + +TBD upon deciding on a design. + +## How to teach this + +TBD upon deciding on a design. + +## Reference implementation + +N/A + +## Rejected ideas + +N/A + +## Open issues + +- The exact design of how Helm 4 should implement such a sequencing mechanism still needs to be determined. + +## References + +N/A \ No newline at end of file From 3d754ea509a81c2931c937f5389a1dbd37492023 Mon Sep 17 00:00:00 2001 From: Joe Beck Date: Wed, 27 Nov 2024 10:55:54 -0800 Subject: [PATCH 02/40] cleanup sequencing hip and flesh out implementation proposals Signed-off-by: Joe Beck --- hips/hip-00xx-sequencing.md | 109 +++++++++++++++++++++++++++++++----- 1 file changed, 95 insertions(+), 14 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 4c0ef9a9..5c90d3c9 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -9,35 +9,115 @@ status: "draft" ## Abstract -When an applicaiton requires dependencies to be spun up first before the application can start. Writing this sequencing into a helm chart today is very tedious using hooks and weights to sequence the resources, or some built-in application safety checks at runtime to achieve similar sequencing. I'd like to propose helm has a better sequencing mechanism that allows chart developers to specify dependency chains between resources, that doesn't leverage the current hooks and weights system. - -I'm not sure what exactly the best implementation of this would be, but my first naive approach would be a "depends on" system where a chart developer can specify something such as "this deployment `foo` depends on job `bar` completing before it is created". Then during installation/upgrades, helm would resolve the entire dependency chain and add resources to the cluster in the chain order. +This HIP is to propose a new featureset in Helm 4 to provide Helm developers a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. ## Motivation -Today, to accomplish any kind of sequencing you have two options. The first is leveraging hooks and weights, the second is building the sequencing into the application itself. - -Using hooks and weights can be a tedious process, as for example, running one pre-install job at weight 0, may require other resources such as config maps, secrets, and service accounts, to be created at weight "-1" to ensure the job can run. In a lot of cases, those same resources that exist at weight "-1" will need to be duplicated and exist outside of the context of the hook, which adds bloat to the chart installation. Additionally, for mor complex use cases, the weight system on hooks can become very tedious to maintain and get right on the chart developer side. +Today, to accomplish resource sequencing you have two options. The first is leveraging helm hooks, the second is building required sequencing into the application (via startup code or init containers). The existing hooks and weights can be a tedious to build and maintain for Helm developers, and built-in app sequencing can unecessarily increase complexity of a Helm application that needs to be maintained by Application Developers. Helm as a package manager should be capable of enabling developers to properly sequence how their applications are deployed, and by providing such a mechanism to developers, this will significantly improve the Helm and Application developer experiences. -Building the sequencing into the application may reduce complexity in the chart itself, but it significantly increases complexity of the application when safety checks or the equivalent of `while(database not ready) { thread.sleep() }` need to be added just to satisfy the helm deployment use case, when that may be one of many ways to deploy an application. - -By providing a mechanism to the chart developer that allows them to explicitly determine when resources are deployed to a cluster, we work around both of these issues, by reducing applicaiton complexity, and reducing the need to leverage hooks. +Additionally, Helm currently doesn't provide a way to sequence when chart dependencies are deployed, and this featureset would ideally address this. ## Rationale -TBD upon deciding on a design. +### Proposal 1: Annotations +This design was chosen due to simplicity and clarity of how it works for both the Helm developer and Application Operator + +### Proposal 2: Weighted Resouces (without hooks) +This design was proposed as an alternative that functions very similarly to hooks today, to provide the same sequencing functionality outside of the scope of hooks. ## Specification -TBD upon deciding on a design. +### Proposal 1: Annotations +At a high level, allow Chart Developers to assign annotations to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. Three new annotations would be added to enable this desribed next. + +- `helm.sh/layer`: Declare a layer that a given resource belongs to. Any numebr of resources can belong to a layer. +- `helm.sh/dependson/layer`: Declare a layer that must exist and in a ready state before this resource can be deployed. +- `helm.sh/dependson/chart`: Declare a chart dependency that must exist and in a ready state before this resource can be deployed. + +A layer would not be considered "ready" and the next layer installed until: +- all jobs in the layer are DONE +- all other resources in the layer are READY + +#### Example: +``` +# resource 1 +metadata: + name: foo + annotations: + helm.sh/layer: layer1 +--- +# resource 2 +metadata: + name: bar + annotations: + helm.sh/layer: layer2 + helm.sh/dependson/layer: layer1 +--- +# resource 3 +metadata: + name: fizz + annotations: + helm.sh/layer: layer3 + helm.sh/dependson/layer: layer1 + helm.sh/dependson/chart: someChartDependency +``` +In this example, Helm would be responsible for resolving the annotations on these three resources and deploy all resources in the following order: + +``` +layer1: [foo] + || + || [someChartDependency] + || || + \/ || +layer2: [bar] || + \\ // + \/ \/ +layer3: [fizz] +``` + +### Proposal 2: Weighted Resouces (without hooks) + +Expand on the the weight capabilities of hooks, but allow them to be defined outside of the context of hooks for the chart. Add a new annotation `helm.sh/weight` that helm then uses to order when resources are deployed based on the same weighting rules that hooks use today. + +#### Example: +``` +# resource 1 +metadata: + name: foo + annotations: + helm.sh/weight: "-2" +--- +# resource 2 +metadata: + name: bar + annotations: + helm.sh/weight: "-1" +--- +# resource 3 +metadata: + name: fizz + annotations: + helm.sh/weight: 0 +``` +In this example, Helm would be responsible for resolving the annotations on these three resources and deploy all resources in the following order: + +``` +-2: [foo] + || + \/ +-1: [bar] + || + \/ +0: [fizz] +``` ## Backwards compatibility -TBD upon deciding on a design. +Helm will continue to deploy all resources and dependencies at once, as adding the above defined annotations is when the installation behavior would change. ## Security implications -TBD upon deciding on a design. +None. ## How to teach this @@ -53,7 +133,8 @@ N/A ## Open issues -- The exact design of how Helm 4 should implement such a sequencing mechanism still needs to be determined. +- Choose between Proposal 1 and Proposal 2 +- Should this featureset take into account allowing Helm Developers to declare custom "readiness" definitions for a given layer, besides the default? ## References From 78075e28daef0d50209bdf56868a7622cd30f7b5 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Thu, 28 Nov 2024 22:58:59 +0000 Subject: [PATCH 03/40] Additional information - Seuencing logic - Readiness logic Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 72 ++++++++++++++++++++++++++++++------- 1 file changed, 60 insertions(+), 12 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 5c90d3c9..66eb423b 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -1,7 +1,7 @@ --- hip: 9999 title: "Better Support for Resource Creation Sequencing" -authors: [ "Joe Beck " ] +authors: [ "Joe Beck ", "Evans Mungai " ] created: "2024-11-15" type: "feature" status: "draft" @@ -20,7 +20,7 @@ Additionally, Helm currently doesn't provide a way to sequence when chart depend ## Rationale ### Proposal 1: Annotations -This design was chosen due to simplicity and clarity of how it works for both the Helm developer and Application Operator +This design was chosen due to simplicity and clarity of how it works for both the Helm developer and Application Operator ### Proposal 2: Weighted Resouces (without hooks) This design was proposed as an alternative that functions very similarly to hooks today, to provide the same sequencing functionality outside of the scope of hooks. @@ -28,18 +28,18 @@ This design was proposed as an alternative that functions very similarly to hook ## Specification ### Proposal 1: Annotations -At a high level, allow Chart Developers to assign annotations to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. Three new annotations would be added to enable this desribed next. +At a high level, allow Chart Developers to assign annotations to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. Three new annotations would be added to enable this described next. - `helm.sh/layer`: Declare a layer that a given resource belongs to. Any numebr of resources can belong to a layer. -- `helm.sh/dependson/layer`: Declare a layer that must exist and in a ready state before this resource can be deployed. -- `helm.sh/dependson/chart`: Declare a chart dependency that must exist and in a ready state before this resource can be deployed. +- `helm.sh/depends-on/layer`: Declare a layer that must exist and in a ready state before this resource can be deployed. +- `helm.sh/depends-on/chart`: Declare a chart dependency that must exist and in a ready state before this resource can be deployed. For the chart to be declared ready, all of its resources, with their sequencing order taken into consideration, would need to be deployed and declared ready. A layer would not be considered "ready" and the next layer installed until: - all jobs in the layer are DONE - all other resources in the layer are READY #### Example: -``` +```yaml # resource 1 metadata: name: foo @@ -64,8 +64,8 @@ metadata: In this example, Helm would be responsible for resolving the annotations on these three resources and deploy all resources in the following order: ``` -layer1: [foo] - || +layer1: [foo] + || || [someChartDependency] || || \/ || @@ -75,12 +75,14 @@ layer2: [bar] || layer3: [fizz] ``` +This approach is prone to circular dependencies. During the templating phase, Helm will have logic to detect, and report any circular dependencies found in the chart templates. + ### Proposal 2: Weighted Resouces (without hooks) Expand on the the weight capabilities of hooks, but allow them to be defined outside of the context of hooks for the chart. Add a new annotation `helm.sh/weight` that helm then uses to order when resources are deployed based on the same weighting rules that hooks use today. #### Example: -``` +```yaml # resource 1 metadata: name: foo @@ -102,8 +104,8 @@ metadata: In this example, Helm would be responsible for resolving the annotations on these three resources and deploy all resources in the following order: ``` --2: [foo] - || +-2: [foo] + || \/ -1: [bar] || @@ -111,6 +113,52 @@ In this example, Helm would be responsible for resolving the annotations on thes 0: [fizz] ``` +### Readiness + +In order to enforce sequencing, new `helm.sh/resource-ready` annotation would be used to determine when a resource is ready, allowing helm to proceed deploying the next group of resources, or failing a deployment. Helm would query, using jsonpath syntax, status fields of the annotated resource. Some native kubernetes resources that have stable APIs e.g `v1` resources such as `Pod` and `Deployment`, would have default queries which can be overriden. In Helm cannot determine how to check a resource's readiness when it should, it will do nothing and log this. + +#### Example +```yaml +kind: Job +metadata: + name: barz + annotations: + helm.sh/resource-ready: "status.succeeded==1" +status: + succeeded: 1 +``` + +#### Sequencing order + +Resources with sequencing annotations would be deployed first followed by resources without. If a resource has a `helm.sh/depends-on/chart` annotation, all resources of the referred subchart would be deployed and checked for readiness before deploying the dependant resource. Only resources that Helm can determine readiness for will be checked. Chart authors would need to annotate their chart resources accordingly. + +Helm would scope each subchart layer annotation names using a delimiter such as `.` e.g `mysubchart.mylayer` to avoid any name collisions. This is an internal implementation detail rather than feature chart authors or operators would need to know. + +`helm template` would pring all the resources in the order they would be deployed. The groups would be delimited using a `## Layer: ` comment indicate the beginning of each layer + +```yaml +## Layer: layer1 +# resource 1 +metadata: + name: foo + annotations: + helm.sh/weight: "-2" +--- +# resource 2 +metadata: + name: bar + annotations: + helm.sh/weight: "-1" +--- +## Layer: layer2 +# resource 3 +metadata: + name: fizz + annotations: + helm.sh/weight: 0 + +``` + ## Backwards compatibility Helm will continue to deploy all resources and dependencies at once, as adding the above defined annotations is when the installation behavior would change. @@ -138,4 +186,4 @@ N/A ## References -N/A \ No newline at end of file +N/A From 163c6bf91994e35c2fbdc446ed3346b59fb514e2 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Fri, 29 Nov 2024 11:19:21 +0000 Subject: [PATCH 04/40] Rename Annotations and some minor improvements Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 66eb423b..387a80a9 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -19,7 +19,7 @@ Additionally, Helm currently doesn't provide a way to sequence when chart depend ## Rationale -### Proposal 1: Annotations +### Proposal 1: Named Dependencies This design was chosen due to simplicity and clarity of how it works for both the Helm developer and Application Operator ### Proposal 2: Weighted Resouces (without hooks) @@ -27,16 +27,14 @@ This design was proposed as an alternative that functions very similarly to hook ## Specification -### Proposal 1: Annotations -At a high level, allow Chart Developers to assign annotations to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. Three new annotations would be added to enable this described next. +### Proposal 1: Named Dependencies +At a high level, allow Chart Developers to assign named dependency annotations to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. Three new annotations would be added to enable this described next. - `helm.sh/layer`: Declare a layer that a given resource belongs to. Any numebr of resources can belong to a layer. - `helm.sh/depends-on/layer`: Declare a layer that must exist and in a ready state before this resource can be deployed. - `helm.sh/depends-on/chart`: Declare a chart dependency that must exist and in a ready state before this resource can be deployed. For the chart to be declared ready, all of its resources, with their sequencing order taken into consideration, would need to be deployed and declared ready. -A layer would not be considered "ready" and the next layer installed until: -- all jobs in the layer are DONE -- all other resources in the layer are READY +A layer would not be considered "ready" and the next layer deployed until all resources in that layer are considered "ready". Readiness is described in a later section. #### Example: ```yaml @@ -51,15 +49,15 @@ metadata: name: bar annotations: helm.sh/layer: layer2 - helm.sh/dependson/layer: layer1 + helm.sh/depends-on/layer: layer1 --- # resource 3 metadata: name: fizz annotations: helm.sh/layer: layer3 - helm.sh/dependson/layer: layer1 - helm.sh/dependson/chart: someChartDependency + helm.sh/depends-on/layer: layer1 + helm.sh/depends-on/chart: someChartDependency ``` In this example, Helm would be responsible for resolving the annotations on these three resources and deploy all resources in the following order: @@ -132,9 +130,9 @@ status: Resources with sequencing annotations would be deployed first followed by resources without. If a resource has a `helm.sh/depends-on/chart` annotation, all resources of the referred subchart would be deployed and checked for readiness before deploying the dependant resource. Only resources that Helm can determine readiness for will be checked. Chart authors would need to annotate their chart resources accordingly. -Helm would scope each subchart layer annotation names using a delimiter such as `.` e.g `mysubchart.mylayer` to avoid any name collisions. This is an internal implementation detail rather than feature chart authors or operators would need to know. +Helm would scope each subchart layer annotation names using a delimiter such as `.` e.g `someChartDependency.mylayer` to avoid any name collisions. This is an internal implementation detail rather than feature chart authors or operators would need to know. -`helm template` would pring all the resources in the order they would be deployed. The groups would be delimited using a `## Layer: ` comment indicate the beginning of each layer +`helm template` would print all resources in the order they would be deployed. Groups of resources in a layer would be delimited using a `## Layer: ` comment indicate the beginning of each layer ```yaml ## Layer: layer1 From a54be4d9cdb7cd30d86f5b1e92fb97c1f5a22e0a Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Mon, 2 Dec 2024 12:12:48 +0000 Subject: [PATCH 05/40] Add prior open issues to HIP Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 387a80a9..7d4063f3 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -182,6 +182,11 @@ N/A - Choose between Proposal 1 and Proposal 2 - Should this featureset take into account allowing Helm Developers to declare custom "readiness" definitions for a given layer, besides the default? -## References +## Prior raised issues + +- https://github.com/helm/helm/pull/12541 +- https://github.com/helm/helm/pull/9534 +- https://github.com/helm/helm/issues/8439 +- https://github.com/helm/community/pull/230 N/A From 9fe061a033e712df9465ffefca49c78ed2613ebf Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Mon, 2 Dec 2024 12:39:35 +0000 Subject: [PATCH 06/40] A few more changes Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 7d4063f3..abfe5fcb 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -9,11 +9,11 @@ status: "draft" ## Abstract -This HIP is to propose a new featureset in Helm 4 to provide Helm developers a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. +This HIP is to propose a new featureset in Helm 4 to provide Application developers, who create heam charts for their applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. ## Motivation -Today, to accomplish resource sequencing you have two options. The first is leveraging helm hooks, the second is building required sequencing into the application (via startup code or init containers). The existing hooks and weights can be a tedious to build and maintain for Helm developers, and built-in app sequencing can unecessarily increase complexity of a Helm application that needs to be maintained by Application Developers. Helm as a package manager should be capable of enabling developers to properly sequence how their applications are deployed, and by providing such a mechanism to developers, this will significantly improve the Helm and Application developer experiences. +Today, to accomplish resource sequencing you have two options. The first is leveraging helm hooks, the second is building required sequencing into the application (via startup code or init containers). The existing hooks and weights can be tedious to build and maintain for Application developers, and built-in app sequencing can unecessarily increase complexity of a Helm application that needs to be maintained by Application Developers. Helm as a package manager should be capable of enabling developers to properly sequence how their applications are deployed, and by providing such a mechanism to developers, this will significantly improve the Application developer's experience. Additionally, Helm currently doesn't provide a way to sequence when chart dependencies are deployed, and this featureset would ideally address this. @@ -128,9 +128,9 @@ status: #### Sequencing order -Resources with sequencing annotations would be deployed first followed by resources without. If a resource has a `helm.sh/depends-on/chart` annotation, all resources of the referred subchart would be deployed and checked for readiness before deploying the dependant resource. Only resources that Helm can determine readiness for will be checked. Chart authors would need to annotate their chart resources accordingly. +Resources with sequencing annotations would be deployed first followed by resources without. If a resource has a `helm.sh/depends-on/chart` annotation, all resources of the referred subchart would be deployed and checked for readiness before deploying the dependant resource. Only resources that Helm can determine their readiness for will be checked. Chart authors would need to annotate their chart resources accordingly. Helm will use default readiness checks for subcharts that do have their templates annotated. -Helm would scope each subchart layer annotation names using a delimiter such as `.` e.g `someChartDependency.mylayer` to avoid any name collisions. This is an internal implementation detail rather than feature chart authors or operators would need to know. +Helm would scope each subchart layer annotation names using a delimiter e.g `someChartDependency#mylayer` to avoid any name collisions. This is an internal implementation detail rather than feature chart authors or operators would need to know. `helm template` would print all resources in the order they would be deployed. Groups of resources in a layer would be delimited using a `## Layer: ` comment indicate the beginning of each layer @@ -159,7 +159,7 @@ metadata: ## Backwards compatibility -Helm will continue to deploy all resources and dependencies at once, as adding the above defined annotations is when the installation behavior would change. +Helm will continue to deploy all resources and dependencies at once, as adding the above defined annotations is when the installation behavior would change. If a template in a chart defines `helm.sh/depends-on/chart: someChartDependency` annotation, Helm will wait for the subchart to be ready for all resources with default readiness checks. This will also apply to subcharts that do not have any sequencing annotations. ## Security implications From de97e1ce2bcab5c1da4656b66a3cf5cf3bc7a6d2 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Mon, 2 Dec 2024 15:46:31 +0000 Subject: [PATCH 07/40] Add one more wording change Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index abfe5fcb..1684a684 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -180,7 +180,7 @@ N/A ## Open issues - Choose between Proposal 1 and Proposal 2 -- Should this featureset take into account allowing Helm Developers to declare custom "readiness" definitions for a given layer, besides the default? +- Should this featureset take into account allowing Application developers to declare custom "readiness" definitions for given resources, besides the default? ## Prior raised issues From e83cb0aaced15ce164cd76b6e098f421dbec3796 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Fri, 6 Dec 2024 18:51:57 +0000 Subject: [PATCH 08/40] Make helm.sh/depends-on/ annotations plural Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 1684a684..e7952512 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -9,7 +9,7 @@ status: "draft" ## Abstract -This HIP is to propose a new featureset in Helm 4 to provide Application developers, who create heam charts for their applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. +This HIP is to propose a new featureset in Helm 4 to provide Application developers, who create helm charts for their applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. ## Motivation @@ -30,9 +30,9 @@ This design was proposed as an alternative that functions very similarly to hook ### Proposal 1: Named Dependencies At a high level, allow Chart Developers to assign named dependency annotations to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. Three new annotations would be added to enable this described next. -- `helm.sh/layer`: Declare a layer that a given resource belongs to. Any numebr of resources can belong to a layer. -- `helm.sh/depends-on/layer`: Declare a layer that must exist and in a ready state before this resource can be deployed. -- `helm.sh/depends-on/chart`: Declare a chart dependency that must exist and in a ready state before this resource can be deployed. For the chart to be declared ready, all of its resources, with their sequencing order taken into consideration, would need to be deployed and declared ready. +- `helm.sh/layer`: Declare a layer that a given resource belongs to. Any numebr of resources can belong to a layer. A resource can only belong to one layer. +- `helm.sh/depends-on/layers`: Declare layers that must exist and in a ready state before this resource can be deployed. Order of the layers has no significance +- `helm.sh/depends-on/charts`: Declare chart dependencies that must exist and in a ready state before the parent chart can be deployed. For the dependency or subchart chart to be declared ready, all of its resources, with their sequencing order taken into consideration, would need to be deployed and declared ready. Order of the charts has no significance. A layer would not be considered "ready" and the next layer deployed until all resources in that layer are considered "ready". Readiness is described in a later section. @@ -40,24 +40,24 @@ A layer would not be considered "ready" and the next layer deployed until all re ```yaml # resource 1 metadata: - name: foo + name: db-service annotations: - helm.sh/layer: layer1 + helm.sh/layer: database --- # resource 2 metadata: - name: bar + name: app-ingress annotations: - helm.sh/layer: layer2 - helm.sh/depends-on/layer: layer1 + helm.sh/layer: app + helm.sh/depends-on/layers: "database, queue" --- # resource 3 metadata: - name: fizz + name: queue-processor annotations: - helm.sh/layer: layer3 - helm.sh/depends-on/layer: layer1 - helm.sh/depends-on/chart: someChartDependency + helm.sh/layer: queue + helm.sh/depends-on/layers: another-layer + helm.sh/depends-on/charts: "rabbitmq, minio" ``` In this example, Helm would be responsible for resolving the annotations on these three resources and deploy all resources in the following order: @@ -128,7 +128,7 @@ status: #### Sequencing order -Resources with sequencing annotations would be deployed first followed by resources without. If a resource has a `helm.sh/depends-on/chart` annotation, all resources of the referred subchart would be deployed and checked for readiness before deploying the dependant resource. Only resources that Helm can determine their readiness for will be checked. Chart authors would need to annotate their chart resources accordingly. Helm will use default readiness checks for subcharts that do have their templates annotated. +Resources with sequencing annotations would be deployed first followed by resources without. If a resource has a `helm.sh/depends-on/charts` annotation, all resources of the referred subchart would be deployed and checked for readiness before deploying the dependant resource. Only resources that Helm can determine their readiness for will be checked. Chart authors would need to annotate their chart resources accordingly. Helm will use default readiness checks for subcharts that do have their templates annotated. Helm would scope each subchart layer annotation names using a delimiter e.g `someChartDependency#mylayer` to avoid any name collisions. This is an internal implementation detail rather than feature chart authors or operators would need to know. @@ -159,7 +159,7 @@ metadata: ## Backwards compatibility -Helm will continue to deploy all resources and dependencies at once, as adding the above defined annotations is when the installation behavior would change. If a template in a chart defines `helm.sh/depends-on/chart: someChartDependency` annotation, Helm will wait for the subchart to be ready for all resources with default readiness checks. This will also apply to subcharts that do not have any sequencing annotations. +Helm will continue to deploy all resources and dependencies at once, as adding the above defined annotations is when the installation behavior would change. If a template in a chart defines `helm.sh/depends-on/charts: someChartDependency` annotation, Helm will wait for the subchart to be ready for all resources with default readiness checks. This will also apply to subcharts that do not have any sequencing annotations. ## Security implications From 6712256a0ab09af7ac59ff119613b970e3f8fb82 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Fri, 6 Dec 2024 18:57:45 +0000 Subject: [PATCH 09/40] Update hips/hip-00xx-sequencing.md Co-authored-by: Scott Rigby Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index e7952512..076f640d 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -113,7 +113,7 @@ In this example, Helm would be responsible for resolving the annotations on thes ### Readiness -In order to enforce sequencing, new `helm.sh/resource-ready` annotation would be used to determine when a resource is ready, allowing helm to proceed deploying the next group of resources, or failing a deployment. Helm would query, using jsonpath syntax, status fields of the annotated resource. Some native kubernetes resources that have stable APIs e.g `v1` resources such as `Pod` and `Deployment`, would have default queries which can be overriden. In Helm cannot determine how to check a resource's readiness when it should, it will do nothing and log this. +In order to enforce sequencing, new `helm.sh/resource-ready` annotation would be used to determine when a resource is ready, allowing helm to proceed deploying the next group of resources, or failing a deployment. Helm would query, using jsonpath syntax, status fields of the annotated resource. Some native kubernetes resources that have stable APIs e.g `v1` resources such as `Pod` and `Deployment`, would have default queries which can be overriden. If Helm cannot determine how to check a resource's readiness when it should, it will do nothing and log this. #### Example ```yaml From 778b4d15b09157e44eef0467d8b37ba7c18fdcdc Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Fri, 6 Dec 2024 19:30:51 +0000 Subject: [PATCH 10/40] Add --wait/--wait-for-job comment Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 1 + 1 file changed, 1 insertion(+) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 076f640d..334965a6 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -181,6 +181,7 @@ N/A - Choose between Proposal 1 and Proposal 2 - Should this featureset take into account allowing Application developers to declare custom "readiness" definitions for given resources, besides the default? +- How will `--wait` and `--wait-for-jobs` work with sequencing annotations? ## Prior raised issues From 66b3861e83e89dbd8ac24a09a478d2ff5fd1348f Mon Sep 17 00:00:00 2001 From: Joe Beck Date: Tue, 10 Dec 2024 12:00:24 -0800 Subject: [PATCH 11/40] Updates from PR comments. Move proposal 2 to rejected ideas, clarify intentions on how/why this should be built Signed-off-by: Joe Beck --- hips/hip-00xx-sequencing.md | 61 ++++++++----------------------------- 1 file changed, 13 insertions(+), 48 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 334965a6..3363bfd9 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -13,28 +13,25 @@ This HIP is to propose a new featureset in Helm 4 to provide Application develop ## Motivation +The driving motivator here is to allow application developers to control what order resources are bundled and sent to the K8s API server, referred to as resource sequencing for the rest of this HIP. + Today, to accomplish resource sequencing you have two options. The first is leveraging helm hooks, the second is building required sequencing into the application (via startup code or init containers). The existing hooks and weights can be tedious to build and maintain for Application developers, and built-in app sequencing can unecessarily increase complexity of a Helm application that needs to be maintained by Application Developers. Helm as a package manager should be capable of enabling developers to properly sequence how their applications are deployed, and by providing such a mechanism to developers, this will significantly improve the Application developer's experience. Additionally, Helm currently doesn't provide a way to sequence when chart dependencies are deployed, and this featureset would ideally address this. ## Rationale -### Proposal 1: Named Dependencies This design was chosen due to simplicity and clarity of how it works for both the Helm developer and Application Operator -### Proposal 2: Weighted Resouces (without hooks) -This design was proposed as an alternative that functions very similarly to hooks today, to provide the same sequencing functionality outside of the scope of hooks. - ## Specification -### Proposal 1: Named Dependencies At a high level, allow Chart Developers to assign named dependency annotations to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. Three new annotations would be added to enable this described next. - `helm.sh/layer`: Declare a layer that a given resource belongs to. Any numebr of resources can belong to a layer. A resource can only belong to one layer. - `helm.sh/depends-on/layers`: Declare layers that must exist and in a ready state before this resource can be deployed. Order of the layers has no significance - `helm.sh/depends-on/charts`: Declare chart dependencies that must exist and in a ready state before the parent chart can be deployed. For the dependency or subchart chart to be declared ready, all of its resources, with their sequencing order taken into consideration, would need to be deployed and declared ready. Order of the charts has no significance. -A layer would not be considered "ready" and the next layer deployed until all resources in that layer are considered "ready". Readiness is described in a later section. +The installation process would group resources in the same layer and send them to the K8s API Server in one bundle, and once all resources are "ready", the next layer would be installed. A layer would not be considered "ready" and the next layer installed until all resources in that layer are considered "ready". Readiness is described in a later section. A similar process would apply for upgrades. Uninstalls would function on the same layer order, but backwards, where a layer is not uninstalled until all layers that depend on it are first uninstalled. #### Example: ```yaml @@ -75,42 +72,6 @@ layer3: [fizz] This approach is prone to circular dependencies. During the templating phase, Helm will have logic to detect, and report any circular dependencies found in the chart templates. -### Proposal 2: Weighted Resouces (without hooks) - -Expand on the the weight capabilities of hooks, but allow them to be defined outside of the context of hooks for the chart. Add a new annotation `helm.sh/weight` that helm then uses to order when resources are deployed based on the same weighting rules that hooks use today. - -#### Example: -```yaml -# resource 1 -metadata: - name: foo - annotations: - helm.sh/weight: "-2" ---- -# resource 2 -metadata: - name: bar - annotations: - helm.sh/weight: "-1" ---- -# resource 3 -metadata: - name: fizz - annotations: - helm.sh/weight: 0 -``` -In this example, Helm would be responsible for resolving the annotations on these three resources and deploy all resources in the following order: - -``` --2: [foo] - || - \/ --1: [bar] - || - \/ -0: [fizz] -``` - ### Readiness In order to enforce sequencing, new `helm.sh/resource-ready` annotation would be used to determine when a resource is ready, allowing helm to proceed deploying the next group of resources, or failing a deployment. Helm would query, using jsonpath syntax, status fields of the annotated resource. Some native kubernetes resources that have stable APIs e.g `v1` resources such as `Pod` and `Deployment`, would have default queries which can be overriden. If Helm cannot determine how to check a resource's readiness when it should, it will do nothing and log this. @@ -140,26 +101,27 @@ Helm would scope each subchart layer annotation names using a delimiter e.g `som metadata: name: foo annotations: - helm.sh/weight: "-2" + helm.sh/layer: layer1 --- # resource 2 metadata: name: bar annotations: - helm.sh/weight: "-1" + helm.sh/layer: layer1 --- ## Layer: layer2 # resource 3 metadata: name: fizz annotations: - helm.sh/weight: 0 + helm.sh/layer: layer2 + helm.sh/depends-on/layers: "layer1" ``` ## Backwards compatibility -Helm will continue to deploy all resources and dependencies at once, as adding the above defined annotations is when the installation behavior would change. If a template in a chart defines `helm.sh/depends-on/charts: someChartDependency` annotation, Helm will wait for the subchart to be ready for all resources with default readiness checks. This will also apply to subcharts that do not have any sequencing annotations. +Helm will continue to install/upgrade/uninstall all resources and dependencies at once, as adding the above defined annotations is when the installation behavior would change. If a template in a chart defines `helm.sh/depends-on/charts: someChartDependency` annotation, Helm will wait for the subchart to be ready for all resources with default readiness checks. This will also apply to subcharts that do not have any sequencing annotations. ## Security implications @@ -175,13 +137,16 @@ N/A ## Rejected ideas -N/A +1. A weight based system, similar to Helm hooks + - Static numbering of the order is more challenging to develop and maintain + - Modifying the order can lead to cascading changes. + - Dynamically named system solves these problems for the application developers. ## Open issues -- Choose between Proposal 1 and Proposal 2 - Should this featureset take into account allowing Application developers to declare custom "readiness" definitions for given resources, besides the default? - How will `--wait` and `--wait-for-jobs` work with sequencing annotations? +- Chart dependencies should be part of the Chart.yaml instead. ## Prior raised issues From c018bbb4e7a746661d046aee5a4bffa457c0592e Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Fri, 13 Dec 2024 19:35:14 +0000 Subject: [PATCH 12/40] Descibe how to handle subchart dependencies Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 98 ++++++++++++++++++++++++++----------- 1 file changed, 69 insertions(+), 29 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 3363bfd9..60b7e5fa 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -9,7 +9,8 @@ status: "draft" ## Abstract -This HIP is to propose a new featureset in Helm 4 to provide Application developers, who create helm charts for their applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. +This HIP is to propose a new featureset in Helm 4 to provide Application developers, who create helm charts for their applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. Helm deploys all manifests at the same time. This HIP will propose a way for Helm to deploy manifests in batches, and inspect states of these resources to enforce sequencing. + ## Motivation @@ -17,7 +18,7 @@ The driving motivator here is to allow application developers to control what or Today, to accomplish resource sequencing you have two options. The first is leveraging helm hooks, the second is building required sequencing into the application (via startup code or init containers). The existing hooks and weights can be tedious to build and maintain for Application developers, and built-in app sequencing can unecessarily increase complexity of a Helm application that needs to be maintained by Application Developers. Helm as a package manager should be capable of enabling developers to properly sequence how their applications are deployed, and by providing such a mechanism to developers, this will significantly improve the Application developer's experience. -Additionally, Helm currently doesn't provide a way to sequence when chart dependencies are deployed, and this featureset would ideally address this. +Additionally, Helm currently doesn't provide a way to sequence when chart dependencies are deployed and this featureset would ideally address this. ## Rationale @@ -25,15 +26,19 @@ This design was chosen due to simplicity and clarity of how it works for both th ## Specification -At a high level, allow Chart Developers to assign named dependency annotations to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. Three new annotations would be added to enable this described next. +At a high level, allow Chart Developers to assign named dependencies to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. The following annotations would be added to enable this. + +*Additions to templates* +- `helm.sh/layer`: Annotation to declare a layer that a given resource belongs to. Any number of resources can belong to a layer. A resource can only belong to one layer. +- `helm.sh/depends-on/layers`: Annotation to declare layers that must exist and in a ready state before this resource can be deployed. Order of the layers has no significance -- `helm.sh/layer`: Declare a layer that a given resource belongs to. Any numebr of resources can belong to a layer. A resource can only belong to one layer. -- `helm.sh/depends-on/layers`: Declare layers that must exist and in a ready state before this resource can be deployed. Order of the layers has no significance -- `helm.sh/depends-on/charts`: Declare chart dependencies that must exist and in a ready state before the parent chart can be deployed. For the dependency or subchart chart to be declared ready, all of its resources, with their sequencing order taken into consideration, would need to be deployed and declared ready. Order of the charts has no significance. +*Additions to Chart.yaml* +- `helm.sh/depends-on/charts`: Annotation added to `Chart.yaml` to declare chart dependencies, by name or tags, that must be deployed fully and in a ready state before the chart can be deployed. For the dependency or subchart chart to be declared ready, all of its resources, with their sequencing order taken into consideration, would need to be deployed and declared ready. Order of the charts has no significance. +- `depends-on`: A new field added to `Chart.yaml` `dependencies` fields that is meant to declare a list of subcharts, by name or tag, that need to be ready before the subchart in questions get installed. This will be used to create a dependency graph for subcharts. -The installation process would group resources in the same layer and send them to the K8s API Server in one bundle, and once all resources are "ready", the next layer would be installed. A layer would not be considered "ready" and the next layer installed until all resources in that layer are considered "ready". Readiness is described in a later section. A similar process would apply for upgrades. Uninstalls would function on the same layer order, but backwards, where a layer is not uninstalled until all layers that depend on it are first uninstalled. +The installation process would group resources in the same layer and send them to the K8s API Server in one bundle, and once all resources are "ready", the next layer would be installed. A layer would not be considered "ready" and the next layer installed until all resources in that layer are considered "ready". Readiness is described in a later section. A similar process would apply for upgrades. Uninstalls would function on the same layer order, but backwards, where a layer is not uninstalled until all layers that depend on it are first uninstalled. Upgrades would follow the same order as installation. -#### Example: +#### Templates example: ```yaml # resource 1 metadata: @@ -43,7 +48,7 @@ metadata: --- # resource 2 metadata: - name: app-ingress + name: my-app annotations: helm.sh/layer: app helm.sh/depends-on/layers: "database, queue" @@ -56,25 +61,61 @@ metadata: helm.sh/depends-on/layers: another-layer helm.sh/depends-on/charts: "rabbitmq, minio" ``` -In this example, Helm would be responsible for resolving the annotations on these three resources and deploy all resources in the following order: +In this example, Helm would be responsible for resolving the annotations on these three resources and deploy all resources in the following order. Resources in `database` and `queue` layers would be deployed at the same time. They would need to be ready before attempting to deploy `app` layer resources: + +``` +"database" layer: [db-service] + || +"queue" layer: || [queue-processor] + || || + || || + \/ \/ + \\ // + \\ // + \/ \/ +"app" layer: [my-app] +``` + +#### Chart dependencies example + +```yaml +name: foo +annotations: + helm.sh/depends-on/charts: ["bar", "rabbitmq"] +dependencies: + - name: nginx + version: "18.3.1" + repository: "oci://registry-1.docker.io/bitnamicharts" + - name: rabbitmq + version: "9.3.1" + repository: "oci://registry-1.docker.io/bitnamicharts" + - name: bar # This is a subchart packaged with "foo" in charts dir. It's not pulled from a remote location + version: "0.1.0" + depends-on: ["nginx", "rabbitmq"] + condition: bar.enabled +``` ``` -layer1: [foo] - || - || [someChartDependency] - || || - \/ || -layer2: [bar] || - \\ // - \/ \/ -layer3: [fizz] + [nginx] [rabbitmq] + || || + || || + \/ \/ + \\ // + \/ \/ + [bar] + || + || + \/ + [foo] ``` -This approach is prone to circular dependencies. During the templating phase, Helm will have logic to detect, and report any circular dependencies found in the chart templates. +In this example, helm will first install and wait for all resources of `nginx` and `rabbitmq` dependencies to be "ready" before attempting to install `bar` resources. Once all resources of `bar` are "ready" then and only then will `foo` chart resources be installed. `foo` would require `rabbitmq` to be ready but since the subchart resources would have been installed before `bar`, this requirement would have been fulfilled. + +This approach of building a directed acyclic graph (DAG) is prone to circular dependencies. During the templating phase, Helm will have logic to detect, and report any circular dependencies found in the chart templates. ### Readiness -In order to enforce sequencing, new `helm.sh/resource-ready` annotation would be used to determine when a resource is ready, allowing helm to proceed deploying the next group of resources, or failing a deployment. Helm would query, using jsonpath syntax, status fields of the annotated resource. Some native kubernetes resources that have stable APIs e.g `v1` resources such as `Pod` and `Deployment`, would have default queries which can be overriden. If Helm cannot determine how to check a resource's readiness when it should, it will do nothing and log this. +In order to enforce sequencing, a new `helm.sh/resource-ready` annotation would be used to determine when a resource is ready, allowing helm to proceed deploying the next group of resources, or failing a deployment. Helm would query, using jsonpath syntax, status fields of the annotated resource. Some native kubernetes resources that have stable APIs e.g `v1` resources such as `Pod` and `Deployment`, would have default queries which can be overriden. If Helm cannot determine how to check a resource's readiness when it should, it will do nothing and log this. #### Example ```yaml @@ -89,14 +130,12 @@ status: #### Sequencing order -Resources with sequencing annotations would be deployed first followed by resources without. If a resource has a `helm.sh/depends-on/charts` annotation, all resources of the referred subchart would be deployed and checked for readiness before deploying the dependant resource. Only resources that Helm can determine their readiness for will be checked. Chart authors would need to annotate their chart resources accordingly. Helm will use default readiness checks for subcharts that do have their templates annotated. - -Helm would scope each subchart layer annotation names using a delimiter e.g `someChartDependency#mylayer` to avoid any name collisions. This is an internal implementation detail rather than feature chart authors or operators would need to know. +Resources with sequencing annotations would be deployed first followed by resources without. If the chart has a `helm.sh/depends-on/charts` annotation in `Chart.yaml`, all resources of the defined subcharts would be deployed and checked for readiness before deploying the chart. Only resources that Helm can determine their readiness for will be checked. Chart authors would need to annotate their chart resources accordingly. Helm will use default readiness checks for subcharts that do have their templates annotated. -`helm template` would print all resources in the order they would be deployed. Groups of resources in a layer would be delimited using a `## Layer: ` comment indicate the beginning of each layer +`helm template` would print all resources in the order they would be deployed. Groups of resources in a layer would be delimited using a `## START layer: ` comment indicating the beginning of each layer and `END layer: `. ```yaml -## Layer: layer1 +## START layer: foo layer1 # resource 1 metadata: name: foo @@ -108,20 +147,21 @@ metadata: name: bar annotations: helm.sh/layer: layer1 +## END layer: foo layer1 --- -## Layer: layer2 +## START layer: bar layer1 # resource 3 metadata: name: fizz annotations: helm.sh/layer: layer2 helm.sh/depends-on/layers: "layer1" - +## END layer: bar layer2 ``` ## Backwards compatibility -Helm will continue to install/upgrade/uninstall all resources and dependencies at once, as adding the above defined annotations is when the installation behavior would change. If a template in a chart defines `helm.sh/depends-on/charts: someChartDependency` annotation, Helm will wait for the subchart to be ready for all resources with default readiness checks. This will also apply to subcharts that do not have any sequencing annotations. +Helm will continue to install/upgrade/uninstall all resources and dependencies at once, as adding the above defined annotations and depends-on field is when the installation behaviour would change. If a chart defines `helm.sh/depends-on/charts: ["subchart"]` annotation in `Chart.yaml`, Helm will wait for the subchart to be ready for all resources with default readiness checks. This will also apply to older subcharts that do not have any sequencing annotations. ## Security implications From 48347585154a7012b86118ef5570243e166f2f41 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Tue, 17 Dec 2024 19:11:43 +0000 Subject: [PATCH 13/40] Rename application developers to application distributors Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 60b7e5fa..b8244107 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -9,14 +9,14 @@ status: "draft" ## Abstract -This HIP is to propose a new featureset in Helm 4 to provide Application developers, who create helm charts for their applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. Helm deploys all manifests at the same time. This HIP will propose a way for Helm to deploy manifests in batches, and inspect states of these resources to enforce sequencing. +This HIP is to propose a new featureset in Helm 4 to provide application distributors, who create helm charts for applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. Helm deploys all manifests at the same time. This HIP will propose a way for Helm to deploy manifests in batches, and inspect states of these resources to enforce sequencing. ## Motivation -The driving motivator here is to allow application developers to control what order resources are bundled and sent to the K8s API server, referred to as resource sequencing for the rest of this HIP. +The driving motivator here is to allow application distributors to control what order resources are bundled and sent to the K8s API server, referred to as resource sequencing for the rest of this HIP. -Today, to accomplish resource sequencing you have two options. The first is leveraging helm hooks, the second is building required sequencing into the application (via startup code or init containers). The existing hooks and weights can be tedious to build and maintain for Application developers, and built-in app sequencing can unecessarily increase complexity of a Helm application that needs to be maintained by Application Developers. Helm as a package manager should be capable of enabling developers to properly sequence how their applications are deployed, and by providing such a mechanism to developers, this will significantly improve the Application developer's experience. +Today, to accomplish resource sequencing you have two options. The first is leveraging helm hooks, the second is building required sequencing into the application (via startup code or init containers). The existing hooks and weights can be tedious to build and maintain for application distributors, and built-in app sequencing can unecessarily increase complexity of a Helm application that needs to be maintained by application distributors. Helm as a package manager should be capable of enabling application distributors to properly sequence how their chart resources are deployed, and by providing such a mechanism to distributors, this will significantly improve the application distributor's experience. Additionally, Helm currently doesn't provide a way to sequence when chart dependencies are deployed and this featureset would ideally address this. @@ -180,11 +180,11 @@ N/A 1. A weight based system, similar to Helm hooks - Static numbering of the order is more challenging to develop and maintain - Modifying the order can lead to cascading changes. - - Dynamically named system solves these problems for the application developers. + - Dynamically named system solves these problems for the application distributors. ## Open issues -- Should this featureset take into account allowing Application developers to declare custom "readiness" definitions for given resources, besides the default? +- Should this featureset take into account allowing application distributors to declare custom "readiness" definitions for given resources, besides the default? - How will `--wait` and `--wait-for-jobs` work with sequencing annotations? - Chart dependencies should be part of the Chart.yaml instead. From 11d56eae6f0a8dc058305edf7871f7394cb38a05 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Wed, 26 Feb 2025 14:59:07 -0800 Subject: [PATCH 14/40] Explicitly state the separation of the ordering between charts and resources Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index b8244107..920df991 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -38,6 +38,8 @@ At a high level, allow Chart Developers to assign named dependencies to both the The installation process would group resources in the same layer and send them to the K8s API Server in one bundle, and once all resources are "ready", the next layer would be installed. A layer would not be considered "ready" and the next layer installed until all resources in that layer are considered "ready". Readiness is described in a later section. A similar process would apply for upgrades. Uninstalls would function on the same layer order, but backwards, where a layer is not uninstalled until all layers that depend on it are first uninstalled. Upgrades would follow the same order as installation. +`helm.sh/layer` and `helm.sh/depends-on/layers` ordering annotations are only used when ordering resources in the same chart. When it comes to resources across charts one needs to declare `helm.sh/depends-on/charts` or `depends-on` in Chart.yaml to allow Helm to reason how to order install/upgrade/uninstall subcharts. + #### Templates example: ```yaml # resource 1 From ea37905d8c844e8357fe6941bc8305a052610b87 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Thu, 20 Mar 2025 18:04:00 +0000 Subject: [PATCH 15/40] chore: add documentation bullet Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 1 + 1 file changed, 1 insertion(+) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 920df991..0b71fc1d 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -172,6 +172,7 @@ None. ## How to teach this TBD upon deciding on a design. +- Document how sequencing works in the official helm documentation website. Include ordering of kubernetes resources the Helm enforces when applying resources to the cluster. ## Reference implementation From 04b860418c517d1fab6bcb2a5d93ce9dfe41f28b Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Tue, 25 Mar 2025 18:30:04 +0000 Subject: [PATCH 16/40] chore: clarify that hooks are not ordered Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 0b71fc1d..cf322a83 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -11,6 +11,7 @@ status: "draft" This HIP is to propose a new featureset in Helm 4 to provide application distributors, who create helm charts for applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. Helm deploys all manifests at the same time. This HIP will propose a way for Helm to deploy manifests in batches, and inspect states of these resources to enforce sequencing. +The HIP only targets resources deployed in the Helm install phase. Resources deployed as hooks are not sequenced using changes proposed here. Any sequencing of hooks will still rely in using `"hook-weight"` annotations. ## Motivation @@ -40,7 +41,7 @@ The installation process would group resources in the same layer and send them t `helm.sh/layer` and `helm.sh/depends-on/layers` ordering annotations are only used when ordering resources in the same chart. When it comes to resources across charts one needs to declare `helm.sh/depends-on/charts` or `depends-on` in Chart.yaml to allow Helm to reason how to order install/upgrade/uninstall subcharts. -#### Templates example: +#### Template examples: ```yaml # resource 1 metadata: From 8a8a9061b8e84f868e854cef7fcff73601d5d83a Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Tue, 25 Mar 2025 18:44:32 +0000 Subject: [PATCH 17/40] chore: some minor changes after self review Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index cf322a83..db1c3f4b 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -11,7 +11,7 @@ status: "draft" This HIP is to propose a new featureset in Helm 4 to provide application distributors, who create helm charts for applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. Helm deploys all manifests at the same time. This HIP will propose a way for Helm to deploy manifests in batches, and inspect states of these resources to enforce sequencing. -The HIP only targets resources deployed in the Helm install phase. Resources deployed as hooks are not sequenced using changes proposed here. Any sequencing of hooks will still rely in using `"hook-weight"` annotations. +The HIP only targets resources deployed in the Helm install phase. Resources deployed as hooks are not sequenced using changes proposed here. Any sequencing of hooks will still rely in using `"helm.sh/hook-weight"` annotations. ## Motivation @@ -54,15 +54,14 @@ metadata: name: my-app annotations: helm.sh/layer: app - helm.sh/depends-on/layers: "database, queue" + helm.sh/depends-on/layers: ["database, queue"] --- # resource 3 metadata: name: queue-processor annotations: helm.sh/layer: queue - helm.sh/depends-on/layers: another-layer - helm.sh/depends-on/charts: "rabbitmq, minio" + helm.sh/depends-on/layers: ["another-layer"] ``` In this example, Helm would be responsible for resolving the annotations on these three resources and deploy all resources in the following order. Resources in `database` and `queue` layers would be deployed at the same time. They would need to be ready before attempting to deploy `app` layer resources: @@ -126,7 +125,7 @@ kind: Job metadata: name: barz annotations: - helm.sh/resource-ready: "status.succeeded==1" + helm.sh/resource-ready: ["status.succeeded==1", "status.failed==1"] status: succeeded: 1 ``` @@ -189,7 +188,6 @@ N/A ## Open issues - Should this featureset take into account allowing application distributors to declare custom "readiness" definitions for given resources, besides the default? -- How will `--wait` and `--wait-for-jobs` work with sequencing annotations? - Chart dependencies should be part of the Chart.yaml instead. ## Prior raised issues From 2289f35a71bdab919d6a928700dabe4edcc00f59 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Tue, 25 Mar 2025 19:25:41 +0000 Subject: [PATCH 18/40] chore: add how --wait flag behaves with sequencing Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index db1c3f4b..2fbe3b47 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -27,7 +27,11 @@ This design was chosen due to simplicity and clarity of how it works for both th ## Specification -At a high level, allow Chart Developers to assign named dependencies to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. The following annotations would be added to enable this. +At a high level, allow Chart Developers to assign named dependencies to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. + +For Helm CLI, the `--wait` flag will enable sequencing where resources are applied in groups of layers. SDK users will also be able to enable sequencing by setting a `Wait` boolean flag. Without this flag being enabled, resources will all be applied in one go. + +The following annotations would be added to enable this. *Additions to templates* - `helm.sh/layer`: Annotation to declare a layer that a given resource belongs to. Any number of resources can belong to a layer. A resource can only belong to one layer. @@ -172,7 +176,7 @@ None. ## How to teach this TBD upon deciding on a design. -- Document how sequencing works in the official helm documentation website. Include ordering of kubernetes resources the Helm enforces when applying resources to the cluster. +- Document how sequencing works in the official helm documentation website. Include ordering of kubernetes resources the Helm enforces when applying resources to the cluster. Examples will be added to best demonstrate how this feature works. ## Reference implementation @@ -189,6 +193,8 @@ N/A - Should this featureset take into account allowing application distributors to declare custom "readiness" definitions for given resources, besides the default? - Chart dependencies should be part of the Chart.yaml instead. +- How does `--atomic` flag work? +- How will `--wait-for-jobs` work with sequencing? Should we keep the flag, and wait for jobs after all other resources have been applied and waited for? ## Prior raised issues From bb2da558596723272f5519ad62a2440147ba5761 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Tue, 25 Mar 2025 19:27:00 +0000 Subject: [PATCH 19/40] chore: rename charts to subcharts Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 2fbe3b47..71f200c2 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -38,12 +38,12 @@ The following annotations would be added to enable this. - `helm.sh/depends-on/layers`: Annotation to declare layers that must exist and in a ready state before this resource can be deployed. Order of the layers has no significance *Additions to Chart.yaml* -- `helm.sh/depends-on/charts`: Annotation added to `Chart.yaml` to declare chart dependencies, by name or tags, that must be deployed fully and in a ready state before the chart can be deployed. For the dependency or subchart chart to be declared ready, all of its resources, with their sequencing order taken into consideration, would need to be deployed and declared ready. Order of the charts has no significance. +- `helm.sh/depends-on/subcharts`: Annotation added to `Chart.yaml` to declare chart dependencies, by name or tags, that must be deployed fully and in a ready state before the chart can be deployed. For the dependency or subchart chart to be declared ready, all of its resources, with their sequencing order taken into consideration, would need to be deployed and declared ready. Order of the charts has no significance. - `depends-on`: A new field added to `Chart.yaml` `dependencies` fields that is meant to declare a list of subcharts, by name or tag, that need to be ready before the subchart in questions get installed. This will be used to create a dependency graph for subcharts. The installation process would group resources in the same layer and send them to the K8s API Server in one bundle, and once all resources are "ready", the next layer would be installed. A layer would not be considered "ready" and the next layer installed until all resources in that layer are considered "ready". Readiness is described in a later section. A similar process would apply for upgrades. Uninstalls would function on the same layer order, but backwards, where a layer is not uninstalled until all layers that depend on it are first uninstalled. Upgrades would follow the same order as installation. -`helm.sh/layer` and `helm.sh/depends-on/layers` ordering annotations are only used when ordering resources in the same chart. When it comes to resources across charts one needs to declare `helm.sh/depends-on/charts` or `depends-on` in Chart.yaml to allow Helm to reason how to order install/upgrade/uninstall subcharts. +`helm.sh/layer` and `helm.sh/depends-on/layers` ordering annotations are only used when ordering resources in the same chart. When it comes to resources across charts one needs to declare `helm.sh/depends-on/subcharts` or `depends-on` in Chart.yaml to allow Helm to reason how to order install/upgrade/uninstall subcharts. #### Template examples: ```yaml @@ -87,7 +87,7 @@ In this example, Helm would be responsible for resolving the annotations on thes ```yaml name: foo annotations: - helm.sh/depends-on/charts: ["bar", "rabbitmq"] + helm.sh/depends-on/subcharts: ["bar", "rabbitmq"] dependencies: - name: nginx version: "18.3.1" From 1c9d5dbff1648d05539ffdd7859f27b0184bff4e Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Wed, 26 Mar 2025 13:11:06 +0000 Subject: [PATCH 20/40] chore: add high level summary of added features in abstract section Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 71f200c2..8cde0f45 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -11,7 +11,12 @@ status: "draft" This HIP is to propose a new featureset in Helm 4 to provide application distributors, who create helm charts for applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. Helm deploys all manifests at the same time. This HIP will propose a way for Helm to deploy manifests in batches, and inspect states of these resources to enforce sequencing. -The HIP only targets resources deployed in the Helm install phase. Resources deployed as hooks are not sequenced using changes proposed here. Any sequencing of hooks will still rely in using `"helm.sh/hook-weight"` annotations. +At a high level, this HIP proposes the following +- Ability for chart authors to specify how to sequence deployment of **resources within a single chart** +- Ability for chart authors to specify how to sequence **subcharts within a parent chart** +- Ability for Helm operators and tool developers to enable sequencing behaviour using `--wait` CLI flag and `Wait` SDK boolean parameter respectively + +The HIP only targets resources deployed in the Helm install phase. Resources deployed as hooks are not sequenced using changes proposed here. Any sequencing of hooks will still rely on using `"helm.sh/hook-weight"` annotations. Annotations added to resources in hooks will be ignored. ## Motivation @@ -29,7 +34,7 @@ This design was chosen due to simplicity and clarity of how it works for both th At a high level, allow Chart Developers to assign named dependencies to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. -For Helm CLI, the `--wait` flag will enable sequencing where resources are applied in groups of layers. SDK users will also be able to enable sequencing by setting a `Wait` boolean flag. Without this flag being enabled, resources will all be applied in one go. +For Helm CLI, the `--wait` flag will enable sequencing where resources are applied in groups of layers. SDK users will also be able to enable sequencing by setting a `Wait` boolean flag. Without this flag being enabled, resources will all be applied in one go which is the same behaviour in Helm v3. The following annotations would be added to enable this. From 8082240749f055da89bfe27bfe9085dc4c8a9a4e Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Wed, 30 Apr 2025 14:38:23 +0100 Subject: [PATCH 21/40] Document how install/uninstall/rollback/upgrade works with sequencing Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 36 +++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 8cde0f45..a2290adc 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -9,7 +9,7 @@ status: "draft" ## Abstract -This HIP is to propose a new featureset in Helm 4 to provide application distributors, who create helm charts for applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. Helm deploys all manifests at the same time. This HIP will propose a way for Helm to deploy manifests in batches, and inspect states of these resources to enforce sequencing. +This HIP is to propose a new featureset in Chart v3, to provide application distributors, who create helm charts for applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. Helm deploys all manifests at the same time. This HIP will propose a way for Helm to deploy manifests in batches, and inspect states of these resources to enforce sequencing. At a high level, this HIP proposes the following - Ability for chart authors to specify how to sequence deployment of **resources within a single chart** @@ -34,7 +34,9 @@ This design was chosen due to simplicity and clarity of how it works for both th At a high level, allow Chart Developers to assign named dependencies to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. -For Helm CLI, the `--wait` flag will enable sequencing where resources are applied in groups of layers. SDK users will also be able to enable sequencing by setting a `Wait` boolean flag. Without this flag being enabled, resources will all be applied in one go which is the same behaviour in Helm v3. +For Helm CLI, the `--wait` flag will enable sequencing where resources are applied in groups of layers. SDK users will also be able to enable sequencing by setting a `Wait` boolean flag. Without this flag being enabled, resources will all be applied in one go which is the same behaviour in Chart v2. + +Each release will store information of whether sequencing was used or not. This information is used when performing uninstalls and rollbacks. The following annotations would be added to enable this. @@ -122,11 +124,11 @@ dependencies: In this example, helm will first install and wait for all resources of `nginx` and `rabbitmq` dependencies to be "ready" before attempting to install `bar` resources. Once all resources of `bar` are "ready" then and only then will `foo` chart resources be installed. `foo` would require `rabbitmq` to be ready but since the subchart resources would have been installed before `bar`, this requirement would have been fulfilled. -This approach of building a directed acyclic graph (DAG) is prone to circular dependencies. During the templating phase, Helm will have logic to detect, and report any circular dependencies found in the chart templates. +This approach of building a directed acyclic graph (DAG) is prone to circular dependencies. During the templating phase, Helm will have logic to detect, and report any circular dependencies found in the chart templates. Helm will also provide a command to print the DAG for development and troubleshooting purposes. ### Readiness -In order to enforce sequencing, a new `helm.sh/resource-ready` annotation would be used to determine when a resource is ready, allowing helm to proceed deploying the next group of resources, or failing a deployment. Helm would query, using jsonpath syntax, status fields of the annotated resource. Some native kubernetes resources that have stable APIs e.g `v1` resources such as `Pod` and `Deployment`, would have default queries which can be overriden. If Helm cannot determine how to check a resource's readiness when it should, it will do nothing and log this. +In order to enforce sequencing, Helm will check that resources are ready using `kstatus` [ready condition](https://github.com/kubernetes-sigs/cli-utils/blob/master/pkg/kstatus/README.md#the-ready-condition) allowing helm to proceed installing the next group of resources, or fail the install. Users can override checks using optional `helm.sh/readiness-failure` and `helm.sh/readiness-success` annotations which are jsonpath queries to check status fields of the annotated resource. If any of the checks in the lists evaluate as true, the readiness check is determined as successful/failed. `helm.sh/readiness-timeout` annotation can be used to override how long Helm should wait when performing readiness checks before bailing out. It defaults to `10s`. #### Example ```yaml @@ -134,16 +136,23 @@ kind: Job metadata: name: barz annotations: - helm.sh/resource-ready: ["status.succeeded==1", "status.failed==1"] + helm.sh/readiness-success: ["succeeded==1"] # status fields + helm.sh/readiness-failure: ["failed==1"] # status fields + helm.sh/readiness-timeout: 20s # timeout status: succeeded: 1 ``` +A resources readiness is checked if there is a resource that depends on it as per the sequencing DAG. + #### Sequencing order -Resources with sequencing annotations would be deployed first followed by resources without. If the chart has a `helm.sh/depends-on/charts` annotation in `Chart.yaml`, all resources of the defined subcharts would be deployed and checked for readiness before deploying the chart. Only resources that Helm can determine their readiness for will be checked. Chart authors would need to annotate their chart resources accordingly. Helm will use default readiness checks for subcharts that do have their templates annotated. +Resources with sequencing annotations in a chart would be deployed first followed by resources without. If the chart has a `helm.sh/depends-on/subcharts` annotation in `Chart.yaml`, all resources of the defined subcharts would be deployed before deploying the main chart. If any sequencing annotations are defined in the subchart resources, Helm will enforce ordering of resources within. Sequencing of resources in a chart are sandboxed within the chart. Sequencing annotations will not affect resources in other charts. -`helm template` would print all resources in the order they would be deployed. Groups of resources in a layer would be delimited using a `## START layer: ` comment indicating the beginning of each layer and `END layer: `. +- Installs: Helm will install resources in the order defined by the DAG. If any of the readiness checks fail or timeout, the entire install would fail and the release marked as failed. If `--atomic`, or its SDK equivalent is used, a rollback to the last successful install would take place. +- Uninstalls: Helm would uninstall resources in the reverse order they were installed, as per the sequencing order. The logic to delete each resource will not change. +- Rollbacks: Helm will check from the release object whether the revision being rolled back to, was installed in a sequenced manner. If it was, Helm will respect and enforce this order when installing resources from that revision. When deleting unneeded resources of the revision being rolled back from, the reverse order is followed just like uninstalls. +- `helm template` would print all resources in the order they would be deployed. Groups of resources in a layer would be delimited using a `## START layer: ` comment indicating the beginning of each layer and `END layer: `. ```yaml ## START layer: foo layer1 @@ -172,7 +181,7 @@ metadata: ## Backwards compatibility -Helm will continue to install/upgrade/uninstall all resources and dependencies at once, as adding the above defined annotations and depends-on field is when the installation behaviour would change. If a chart defines `helm.sh/depends-on/charts: ["subchart"]` annotation in `Chart.yaml`, Helm will wait for the subchart to be ready for all resources with default readiness checks. This will also apply to older subcharts that do not have any sequencing annotations. +Helm will continue to install/upgrade/uninstall/rollback all resources and dependencies at one go for all charts using `Charts v2` and below. ## Security implications @@ -180,8 +189,8 @@ None. ## How to teach this -TBD upon deciding on a design. -- Document how sequencing works in the official helm documentation website. Include ordering of kubernetes resources the Helm enforces when applying resources to the cluster. Examples will be added to best demonstrate how this feature works. +- Document how sequencing works in the official helm documentation website. Include ordering of kubernetes resources that Helm enforces when applying resources to the cluster. Examples will be added to best demonstrate how this feature works. +- Document how to this feature works for SDK users. ## Reference implementation @@ -196,16 +205,9 @@ N/A ## Open issues -- Should this featureset take into account allowing application distributors to declare custom "readiness" definitions for given resources, besides the default? -- Chart dependencies should be part of the Chart.yaml instead. -- How does `--atomic` flag work? -- How will `--wait-for-jobs` work with sequencing? Should we keep the flag, and wait for jobs after all other resources have been applied and waited for? - ## Prior raised issues - https://github.com/helm/helm/pull/12541 - https://github.com/helm/helm/pull/9534 - https://github.com/helm/helm/issues/8439 - https://github.com/helm/community/pull/230 - -N/A From cb474def570e5c0f178499ce803364e7794b434b Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Tue, 13 May 2025 17:53:44 +0100 Subject: [PATCH 22/40] Use ordered wait strategy to enable sequencing Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index a2290adc..40b917b8 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -14,7 +14,7 @@ This HIP is to propose a new featureset in Chart v3, to provide application dist At a high level, this HIP proposes the following - Ability for chart authors to specify how to sequence deployment of **resources within a single chart** - Ability for chart authors to specify how to sequence **subcharts within a parent chart** -- Ability for Helm operators and tool developers to enable sequencing behaviour using `--wait` CLI flag and `Wait` SDK boolean parameter respectively +- Ability for Helm operators and tool developers to enable sequencing behaviour using `--wait=ordered` CLI flag and `WaitStrategy=ordered` SDK parameter respectively The HIP only targets resources deployed in the Helm install phase. Resources deployed as hooks are not sequenced using changes proposed here. Any sequencing of hooks will still rely on using `"helm.sh/hook-weight"` annotations. Annotations added to resources in hooks will be ignored. @@ -34,7 +34,7 @@ This design was chosen due to simplicity and clarity of how it works for both th At a high level, allow Chart Developers to assign named dependencies to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. -For Helm CLI, the `--wait` flag will enable sequencing where resources are applied in groups of layers. SDK users will also be able to enable sequencing by setting a `Wait` boolean flag. Without this flag being enabled, resources will all be applied in one go which is the same behaviour in Chart v2. +For Helm CLI, the `--wait=ordered` flag will enable sequencing where resources are applied in groups of layers. SDK users will also be able to enable sequencing by setting a `Wait` boolean flag. Without this flag being enabled, resources will all be applied in one go which is the same behaviour in Chart v2. Each release will store information of whether sequencing was used or not. This information is used when performing uninstalls and rollbacks. From 30a72b8231ce9a9fc1480a7ceb9f034ea894afaa Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Wed, 14 May 2025 17:07:34 +0100 Subject: [PATCH 23/40] Update hips/hip-00xx-sequencing.md Co-authored-by: Andrew Block Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 40b917b8..194e1e08 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -34,7 +34,7 @@ This design was chosen due to simplicity and clarity of how it works for both th At a high level, allow Chart Developers to assign named dependencies to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. -For Helm CLI, the `--wait=ordered` flag will enable sequencing where resources are applied in groups of layers. SDK users will also be able to enable sequencing by setting a `Wait` boolean flag. Without this flag being enabled, resources will all be applied in one go which is the same behaviour in Chart v2. +For Helm CLI, the `--wait=ordered` flag will enable sequencing where resources are applied in groups of layers. SDK users will also be able to enable sequencing by setting a `Wait` boolean flag. Without this flag being enabled, resources will all be applied all at once which is the same behaviour in Chart v2. Each release will store information of whether sequencing was used or not. This information is used when performing uninstalls and rollbacks. From be43171e9f7569c02d3bb375f6c5dae4d92c3711 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Wed, 14 May 2025 17:19:21 +0100 Subject: [PATCH 24/40] Update hips/hip-00xx-sequencing.md Co-authored-by: Andrew Block Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 194e1e08..3565dd60 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -122,7 +122,7 @@ dependencies: [foo] ``` -In this example, helm will first install and wait for all resources of `nginx` and `rabbitmq` dependencies to be "ready" before attempting to install `bar` resources. Once all resources of `bar` are "ready" then and only then will `foo` chart resources be installed. `foo` would require `rabbitmq` to be ready but since the subchart resources would have been installed before `bar`, this requirement would have been fulfilled. +In this example, Helm will first install and wait for all resources of `nginx` and `rabbitmq` dependencies to be "ready" before attempting to install `bar` resources. Once all resources of `bar` are "ready" then and only then will `foo` chart resources be installed. `foo` would require `rabbitmq` to be ready but since the subchart resources would have been installed before `bar`, this requirement would have been fulfilled. This approach of building a directed acyclic graph (DAG) is prone to circular dependencies. During the templating phase, Helm will have logic to detect, and report any circular dependencies found in the chart templates. Helm will also provide a command to print the DAG for development and troubleshooting purposes. From bad5acef69a65a590a517d50fbb4ebe5c885da43 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Wed, 14 May 2025 17:22:54 +0100 Subject: [PATCH 25/40] Update hips/hip-00xx-sequencing.md Co-authored-by: Andrew Block Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 3565dd60..c3bb7d48 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -147,7 +147,7 @@ A resources readiness is checked if there is a resource that depends on it as pe #### Sequencing order -Resources with sequencing annotations in a chart would be deployed first followed by resources without. If the chart has a `helm.sh/depends-on/subcharts` annotation in `Chart.yaml`, all resources of the defined subcharts would be deployed before deploying the main chart. If any sequencing annotations are defined in the subchart resources, Helm will enforce ordering of resources within. Sequencing of resources in a chart are sandboxed within the chart. Sequencing annotations will not affect resources in other charts. +Resources with sequencing annotations in a chart would be deployed first followed by resources without. If the chart has a `helm.sh/depends-on/subcharts` annotation in the `Chart.yaml`, all resources of the defined subcharts would be deployed before deploying the main chart. If any sequencing annotations are defined in the subchart resources, Helm will enforce ordering of resources within. Sequencing of resources in a chart are sandboxed within the chart. Sequencing annotations will not affect resources in other charts. - Installs: Helm will install resources in the order defined by the DAG. If any of the readiness checks fail or timeout, the entire install would fail and the release marked as failed. If `--atomic`, or its SDK equivalent is used, a rollback to the last successful install would take place. - Uninstalls: Helm would uninstall resources in the reverse order they were installed, as per the sequencing order. The logic to delete each resource will not change. From b3ee02ef56208a531501e1f42b56fa723bf2082a Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Wed, 14 May 2025 17:23:19 +0100 Subject: [PATCH 26/40] Update hips/hip-00xx-sequencing.md Co-authored-by: Andrew Block Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index c3bb7d48..2024d2e6 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -128,7 +128,7 @@ This approach of building a directed acyclic graph (DAG) is prone to circular de ### Readiness -In order to enforce sequencing, Helm will check that resources are ready using `kstatus` [ready condition](https://github.com/kubernetes-sigs/cli-utils/blob/master/pkg/kstatus/README.md#the-ready-condition) allowing helm to proceed installing the next group of resources, or fail the install. Users can override checks using optional `helm.sh/readiness-failure` and `helm.sh/readiness-success` annotations which are jsonpath queries to check status fields of the annotated resource. If any of the checks in the lists evaluate as true, the readiness check is determined as successful/failed. `helm.sh/readiness-timeout` annotation can be used to override how long Helm should wait when performing readiness checks before bailing out. It defaults to `10s`. +In order to enforce sequencing, Helm will check that resources are ready using `kstatus` [ready condition](https://github.com/kubernetes-sigs/cli-utils/blob/master/pkg/kstatus/README.md#the-ready-condition) allowing helm to proceed installing the next group of resources, or fail the install. Users can override checks using optional `helm.sh/readiness-failure` and `helm.sh/readiness-success` annotations which are jsonpath queries to check status fields of the annotated resource. If any of the checks in the lists evaluate as true, the readiness check is determined as successful/failed. `helm.sh/readiness-timeout` annotation can be used to override how long Helm should wait when performing readiness checks before raising an error. It defaults to `10s`. #### Example ```yaml From f22f65d1fa50faf2ad0e859a769e1a872e3a6cfd Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Wed, 14 May 2025 17:23:34 +0100 Subject: [PATCH 27/40] Update hips/hip-00xx-sequencing.md Co-authored-by: Andrew Block Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 2024d2e6..ab544fa2 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -189,7 +189,7 @@ None. ## How to teach this -- Document how sequencing works in the official helm documentation website. Include ordering of kubernetes resources that Helm enforces when applying resources to the cluster. Examples will be added to best demonstrate how this feature works. +- Document how sequencing works in the official helm documentation website. Include ordering of Kubernetes resources that Helm enforces when applying resources to the cluster. Examples will be added to best demonstrate how this feature works. - Document how to this feature works for SDK users. ## Reference implementation From c6236f6756ff6b90971080cabe1161425053f439 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Wed, 14 May 2025 17:23:50 +0100 Subject: [PATCH 28/40] Update hips/hip-00xx-sequencing.md Co-authored-by: Andrew Block Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index ab544fa2..f7bb22fc 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -190,7 +190,7 @@ None. ## How to teach this - Document how sequencing works in the official helm documentation website. Include ordering of Kubernetes resources that Helm enforces when applying resources to the cluster. Examples will be added to best demonstrate how this feature works. -- Document how to this feature works for SDK users. +- Document how this feature works for SDK users. ## Reference implementation From 9324191d672a8ab2003edcc3c31b75d622663e77 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Thu, 15 May 2025 16:56:48 +0100 Subject: [PATCH 29/40] Update hips/hip-00xx-sequencing.md Co-authored-by: Scott Rigby Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index f7bb22fc..8675c445 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -9,7 +9,7 @@ status: "draft" ## Abstract -This HIP is to propose a new featureset in Chart v3, to provide application distributors, who create helm charts for applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. Helm deploys all manifests at the same time. This HIP will propose a way for Helm to deploy manifests in batches, and inspect states of these resources to enforce sequencing. +This HIP is to propose a new featureset in Chart v3, to provide chart authors ([Application Distributor](https://github.com/helm/community/blob/main/user-profiles.md#2-application-distributor) Helm user profile), a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. Helm deploys all manifests at the same time. This HIP will propose a way for Helm to deploy manifests in batches, and inspect states of these resources to enforce sequencing. At a high level, this HIP proposes the following - Ability for chart authors to specify how to sequence deployment of **resources within a single chart** From 8f8f2aad0e5986ffaf07e96648b7596b5a72bfed Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Thu, 15 May 2025 16:58:53 +0100 Subject: [PATCH 30/40] Update hips/hip-00xx-sequencing.md Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 8675c445..01510ea9 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -28,7 +28,7 @@ Additionally, Helm currently doesn't provide a way to sequence when chart depend ## Rationale -This design was chosen due to simplicity and clarity of how it works for both the Helm developer and Application Operator +This design was chosen due to simplicity and clarity of how it works for both the Helm developer and Chart creator ## Specification From 707dd264f246fc19c7c66518cbf8331128d5aa65 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Thu, 15 May 2025 18:49:51 +0100 Subject: [PATCH 31/40] Clarify readiness evaluation behaviour Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 41 +++++++++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 8 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 01510ea9..5b34d3c3 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -22,7 +22,7 @@ The HIP only targets resources deployed in the Helm install phase. Resources dep The driving motivator here is to allow application distributors to control what order resources are bundled and sent to the K8s API server, referred to as resource sequencing for the rest of this HIP. -Today, to accomplish resource sequencing you have two options. The first is leveraging helm hooks, the second is building required sequencing into the application (via startup code or init containers). The existing hooks and weights can be tedious to build and maintain for application distributors, and built-in app sequencing can unecessarily increase complexity of a Helm application that needs to be maintained by application distributors. Helm as a package manager should be capable of enabling application distributors to properly sequence how their chart resources are deployed, and by providing such a mechanism to distributors, this will significantly improve the application distributor's experience. +Today, to accomplish resource sequencing, there are currently two main options: using Helm hooks, or building the sequencing logic into the application itself (e.g., using startup code or init containers). The existing hooks and weights can be tedious to build and maintain for application distributors, and built-in app sequencing can unecessarily increase complexity of a Helm application that needs to be maintained by application distributors. Helm as a package manager should be capable of enabling application distributors to properly sequence how their chart resources are deployed, and by providing such a mechanism to distributors, this will significantly improve the application distributor's experience. Additionally, Helm currently doesn't provide a way to sequence when chart dependencies are deployed and this featureset would ideally address this. @@ -128,22 +128,47 @@ This approach of building a directed acyclic graph (DAG) is prone to circular de ### Readiness -In order to enforce sequencing, Helm will check that resources are ready using `kstatus` [ready condition](https://github.com/kubernetes-sigs/cli-utils/blob/master/pkg/kstatus/README.md#the-ready-condition) allowing helm to proceed installing the next group of resources, or fail the install. Users can override checks using optional `helm.sh/readiness-failure` and `helm.sh/readiness-success` annotations which are jsonpath queries to check status fields of the annotated resource. If any of the checks in the lists evaluate as true, the readiness check is determined as successful/failed. `helm.sh/readiness-timeout` annotation can be used to override how long Helm should wait when performing readiness checks before raising an error. It defaults to `10s`. +To enforce sequencing, Helm determines whether resources are “ready” before deploying dependent resources. By default, Helm uses the [`kstatus`](https://github.com/kubernetes-sigs/cli-utils/blob/master/pkg/kstatus/README.md#the-ready-condition) library to assess readiness based on the resource’s type and `.status` field. + +Chart authors can optionally override this behavior using the following annotations: + +* `helm.sh/readiness-success`: A list of custom success conditions. If any are true, the resource is marked **ready**. +* `helm.sh/readiness-failure`: A list of custom failure conditions. If any are true, the resource is marked **failed**, which takes precedence over any success check. +* `helm.sh/readiness-timeout`: How long Helm should wait for the resource to become ready or fail. Defaults to `10s`. + +Both `helm.sh/readiness-success` and `helm.sh/readiness-failure` must be provided to override the default readiness logic. If only one is present, Helm will fall back to `kstatus` and emit a warning. Helm will also fail linting when only one of the two is defined, to prevent ambiguous readiness evaluation. + +#### JsonPath syntax + +The `readiness-success` and `readiness-failure` annotations accept lists of expressions with the format: + +``` +{} +``` + +Where: + +* `` is a [Kubernetes JSONPath](https://kubernetes.io/docs/reference/kubectl/jsonpath/) query scoped to `.status`. +* `` supports: `==`, `!=`, `<`, `<=`, `>`, `>=`. +* `` is the expected literal for comparison. The value should be a scalor (string, number, boolean). Object comparisons will not be supported. + +##### Example -#### Example ```yaml kind: Job metadata: - name: barz + name: db-init annotations: - helm.sh/readiness-success: ["succeeded==1"] # status fields - helm.sh/readiness-failure: ["failed==1"] # status fields - helm.sh/readiness-timeout: 20s # timeout + helm.sh/readiness-success: ["{.succeeded} == 1", "{.succeeded} == 2"] + helm.sh/readiness-failure: ["{.failed} >= 1"] + helm.sh/readiness-timeout: 30s status: succeeded: 1 ``` -A resources readiness is checked if there is a resource that depends on it as per the sequencing DAG. +In this case, Helm will consider the resource ready because `.status.succeeded == 1`. If `.status.failed >= 1` had been true, the Job would instead be marked as failed. + +A resources readiness is checked if there is a resource that depends on it as per the sequencing DAG, otherwise the checks are ignored. #### Sequencing order From 43713e6eee7a398482bc2f13820d904d1a9887b8 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Mon, 19 May 2025 22:12:27 +0100 Subject: [PATCH 32/40] chore: replace layer with resource-group Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 49 ++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 23 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 5b34d3c3..525a2555 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -34,23 +34,23 @@ This design was chosen due to simplicity and clarity of how it works for both th At a high level, allow Chart Developers to assign named dependencies to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. -For Helm CLI, the `--wait=ordered` flag will enable sequencing where resources are applied in groups of layers. SDK users will also be able to enable sequencing by setting a `Wait` boolean flag. Without this flag being enabled, resources will all be applied all at once which is the same behaviour in Chart v2. +For Helm CLI, the `--wait=ordered` flag will enable sequencing where resources are applied in groups. SDK users will also be able to enable sequencing by setting a `WaitStrategy` field. By default, resources are all applied at once which is the same behaviour in Chart v2. Each release will store information of whether sequencing was used or not. This information is used when performing uninstalls and rollbacks. The following annotations would be added to enable this. *Additions to templates* -- `helm.sh/layer`: Annotation to declare a layer that a given resource belongs to. Any number of resources can belong to a layer. A resource can only belong to one layer. -- `helm.sh/depends-on/layers`: Annotation to declare layers that must exist and in a ready state before this resource can be deployed. Order of the layers has no significance +- `helm.sh/resource-group`: Annotation to declare a resource-group that a given resource belongs to. Any number of resources can belong to a group. A resource can only belong to one group. +- `helm.sh/depends-on/resource-groups`: Annotation to declare resource-groups that must exist and in a ready state before this resource can be deployed. Order of the resource-groups has no significance *Additions to Chart.yaml* - `helm.sh/depends-on/subcharts`: Annotation added to `Chart.yaml` to declare chart dependencies, by name or tags, that must be deployed fully and in a ready state before the chart can be deployed. For the dependency or subchart chart to be declared ready, all of its resources, with their sequencing order taken into consideration, would need to be deployed and declared ready. Order of the charts has no significance. - `depends-on`: A new field added to `Chart.yaml` `dependencies` fields that is meant to declare a list of subcharts, by name or tag, that need to be ready before the subchart in questions get installed. This will be used to create a dependency graph for subcharts. -The installation process would group resources in the same layer and send them to the K8s API Server in one bundle, and once all resources are "ready", the next layer would be installed. A layer would not be considered "ready" and the next layer installed until all resources in that layer are considered "ready". Readiness is described in a later section. A similar process would apply for upgrades. Uninstalls would function on the same layer order, but backwards, where a layer is not uninstalled until all layers that depend on it are first uninstalled. Upgrades would follow the same order as installation. +The installation process would group resources in the same group and send them to the K8s API Server in one bundle, and once all resources are "ready", the next group would be installed. A resource-group would not be considered "ready" and the next group installed until all resources in that group are considered "ready". Readiness is described in a later section. A similar process would apply for upgrades. Uninstalls would function on the same resource-group order, but in reverse, where a resource-group is not uninstalled until all resource-groups that depend on it are first uninstalled. Upgrades would follow the same order as installation. -`helm.sh/layer` and `helm.sh/depends-on/layers` ordering annotations are only used when ordering resources in the same chart. When it comes to resources across charts one needs to declare `helm.sh/depends-on/subcharts` or `depends-on` in Chart.yaml to allow Helm to reason how to order install/upgrade/uninstall subcharts. +The `helm.sh/resource-group` and `helm.sh/depends-on/resource-groups` annotations are used exclusively to control the ordering of resources **within a single chart**. These annotations have no effect across chart boundaries and cannot be used to influence the order of resources in other charts or subcharts. #### Template examples: ```yaml @@ -58,39 +58,42 @@ The installation process would group resources in the same layer and send them t metadata: name: db-service annotations: - helm.sh/layer: database + helm.sh/resource-group: database --- # resource 2 metadata: name: my-app annotations: - helm.sh/layer: app - helm.sh/depends-on/layers: ["database, queue"] + helm.sh/resource-group: app + helm.sh/depends-on/resource-groups: ["database", "queue"] --- # resource 3 metadata: name: queue-processor annotations: - helm.sh/layer: queue - helm.sh/depends-on/layers: ["another-layer"] + helm.sh/resource-group: queue + helm.sh/depends-on/resource-groups: ["another-group"] + ``` -In this example, Helm would be responsible for resolving the annotations on these three resources and deploy all resources in the following order. Resources in `database` and `queue` layers would be deployed at the same time. They would need to be ready before attempting to deploy `app` layer resources: +In this example, Helm would be responsible for resolving the annotations on these three resources and deploy all resources in the following order. Resources in `database` and `queue` resource-groups would be deployed at the same time. They would need to be ready before attempting to deploy `app` resource-group: ``` -"database" layer: [db-service] +"database" group: [db-service] || -"queue" layer: || [queue-processor] +"queue" group: || [queue-processor] || || || || \/ \/ \\ // \\ // \/ \/ -"app" layer: [my-app] +"app" group: [my-app] ``` #### Chart dependencies example +To control the order in which subcharts are installed, upgraded, or uninstalled, chart authors must use the `helm.sh/depends-on/subcharts` annotation or the `depends-on` field in the `Chart.yaml`. These declarations enable Helm to determine the correct sequencing of subchart operations, as illustrated below. + ```yaml name: foo annotations: @@ -177,31 +180,31 @@ Resources with sequencing annotations in a chart would be deployed first followe - Installs: Helm will install resources in the order defined by the DAG. If any of the readiness checks fail or timeout, the entire install would fail and the release marked as failed. If `--atomic`, or its SDK equivalent is used, a rollback to the last successful install would take place. - Uninstalls: Helm would uninstall resources in the reverse order they were installed, as per the sequencing order. The logic to delete each resource will not change. - Rollbacks: Helm will check from the release object whether the revision being rolled back to, was installed in a sequenced manner. If it was, Helm will respect and enforce this order when installing resources from that revision. When deleting unneeded resources of the revision being rolled back from, the reverse order is followed just like uninstalls. -- `helm template` would print all resources in the order they would be deployed. Groups of resources in a layer would be delimited using a `## START layer: ` comment indicating the beginning of each layer and `END layer: `. +- `helm template` would print all resources in the order they would be deployed. Groups of resources in a resource-group would be delimited using a `## START resource-group: / ` comment indicating the beginning of each resource-group and `END resource-group: / `. ```yaml -## START layer: foo layer1 +## START resource-group: foo group1 # resource 1 metadata: name: foo annotations: - helm.sh/layer: layer1 + helm.sh/resource-group: group1 --- # resource 2 metadata: name: bar annotations: - helm.sh/layer: layer1 -## END layer: foo layer1 + helm.sh/resource-group: group1 +## END resource-group: foo group1 --- -## START layer: bar layer1 +## START resource-group: foo/bar group2 # resource 3 metadata: name: fizz annotations: - helm.sh/layer: layer2 - helm.sh/depends-on/layers: "layer1" -## END layer: bar layer2 + helm.sh/resource-group: group2 + helm.sh/depends-on/resource-groups: ["group1"] +## END resource-group: foo/bar group2 ``` ## Backwards compatibility From 3fdfe070c1a0f4a6b3f280618036e184bc3b9f64 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Mon, 19 May 2025 22:17:09 +0100 Subject: [PATCH 33/40] chore: change tags to alias in depends-on Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 525a2555..db4944a9 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -45,8 +45,8 @@ The following annotations would be added to enable this. - `helm.sh/depends-on/resource-groups`: Annotation to declare resource-groups that must exist and in a ready state before this resource can be deployed. Order of the resource-groups has no significance *Additions to Chart.yaml* -- `helm.sh/depends-on/subcharts`: Annotation added to `Chart.yaml` to declare chart dependencies, by name or tags, that must be deployed fully and in a ready state before the chart can be deployed. For the dependency or subchart chart to be declared ready, all of its resources, with their sequencing order taken into consideration, would need to be deployed and declared ready. Order of the charts has no significance. -- `depends-on`: A new field added to `Chart.yaml` `dependencies` fields that is meant to declare a list of subcharts, by name or tag, that need to be ready before the subchart in questions get installed. This will be used to create a dependency graph for subcharts. +* `helm.sh/depends-on/subcharts`: An annotation added to `Chart.yaml` to specify chart dependencies—identified by their `name` or `alias`—that must be fully deployed and in a ready state before the current chart can be installed. A dependent chart is considered ready only when all of its resources, including any defined sequencing, have been successfully deployed and marked ready. The order in which dependencies are listed has no effect on execution. +- `depends-on`: A new field added to `Chart.yaml` `dependencies` fields that is meant to declare a list of subcharts, by `name` or `alias`, that need to be ready before the subchart in question get installed. This will be used to create a dependency graph for subcharts. The installation process would group resources in the same group and send them to the K8s API Server in one bundle, and once all resources are "ready", the next group would be installed. A resource-group would not be considered "ready" and the next group installed until all resources in that group are considered "ready". Readiness is described in a later section. A similar process would apply for upgrades. Uninstalls would function on the same resource-group order, but in reverse, where a resource-group is not uninstalled until all resource-groups that depend on it are first uninstalled. Upgrades would follow the same order as installation. From 05616838a6b2b85c14f6f2dcae9aa6286ee518da Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Mon, 19 May 2025 22:38:57 +0100 Subject: [PATCH 34/40] chore: introduce readiness-timeout to replace timeout annotations Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index db4944a9..5b130989 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -131,13 +131,12 @@ This approach of building a directed acyclic graph (DAG) is prone to circular de ### Readiness -To enforce sequencing, Helm determines whether resources are “ready” before deploying dependent resources. By default, Helm uses the [`kstatus`](https://github.com/kubernetes-sigs/cli-utils/blob/master/pkg/kstatus/README.md#the-ready-condition) library to assess readiness based on the resource’s type and `.status` field. +To enforce sequencing, Helm determines whether resources are “ready” before deploying dependent resources. By default, Helm uses [`kstatus`](https://github.com/kubernetes-sigs/cli-utils/blob/master/pkg/kstatus/README.md#the-ready-condition) library to assess readiness based on the resource’s type and `.status` field. Chart authors can optionally override this behavior using the following annotations: * `helm.sh/readiness-success`: A list of custom success conditions. If any are true, the resource is marked **ready**. * `helm.sh/readiness-failure`: A list of custom failure conditions. If any are true, the resource is marked **failed**, which takes precedence over any success check. -* `helm.sh/readiness-timeout`: How long Helm should wait for the resource to become ready or fail. Defaults to `10s`. Both `helm.sh/readiness-success` and `helm.sh/readiness-failure` must be provided to override the default readiness logic. If only one is present, Helm will fall back to `kstatus` and emit a warning. Helm will also fail linting when only one of the two is defined, to prevent ambiguous readiness evaluation. @@ -164,7 +163,6 @@ metadata: annotations: helm.sh/readiness-success: ["{.succeeded} == 1", "{.succeeded} == 2"] helm.sh/readiness-failure: ["{.failed} >= 1"] - helm.sh/readiness-timeout: 30s status: succeeded: 1 ``` @@ -173,6 +171,8 @@ In this case, Helm will consider the resource ready because `.status.succeeded = A resources readiness is checked if there is a resource that depends on it as per the sequencing DAG, otherwise the checks are ignored. +Helm will wait up to a default of **1 minute** for a resource to either succeed or fail. If the resource does not reach a success or failure state within this period, the operation will time out, causing the chart install or upgrade to fail. This timeout can be customized using the `--readiness-timeout` CLI flag or the `ReadinessTimeout` field in the SDK. However, the specified readiness timeout must not exceed the overall `--timeout` value, which defines the maximum duration allowed for the entire chart installation or upgrade process. + #### Sequencing order Resources with sequencing annotations in a chart would be deployed first followed by resources without. If the chart has a `helm.sh/depends-on/subcharts` annotation in the `Chart.yaml`, all resources of the defined subcharts would be deployed before deploying the main chart. If any sequencing annotations are defined in the subchart resources, Helm will enforce ordering of resources within. Sequencing of resources in a chart are sandboxed within the chart. Sequencing annotations will not affect resources in other charts. From 123a6c3f0ede5b5f974e627b0b68ae801fc13c14 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Mon, 19 May 2025 22:59:40 +0100 Subject: [PATCH 35/40] chore: grammar edit an a bit of restructure Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 5b130989..14bdf83b 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -9,7 +9,7 @@ status: "draft" ## Abstract -This HIP is to propose a new featureset in Chart v3, to provide chart authors ([Application Distributor](https://github.com/helm/community/blob/main/user-profiles.md#2-application-distributor) Helm user profile), a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes. Helm deploys all manifests at the same time. This HIP will propose a way for Helm to deploy manifests in batches, and inspect states of these resources to enforce sequencing. +This HIP proposes a new feature set in Chart v3 to provide [Application Distributors](https://github.com/helm/community/blob/main/user-profiles.md#2-application-distributor)—a key Helm user profile—with a first-class mechanism for defining the deployment order of chart resources and subcharts. By default, Helm applies all rendered manifests simultaneously. This HIP introduces the ability for Helm to deploy resources in ordered batches and evaluate their readiness before proceeding. At a high level, this HIP proposes the following - Ability for chart authors to specify how to sequence deployment of **resources within a single chart** @@ -22,13 +22,13 @@ The HIP only targets resources deployed in the Helm install phase. Resources dep The driving motivator here is to allow application distributors to control what order resources are bundled and sent to the K8s API server, referred to as resource sequencing for the rest of this HIP. -Today, to accomplish resource sequencing, there are currently two main options: using Helm hooks, or building the sequencing logic into the application itself (e.g., using startup code or init containers). The existing hooks and weights can be tedious to build and maintain for application distributors, and built-in app sequencing can unecessarily increase complexity of a Helm application that needs to be maintained by application distributors. Helm as a package manager should be capable of enabling application distributors to properly sequence how their chart resources are deployed, and by providing such a mechanism to distributors, this will significantly improve the application distributor's experience. +Today, to accomplish resource sequencing, there are currently two main options: using Helm hooks, or building the sequencing logic into the application itself (e.g., using startup code or init containers). The existing hooks and weights can be tedious to build and maintain for application distributors, and built-in app sequencing can unnecessarily increase complexity of a Helm application that needs to be maintained by application distributors. Helm, as a package manager, should provide built-in mechanisms for sequencing resource deployment, reducing reliance on complex hooks or in-application logic. This will significantly improve the application distributor's experience. Additionally, Helm currently doesn't provide a way to sequence when chart dependencies are deployed and this featureset would ideally address this. ## Rationale -This design was chosen due to simplicity and clarity of how it works for both the Helm developer and Chart creator +The proposed design prioritizes simplicity and ease of use for both Helm developers and chart authors. It leverages familiar YAML patterns and Helm conventions, avoiding heavyweight solutions while offering powerful orchestration capabilities. ## Specification @@ -42,7 +42,9 @@ The following annotations would be added to enable this. *Additions to templates* - `helm.sh/resource-group`: Annotation to declare a resource-group that a given resource belongs to. Any number of resources can belong to a group. A resource can only belong to one group. -- `helm.sh/depends-on/resource-groups`: Annotation to declare resource-groups that must exist and in a ready state before this resource can be deployed. Order of the resource-groups has no significance +- `helm.sh/depends-on/resource-groups`: Annotation to declare resource-groups that must exist and in a ready state before this resource can be deployed. The order in which they are listed does not affect deployment sequencing. + +These annotations are only used for sequencing resources within the same chart. They do not influence or interact with resources across charts or subcharts. *Additions to Chart.yaml* * `helm.sh/depends-on/subcharts`: An annotation added to `Chart.yaml` to specify chart dependencies—identified by their `name` or `alias`—that must be fully deployed and in a ready state before the current chart can be installed. A dependent chart is considered ready only when all of its resources, including any defined sequencing, have been successfully deployed and marked ready. The order in which dependencies are listed has no effect on execution. @@ -50,8 +52,6 @@ The following annotations would be added to enable this. The installation process would group resources in the same group and send them to the K8s API Server in one bundle, and once all resources are "ready", the next group would be installed. A resource-group would not be considered "ready" and the next group installed until all resources in that group are considered "ready". Readiness is described in a later section. A similar process would apply for upgrades. Uninstalls would function on the same resource-group order, but in reverse, where a resource-group is not uninstalled until all resource-groups that depend on it are first uninstalled. Upgrades would follow the same order as installation. -The `helm.sh/resource-group` and `helm.sh/depends-on/resource-groups` annotations are used exclusively to control the ordering of resources **within a single chart**. These annotations have no effect across chart boundaries and cannot be used to influence the order of resources in other charts or subcharts. - #### Template examples: ```yaml # resource 1 @@ -138,7 +138,7 @@ Chart authors can optionally override this behavior using the following annotati * `helm.sh/readiness-success`: A list of custom success conditions. If any are true, the resource is marked **ready**. * `helm.sh/readiness-failure`: A list of custom failure conditions. If any are true, the resource is marked **failed**, which takes precedence over any success check. -Both `helm.sh/readiness-success` and `helm.sh/readiness-failure` must be provided to override the default readiness logic. If only one is present, Helm will fall back to `kstatus` and emit a warning. Helm will also fail linting when only one of the two is defined, to prevent ambiguous readiness evaluation. +Both `helm.sh/readiness-success` and `helm.sh/readiness-failure` must both be provided to override the default readiness logic. If only one is present, Helm will fall back to `kstatus` and emit a warning. Helm will also fail linting when only one of the two is defined, to prevent ambiguous readiness evaluation. #### JsonPath syntax From 4971d775b95efd7d68a151be6a53b7da05d1f700 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Mon, 19 May 2025 23:35:18 +0100 Subject: [PATCH 36/40] chore: add section on sequencing execution flow Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 14bdf83b..c26060cf 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -38,9 +38,43 @@ For Helm CLI, the `--wait=ordered` flag will enable sequencing where resources a Each release will store information of whether sequencing was used or not. This information is used when performing uninstalls and rollbacks. -The following annotations would be added to enable this. +### Sequencing Execution Flow + +When sequencing is enabled, Helm installs resources in a structured order across both subcharts and resource-groups: + +**1. Subchart Ordering** + +Helm builds a dependency graph from: + +* `helm.sh/depends-on/subcharts` (in `Chart.yaml`) +* `depends-on` fields in subchart entries + +Subcharts are installed in dependency order. Each subchart must be fully deployed and ready before its dependents begin. + +**2. Resource-Group Sequencing (Per Chart)** + +Within each chart (parent and subcharts), Helm builds a resource-group graph using: + +* `helm.sh/resource-group` +* `helm.sh/depends-on/resource-groups` + +Resources in each group are deployed together, and Helm waits for all to be ready before continuing to the next group. + +**3. Unsequenced Resources** + +Resources that: + +* lack annotations, +* depend on non-existent groups, or +* belong to isolated groups + +will be deployed after all properly sequenced groups have been processed. + +If a resource includes sequencing annotations but falls into this unsequenced category due to misconfiguration (e.g., referencing missing groups), Helm will emit a warning to alert the user of the potential issue. *Additions to templates* + +The following annotations would be added to enable this. - `helm.sh/resource-group`: Annotation to declare a resource-group that a given resource belongs to. Any number of resources can belong to a group. A resource can only belong to one group. - `helm.sh/depends-on/resource-groups`: Annotation to declare resource-groups that must exist and in a ready state before this resource can be deployed. The order in which they are listed does not affect deployment sequencing. From ccc7213a59f9c8096d5403b6313c2100c397c65c Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Tue, 20 May 2025 18:11:14 +0100 Subject: [PATCH 37/40] Update hips/hip-00xx-sequencing.md Co-authored-by: Scott Rigby Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index c26060cf..45ad482f 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -81,7 +81,7 @@ The following annotations would be added to enable this. These annotations are only used for sequencing resources within the same chart. They do not influence or interact with resources across charts or subcharts. *Additions to Chart.yaml* -* `helm.sh/depends-on/subcharts`: An annotation added to `Chart.yaml` to specify chart dependencies—identified by their `name` or `alias`—that must be fully deployed and in a ready state before the current chart can be installed. A dependent chart is considered ready only when all of its resources, including any defined sequencing, have been successfully deployed and marked ready. The order in which dependencies are listed has no effect on execution. +* `helm.sh/depends-on/subcharts`: An annotation added to `Chart.yaml` to specify chart dependencies—identified by their `name` or `alias`—that must be fully deployed and in a ready state before the current chart resources can be installed. A dependent chart is considered ready only when all of its resources, including any defined sequencing, have been successfully deployed and marked ready. The order in which dependencies are listed has no effect on execution. - `depends-on`: A new field added to `Chart.yaml` `dependencies` fields that is meant to declare a list of subcharts, by `name` or `alias`, that need to be ready before the subchart in question get installed. This will be used to create a dependency graph for subcharts. The installation process would group resources in the same group and send them to the K8s API Server in one bundle, and once all resources are "ready", the next group would be installed. A resource-group would not be considered "ready" and the next group installed until all resources in that group are considered "ready". Readiness is described in a later section. A similar process would apply for upgrades. Uninstalls would function on the same resource-group order, but in reverse, where a resource-group is not uninstalled until all resource-groups that depend on it are first uninstalled. Upgrades would follow the same order as installation. From 4414576b61f3fdb3284fe3b6b9d0593c25de6719 Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Tue, 20 May 2025 18:11:24 +0100 Subject: [PATCH 38/40] Update hips/hip-00xx-sequencing.md Co-authored-by: Scott Rigby Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index 45ad482f..b299b20e 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -207,7 +207,7 @@ A resources readiness is checked if there is a resource that depends on it as pe Helm will wait up to a default of **1 minute** for a resource to either succeed or fail. If the resource does not reach a success or failure state within this period, the operation will time out, causing the chart install or upgrade to fail. This timeout can be customized using the `--readiness-timeout` CLI flag or the `ReadinessTimeout` field in the SDK. However, the specified readiness timeout must not exceed the overall `--timeout` value, which defines the maximum duration allowed for the entire chart installation or upgrade process. -#### Sequencing order +### Sequencing order Resources with sequencing annotations in a chart would be deployed first followed by resources without. If the chart has a `helm.sh/depends-on/subcharts` annotation in the `Chart.yaml`, all resources of the defined subcharts would be deployed before deploying the main chart. If any sequencing annotations are defined in the subchart resources, Helm will enforce ordering of resources within. Sequencing of resources in a chart are sandboxed within the chart. Sequencing annotations will not affect resources in other charts. From 1c85d16c1596f5011a7fe0d134678237f53034af Mon Sep 17 00:00:00 2001 From: Evans Mungai Date: Tue, 20 May 2025 18:11:51 +0100 Subject: [PATCH 39/40] Update hips/hip-00xx-sequencing.md Co-authored-by: Scott Rigby Signed-off-by: Evans Mungai --- hips/hip-00xx-sequencing.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-00xx-sequencing.md index b299b20e..5cfbe5f1 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-00xx-sequencing.md @@ -44,10 +44,10 @@ When sequencing is enabled, Helm installs resources in a structured order across **1. Subchart Ordering** -Helm builds a dependency graph from: +Helm builds a dependency graph from definitions in `Chart.yaml`: -* `helm.sh/depends-on/subcharts` (in `Chart.yaml`) -* `depends-on` fields in subchart entries +* `helm.sh/depends-on/subcharts` key in `annotations` field +* `depends-on` key on `dependencies` list entries Subcharts are installed in dependency order. Each subchart must be fully deployed and ready before its dependents begin. From bc1007198b9dac5eeb9f10a66021741cf78b5ecf Mon Sep 17 00:00:00 2001 From: Scott Rigby Date: Wed, 21 May 2025 01:27:45 -0400 Subject: [PATCH 40/40] Assign number, update date, add to README Signed-off-by: Scott Rigby --- hips/README.md | 3 ++- hips/{hip-00xx-sequencing.md => hip-0025.md} | 4 ++-- 2 files changed, 4 insertions(+), 3 deletions(-) rename hips/{hip-00xx-sequencing.md => hip-0025.md} (99%) diff --git a/hips/README.md b/hips/README.md index 96abdd42..697db237 100644 --- a/hips/README.md +++ b/hips/README.md @@ -32,4 +32,5 @@ restricted markdown format and can be found in the - [hip-0021: Enhanced logging library for Helm](hip-0021.md) - [hip-0022: Wait With kstatus](hip-0022.md) - [hip-0023: Utilize Server Side Apply for installs/upgrades](hip-0023.md) -- [hip:0024: Improve the Helm Documentation](hip-0024.md) +- [hip-0024: Improve the Helm Documentation](hip-0024.md) +- [hip-0025: Better Support for Resource Creation Sequencing](hip-0025.md) diff --git a/hips/hip-00xx-sequencing.md b/hips/hip-0025.md similarity index 99% rename from hips/hip-00xx-sequencing.md rename to hips/hip-0025.md index 5cfbe5f1..3c8d908f 100644 --- a/hips/hip-00xx-sequencing.md +++ b/hips/hip-0025.md @@ -1,8 +1,8 @@ --- -hip: 9999 +hip: "0025" title: "Better Support for Resource Creation Sequencing" authors: [ "Joe Beck ", "Evans Mungai " ] -created: "2024-11-15" +created: "2025-05-21" type: "feature" status: "draft" ---