H4HIP: Helm Sequencing Proposal #373

Jeb135 · 2024-12-02T16:56:20Z

proposal to add resource sequencing support to Helm for v4

…tallations and upgrades. Signed-off-by: Joseph Beck <[email protected]>

Signed-off-by: Joe Beck <[email protected]>

- Seuencing logic - Readiness logic Signed-off-by: Evans Mungai <[email protected]>

Signed-off-by: Evans Mungai <[email protected]>

scottrigby

Nice proposal 👏 Definitely has come up in conversations a lot. Added some suggestions, and questions.

scottrigby · 2024-12-02T19:18:43Z

hips/hip-00xx-sequencing.md

+
+## Abstract
+
+This HIP is to propose a new featureset in Helm 4 to provide Application developers, who create heam charts for their applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes.


Does this proposal also include sequencing for uninstall, or only install/upgrade? For example, waiting to uninstall a resource until all that depends on it have completely uninstalled?

Good point I didn't think about that. At least with uninstall it can just be done backwards to the installation, but upgrade would add some complexity that I'm not sure how to tackle.

I added a section to clarify that this would apply to all 3, installation/uninstall/upgrade

and rollback, I assume

hips/hip-00xx-sequencing.md

scottrigby · 2024-12-02T19:40:34Z

hips/hip-00xx-sequencing.md

+
+## Rationale
+
+### Proposal 1: Named Dependencies


Personally, I quite like the general direction of proposal 1—as a replacement of numeric weights entirely—over proposal 2, which extends that. My point of view is I would like to move the numeric weighting into the "alternatives considered" section. But would like to hear other maintainers and community feedback on this.

I have wanted to introduce a depends_on functionality for helm resources—as a dynamic alternative to weights—to allow users to specify a specific order they are applied (a similar concept is used by used by projects like terraform, flux, and docker-compose).

My main reasoning for this is numbered weights are static—you must know the entire list of how all resources should be ordered, and this often takes a lot of finagling and fussing to get it right.

While named dependencies are dynamic—creating a DAG on the fly, based only on when and if certain resources have an actual dependency on one or more other resources.

It might be good to note some of this in the proposal. I could sketch up some suggested text for this if you prefer, but would also be happy for you to take a stab at it in your HIP. What do you think?

Go ahead! I'll add you as a contributor, I appreciate the assistance.

I moved the weights proposal to rejected ideas and tried to capture some of the points you made here in why.

mattfarina

This is making me wonder if we need a new version of charts to handle the compatibility issues.

mattfarina · 2024-12-02T21:22:32Z

hips/hip-00xx-sequencing.md

+Today, to accomplish resource sequencing you have two options. The first is leveraging helm hooks, the second is building required sequencing into the application (via startup code or init containers). The existing hooks and weights can be tedious to build and maintain for Application developers, and built-in app sequencing can unecessarily increase complexity of a Helm application that needs to be maintained by Application Developers. Helm as a package manager should be capable of enabling developers to properly sequence how their applications are deployed, and by providing such a mechanism to developers, this will significantly improve the Application developer's experience.
+
+Additionally, Helm currently doesn't provide a way to sequence when chart dependencies are deployed, and this featureset would ideally address this.


When it comes to sequencing, I'm not sure what you mean is clear in the document. I could come to two different conclusions:

The ordering of resources when they are uploaded to the k8s API server in one push

An ordered set of communications to the k8s API server where set it sent and completed prior to the next one being sent

Can you please provide some more clarity on what you mean.

2 is what we're going for here. I'll make a note to try and make that more clear.

Clarified what "resource sequencing" meant in the HIP

mattfarina · 2024-12-02T21:32:00Z

hips/hip-00xx-sequencing.md

+
+Helm would scope each subchart layer annotation names using a delimiter e.g `someChartDependency#mylayer` to avoid any name collisions. This is an internal implementation detail rather than feature chart authors or operators would need to know.
+
+`helm template` would print all resources in the order they would be deployed. Groups of resources in a layer would be delimited using a `## Layer: <name>` comment indicate the beginning of each layer


The # at the front denotes a comment. I'm not sure the k8s yaml parser, that we currently use and is supported by the k8s project, provides a means to parse and use data in comments. This needs to be investigated before this possibility could be considered. See https://github.com/kubernetes-sigs/yaml

For reference, the ## Layer: <name> would be above/below # Source:. Maybe there is no need for 2 #. We can have it be # Layer: <name>.

helm template from an example chart

--- # Source: foo/templates/tests/test-connection.yaml apiVersion: v1 kind: Pod metadata:

Signed-off-by: Evans Mungai <[email protected]>

Co-authored-by: Scott Rigby <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

Signed-off-by: Evans Mungai <[email protected]>

…intentions on how/why this should be built Signed-off-by: Joe Beck <[email protected]>

Signed-off-by: Evans Mungai <[email protected]>

banjoh · 2024-12-13T19:39:38Z

I made some changes to allow a chart author to build a dependency of subcharts. Its based on #373 (comment) post I had made earlier

banjoh · 2024-12-13T19:44:32Z

This HIP will depend on #374 HIP

Signed-off-by: Evans Mungai <[email protected]>

…sources Signed-off-by: Evans Mungai <[email protected]>

joejulian · 2025-03-06T17:49:26Z

What happens when it breaks? How will partial successes be rolled back? What if a rollback fails a stage?

joejulian · 2025-03-06T22:55:26Z

What happens when it breaks? How will partial successes be rolled back? What if a rollback fails a stage?

Which also makes me think about storage. Will each stage need to be a release in order to manage failures and rollbacks? What will that do to storage requirements?

Signed-off-by: Evans Mungai <[email protected]>

scottrigby

LGTM 👍 All feedback has been addressed from Helm dev meetings and in comments on this PR, and the HIP is very clear and actionable. Can’t wait to see the progress unfold 🙂

Signed-off-by: Evans Mungai <[email protected]>

sabre1041

Very promising proposal. A few comments and changes requested

hips/hip-00xx-sequencing.md

sabre1041 · 2025-05-14T04:05:51Z

hips/hip-00xx-sequencing.md

+The following annotations would be added to enable this.
+
+*Additions to templates*
+- `helm.sh/layer`: Annotation to declare a layer that a given resource belongs to. Any number of resources can belong to a layer. A resource can only belong to one layer.


layer is not very descriptive into how its being used. End users may not understand how it can/should be used

layer is not very descriptive into how its being used. End users may not understand how it can/should be used

I'll rename this to resource-group. I think this is a clearer name. Its come up before.

I think resource-group is much more clear. It's self-explanatory, unlike "layer" 👍

@sabre1041 here are the renaming changes

43713e6

sabre1041 · 2025-05-14T04:46:47Z

hips/hip-00xx-sequencing.md

+- `helm.sh/depends-on/layers`: Annotation to declare layers that must exist and in a ready state before this resource can be deployed. Order of the layers has no significance
+
+*Additions to Chart.yaml*
+- `helm.sh/depends-on/subcharts`: Annotation added to `Chart.yaml` to declare chart dependencies, by name or tags, that must be deployed fully and in a ready state before the chart can be deployed. For the dependency or subchart chart to be declared ready, all of its resources, with their sequencing order taken into consideration, would need to be deployed and declared ready. Order of the charts has no significance.


How will the dependency of subcharts be handled as it relates to tags and name? is there a concern if there is an overlap between the two? should a distinction be made?

Good catch. I believe this ought to have been "name" or "alias". Alias would be preferred.

@sabre1041 here is the change

3fdfe07

Yes good note about alias. I believe this would have been addressed in the implementation but glad it's called out in the HIP.

hips/hip-00xx-sequencing.md

sabre1041 · 2025-05-14T07:19:35Z

hips/hip-00xx-sequencing.md

+
+### Readiness
+
+In order to enforce sequencing, Helm will check that resources are ready using `kstatus` [ready condition](https://github.com/kubernetes-sigs/cli-utils/blob/master/pkg/kstatus/README.md#the-ready-condition) allowing helm to proceed installing the next group of resources, or fail the install. Users can override checks using optional `helm.sh/readiness-failure` and `helm.sh/readiness-success` annotations which are jsonpath queries to check status fields of the annotated resource. If any of the checks in the lists evaluate as true, the readiness check is determined as successful/failed. `helm.sh/readiness-timeout` annotation can be used to override how long Helm should wait when performing readiness checks before bailing out. It defaults to `10s`.


Does there need to be some commentary/discussion regarding how to construct readiness-success and/or readiness-failure?

Does there need to be some commentary/discussion regarding how to construct readiness-success and/or readiness-failure?

@sabre1041 I updated the doc to better explain how readiness checks work. https://github.com/Jeb135/helm-community/blob/hip-sequencing/hips/hip-00xx-sequencing.md#readiness

hips/hip-00xx-sequencing.md

Co-authored-by: Andrew Block <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

Co-authored-by: Scott Rigby <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

Signed-off-by: Evans Mungai <[email protected]>

scottrigby

not a blocking review. some minor cleanup suggestions to close out this thread #373 (comment)

hips/hip-00xx-sequencing.md

scottrigby · 2025-05-20T15:54:59Z

hips/hip-00xx-sequencing.md

+
+In this example, helm will first install and wait for all resources of `nginx` and `rabbitmq` dependencies to be "ready" before attempting to install `bar` resources. Once all resources of `bar` are "ready" then and only then will `foo` chart resources be installed. `foo` would require `rabbitmq` to be ready but since the subchart resources would have been installed before `bar`, this requirement would have been fulfilled.
+
+This approach of building a directed acyclic graph (DAG) is prone to circular dependencies. During the templating phase, Helm will have logic to detect, and report any circular dependencies found in the chart templates. Helm will also provide a command to print the DAG for development and troubleshooting purposes.


I think this is good and clear.

hips/hip-00xx-sequencing.md

Co-authored-by: Scott Rigby <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

sabre1041

LGTM.

Thanks @scottrigby and @banjoh for not only the proposal, but the improvements and iterative enhancements

Signed-off-by: Scott Rigby <[email protected]>

scottrigby · 2025-05-21T05:29:16Z

All feedback has been addressed. Merging! Looking forward to seeing this implemented for Helm 4 🚀

Jeb135 and others added 7 commits November 15, 2024 16:29

Add WIP HIP for a proposal to add a sequencing featureset to Helm ins…

e805ed5

…tallations and upgrades. Signed-off-by: Joseph Beck <[email protected]>

cleanup sequencing hip and flesh out implementation proposals

3d754ea

Signed-off-by: Joe Beck <[email protected]>

Additional information

78075e2

- Seuencing logic - Readiness logic Signed-off-by: Evans Mungai <[email protected]>

Rename Annotations and some minor improvements

163c6bf

Signed-off-by: Evans Mungai <[email protected]>

Add prior open issues to HIP

a54be4d

Signed-off-by: Evans Mungai <[email protected]>

A few more changes

9fe061a

Signed-off-by: Evans Mungai <[email protected]>

Add one more wording change

de97e1c

Signed-off-by: Evans Mungai <[email protected]>

pull-request-size bot added the size/L label Dec 2, 2024

Jeb135 changed the title ~~Hip for Helm Sequencing Proposal~~ H4HIP: Helm Sequencing Proposal Dec 2, 2024

scottrigby reviewed Dec 2, 2024

View reviewed changes

mattfarina reviewed Dec 2, 2024

View reviewed changes

banjoh and others added 5 commits December 6, 2024 18:51

Make helm.sh/depends-on/ annotations plural

e83cb0a

Signed-off-by: Evans Mungai <[email protected]>

Update hips/hip-00xx-sequencing.md

6712256

Co-authored-by: Scott Rigby <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

Add --wait/--wait-for-job comment

778b4d1

Signed-off-by: Evans Mungai <[email protected]>

Updates from PR comments. Move proposal 2 to rejected ideas, clarify …

66b3861

…intentions on how/why this should be built Signed-off-by: Joe Beck <[email protected]>

Descibe how to handle subchart dependencies

c018bbb

Signed-off-by: Evans Mungai <[email protected]>

mattfarina mentioned this pull request Dec 14, 2024

Feature: Ordered Install/Upgrade #375

Open

banjoh added 2 commits December 17, 2024 19:11

Rename application developers to application distributors

4834758

Signed-off-by: Evans Mungai <[email protected]>

Explicitly state the separation of the ordering between charts and re…

11d56ea

…sources Signed-off-by: Evans Mungai <[email protected]>

chore: add documentation bullet

ea37905

Signed-off-by: Evans Mungai <[email protected]>

gjenkins8 mentioned this pull request Mar 21, 2025

Feature Request: Sync order for dependencies helm/helm#30579

Open

banjoh added 5 commits March 25, 2025 18:30

chore: clarify that hooks are not ordered

04b8604

Signed-off-by: Evans Mungai <[email protected]>

chore: some minor changes after self review

8a8a906

Signed-off-by: Evans Mungai <[email protected]>

chore: add how --wait flag behaves with sequencing

2289f35

Signed-off-by: Evans Mungai <[email protected]>

chore: rename charts to subcharts

bb2da55

Signed-off-by: Evans Mungai <[email protected]>

chore: add high level summary of added features in abstract section

1c9d5db

Signed-off-by: Evans Mungai <[email protected]>

banjoh force-pushed the hip-sequencing branch from 1e9f6b9 to 8082240 Compare April 30, 2025 13:38

scottrigby approved these changes May 5, 2025

View reviewed changes

Use ordered wait strategy to enable sequencing

cb474de

Signed-off-by: Evans Mungai <[email protected]>

sabre1041 suggested changes May 14, 2025

View reviewed changes

banjoh and others added 9 commits May 14, 2025 17:07

Update hips/hip-00xx-sequencing.md

30a72b8

Co-authored-by: Andrew Block <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

Update hips/hip-00xx-sequencing.md

be43171

Co-authored-by: Andrew Block <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

Update hips/hip-00xx-sequencing.md

bad5ace

Co-authored-by: Andrew Block <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

Update hips/hip-00xx-sequencing.md

b3ee02e

Co-authored-by: Andrew Block <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

Update hips/hip-00xx-sequencing.md

f22f65d

Co-authored-by: Andrew Block <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

Update hips/hip-00xx-sequencing.md

c6236f6

Co-authored-by: Andrew Block <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

Update hips/hip-00xx-sequencing.md

9324191

Co-authored-by: Scott Rigby <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

Update hips/hip-00xx-sequencing.md

8f8f2aa

Signed-off-by: Evans Mungai <[email protected]>

Clarify readiness evaluation behaviour

707dd26

Signed-off-by: Evans Mungai <[email protected]>

banjoh force-pushed the hip-sequencing branch from fd8f3fd to 707dd26 Compare May 15, 2025 17:50

chore: replace layer with resource-group

43713e6

Signed-off-by: Evans Mungai <[email protected]>

banjoh force-pushed the hip-sequencing branch from 2a551cf to 43713e6 Compare May 19, 2025 21:12

banjoh added 4 commits May 19, 2025 22:17

chore: change tags to alias in depends-on

3fdfe07

Signed-off-by: Evans Mungai <[email protected]>

chore: introduce readiness-timeout to replace timeout annotations

0561683

Signed-off-by: Evans Mungai <[email protected]>

chore: grammar edit an a bit of restructure

123a6c3

Signed-off-by: Evans Mungai <[email protected]>

chore: add section on sequencing execution flow

4971d77

Signed-off-by: Evans Mungai <[email protected]>

banjoh force-pushed the hip-sequencing branch from caf427c to 4971d77 Compare May 19, 2025 22:35

scottrigby reviewed May 20, 2025

View reviewed changes

banjoh and others added 3 commits May 20, 2025 18:11

Update hips/hip-00xx-sequencing.md

ccc7213

Co-authored-by: Scott Rigby <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

Update hips/hip-00xx-sequencing.md

4414576

Co-authored-by: Scott Rigby <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

Update hips/hip-00xx-sequencing.md

1c85d16

Co-authored-by: Scott Rigby <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

sabre1041 approved these changes May 21, 2025

View reviewed changes

scottrigby added 2 commits May 21, 2025 01:25

Merge remote-tracking branch 'upstream/main' into hip-sequencing

b5de03b

Assign number, update date, add to README

bc10071

Signed-off-by: Scott Rigby <[email protected]>

scottrigby merged commit 05eab5b into helm:main May 21, 2025
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

H4HIP: Helm Sequencing Proposal #373

H4HIP: Helm Sequencing Proposal #373

Jeb135 commented Dec 2, 2024

scottrigby left a comment

scottrigby Dec 2, 2024

Jeb135 Dec 3, 2024

Jeb135 Dec 10, 2024

joejulian Mar 6, 2025

scottrigby Dec 2, 2024

Jeb135 Dec 3, 2024

Jeb135 Dec 10, 2024

mattfarina left a comment

mattfarina Dec 2, 2024

Jeb135 Dec 3, 2024

Jeb135 Dec 10, 2024

mattfarina Dec 2, 2024

banjoh Dec 6, 2024 •

edited

Loading

banjoh commented Dec 13, 2024

banjoh commented Dec 13, 2024

joejulian commented Mar 6, 2025

joejulian commented Mar 6, 2025

scottrigby left a comment

sabre1041 left a comment

sabre1041 May 14, 2025

banjoh May 15, 2025

scottrigby May 19, 2025

banjoh May 19, 2025

sabre1041 May 14, 2025

banjoh May 19, 2025

banjoh May 19, 2025

scottrigby May 20, 2025 •

edited

Loading

sabre1041 May 14, 2025

banjoh May 15, 2025 •

edited

Loading

scottrigby left a comment

scottrigby May 20, 2025

sabre1041 left a comment

scottrigby commented May 21, 2025


		## Abstract

		This HIP is to propose a new featureset in Helm 4 to provide Application developers, who create heam charts for their applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes.

		Today, to accomplish resource sequencing you have two options. The first is leveraging helm hooks, the second is building required sequencing into the application (via startup code or init containers). The existing hooks and weights can be tedious to build and maintain for Application developers, and built-in app sequencing can unecessarily increase complexity of a Helm application that needs to be maintained by Application Developers. Helm as a package manager should be capable of enabling developers to properly sequence how their applications are deployed, and by providing such a mechanism to developers, this will significantly improve the Application developer's experience.

		Additionally, Helm currently doesn't provide a way to sequence when chart dependencies are deployed, and this featureset would ideally address this.


		Helm would scope each subchart layer annotation names using a delimiter e.g `someChartDependency#mylayer` to avoid any name collisions. This is an internal implementation detail rather than feature chart authors or operators would need to know.

		`helm template` would print all resources in the order they would be deployed. Groups of resources in a layer would be delimited using a `## Layer: <name>` comment indicate the beginning of each layer


		### Readiness

		In order to enforce sequencing, Helm will check that resources are ready using `kstatus` [ready condition](https://github.com/kubernetes-sigs/cli-utils/blob/master/pkg/kstatus/README.md#the-ready-condition) allowing helm to proceed installing the next group of resources, or fail the install. Users can override checks using optional `helm.sh/readiness-failure` and `helm.sh/readiness-success` annotations which are jsonpath queries to check status fields of the annotated resource. If any of the checks in the lists evaluate as true, the readiness check is determined as successful/failed. `helm.sh/readiness-timeout` annotation can be used to override how long Helm should wait when performing readiness checks before bailing out. It defaults to `10s`.


		In this example, helm will first install and wait for all resources of `nginx` and `rabbitmq` dependencies to be "ready" before attempting to install `bar` resources. Once all resources of `bar` are "ready" then and only then will `foo` chart resources be installed. `foo` would require `rabbitmq` to be ready but since the subchart resources would have been installed before `bar`, this requirement would have been fulfilled.

		This approach of building a directed acyclic graph (DAG) is prone to circular dependencies. During the templating phase, Helm will have logic to detect, and report any circular dependencies found in the chart templates. Helm will also provide a command to print the DAG for development and troubleshooting purposes.

H4HIP: Helm Sequencing Proposal #373

H4HIP: Helm Sequencing Proposal #373

Conversation

Jeb135 commented Dec 2, 2024

scottrigby left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattfarina left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

banjoh Dec 6, 2024 • edited Loading

Choose a reason for hiding this comment

banjoh commented Dec 13, 2024

banjoh commented Dec 13, 2024

joejulian commented Mar 6, 2025

joejulian commented Mar 6, 2025

scottrigby left a comment

Choose a reason for hiding this comment

sabre1041 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scottrigby May 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

banjoh May 15, 2025 • edited Loading

Choose a reason for hiding this comment

scottrigby left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sabre1041 left a comment

Choose a reason for hiding this comment

scottrigby commented May 21, 2025

banjoh Dec 6, 2024 •

edited

Loading

scottrigby May 20, 2025 •

edited

Loading

banjoh May 15, 2025 •

edited

Loading