RFC300: Programmatic Toolkit #654

mrgrain · 2024-11-12T18:32:31Z

This is a request for comments about Programmatic Toolkit. See #300 for
additional details.

Rendered Version

APIs are signed off by @rix0rrr.

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache-2.0 license

text/0300-programmatic-toolkit.md

rix0rrr · 2024-11-21T16:04:12Z

text/0300-programmatic-toolkit.md

+});
+
+// Synth and store a cached CloudExecutable, this won't run synth again
+const cxCache = await cdk.synth(cx);


Is cxCache the same type as cx? Or implements the same interface?

Yes see "Actions - synth". I don't want the example to be polluted by types. Would it be more clear to use let here?

text/0300-programmatic-toolkit.md

rix0rrr · 2024-11-21T16:35:44Z

text/0300-programmatic-toolkit.md

+A detailed definition of what would constitute a breaking change is included in the readme above.
+
+To support integrations, we will need to publish a list of all messages.
+We will use static code analysis to automatically compile and publish this list without adding additional maintenance burden on to the team.


Any thoughts to how that would work, or TBD?

I was thinking we could use the TS compiler to detect all calls to certain functions/methods and extract it that way.
No POC though.

text/0300-programmatic-toolkit.md

guysqr · 2024-11-25T02:47:19Z

This is going to be a really useful capability that will enable many interesting and valuable use cases, including being able to bake CDK into an Electron app and compose, deploy and destroy CDK apps entirely in code. This will put the benefits of the CDK into the hands of a whole new audience - those who can't or won't install command line tools.

I think it would be good to remove any interdependency between this and the consuming app around SDK, so consider if rather than passing an SdkProvider we could just pass a SDKv3 credentials provider.

Also, one of the challenges I saw when trying to use the CDK programatically in the past is how it sends CFN outputs to STDOUT and STDERR - would be ideal if we wanted to consume the CFN stack creation events via CDK to have a way to tap into that feed.

iliapolo · 2024-11-25T05:15:21Z

text/0300-programmatic-toolkit.md

+* **API Bar Raiser**: @rix0rrr
+
+The [Programmatic Toolkit] allows integrators to build lifecycle management solutions for CDK applications.
+It provides them with full control over all output and allows to interweave actions with custom steps.


and allows to interweave actions with custom steps.

Would be nice to include a quick example of this right off the bat.

iliapolo · 2024-11-25T05:25:23Z

text/0300-programmatic-toolkit.md

+#### Messages and Requests
+
+The toolkit emits *messages* and *requests* to structure interactions.
+A *request* is a *message* that requires the receiver to respond, otherwise the toolkit will continue with a default response.


Having a hard time imagining what the CLI would require a response for, and what default response would be. Can you show an example?

iliapolo · 2024-11-25T05:29:18Z

text/0300-programmatic-toolkit.md

+interface IoMessage<T> {
+  time: string;
+  level: 'error' | 'warn' | 'info' | 'debug';
+  action: 'synth' | 'list' | 'deploy' ... ;


Are these meant to represent the top level CLI command the user invoked? Or can they be sub actions like wait-for-stack-stabilization?

Meant to be the invoked command, i.e. "what is the toolkit currently doing".

code would be used for sub actions.

At a certain level, won't this be annoying?

Either we need to pass the "current command" around to all helpers that might emit log messages.

Or we maintain a global "current command" variable in the CdkToolkit class, that we filter all log messages through just so it can attach the right one.

Well if it's that -- then that exact same bookkeeping could also be done by the CdkToolkit client itself, because it knows which method it called.

Which means we don't need to do that work.

At a certain level, won't this be annoying?

Yeah I can see that. I was hoping we can do something global here, but might not actually work if two commands are executed in parallel (in which case the "current command" approach doesn't work either.

The field exists for the receiver, i.e. the IoHost to know what command an event is attached to. If we cannot find a good solution we might have to punt it.

More generically, should we add a clientReference?: T that gets passed around and emitted, and the consumer can stick whatever they want in there?

iliapolo · 2024-11-25T05:41:01Z

text/0300-programmatic-toolkit.md

+#### IoHost
+
+Message receivers need to implement the `IIoHost` interface.
+The `IoHost` is responsible for handling all logging and interactions:


The naming here is throwing me off. IoHost seems to suggest it responds to I/O events like logging or network calls. But actually, it can respond to any event that we decide can be responded to...is that right? If so, is EventHost a more appropriate name?

iliapolo · 2024-11-25T05:42:52Z

text/0300-programmatic-toolkit.md

+This implementation will do nothing for messages and requests that don't have a registered listener, i.e. the default response will be used as required.
+
+```ts
+const io = new IoEmitter();


This implements IoHost and dispatches to event listeners?

iliapolo · 2024-11-25T07:14:20Z

text/0300-programmatic-toolkit.md

+#### Readme
+
+The `AwsSdkProvider` defines how the toolkit interacts with AWS services.
+A default provider is available.
+You can provide a custom class implementing the `IAwsSdkProvider` interface.
+A standard `SdkProviderForCdk` is available that includes all features of the AWS CDK CLI [TODO: this might end up in a separate package].


I don't follow. Is this part of the previous section? Also it doesn't sound like an alternate solution, but rather a possible future extension?

It's written as if it was part of the README.
The currently proposed alternative is to not offer this as configuration.

text/0300-programmatic-toolkit.md

iliapolo · 2024-11-25T07:29:05Z

text/0300-programmatic-toolkit.md

+It also implies refactoring of existing interfaces to match the new API design.
+Both of these things are prone to human errors.
+
+Mitigation for this is test coverage and separating out PRs that are purely moving code, from actual changes.


We probably want to coordinate rolling this out only after we address the coverage gaps we detected.

text/0300-programmatic-toolkit.md

iliapolo · 2024-11-25T07:38:51Z

text/0300-programmatic-toolkit.md

+| FR10 | Events                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
+|      | In extension to logging focused events, the CDK should emit events for API operations including progress and state transitions (e.g. for deployment: synth complete, assets built, assets published, Hotswap/CFN deployment started ...). If available, these events should include additional data like the arn of the stack or a progress percentage. An integrator must be able to execute blocking code on these events. This will allow them to e.g. schedule pre-build scripts before synth starts or to update a service registry after deployment completed.                                                                                                                             |
+| FR11 | Event hooks                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |


Can you map these requirements to specific use-cases? that is, specific actions we know customers want to take inside events.

This will help us understand if the use case can be addressed in some other way.

misterjoshua · 2024-11-25T18:27:05Z

text/0300-programmatic-toolkit.md

+const deployment = await cdk.deploy(cxCache, { stacks: ['MyTestStack'] });
+
+// Returns a list of ARNs of the deployment
+return deployment.listResources().filter(r => r.arn);


Does this method also allow access to resource attributes like AWS::Lambda::Url's FunctionUrl? I could see this being very helpful for automated testing.

Unfortunately I don't think that's easily possible as CFN stack APIs don't include this information. You'd need to use a Stack output or make the request to the service API manually. But at least that should be much easier.

Stack outputs would work fine, but how would accessing those values work under this proposal?

For example, deployment.listOutputs() to get the outputs? Or perhaps deployment.describe() to wrap calls to DescribeStacks for the descriptions of all the deployed stacks?

misterjoshua · 2024-11-25T18:31:48Z

text/0300-programmatic-toolkit.md

+const deployment = await cdk.deploy(cxCache, { stacks: ['MyTestStack'] });
+
+// Returns a list of ARNs of the deployment
+return deployment.listResources().filter(r => r.arn);


Is there the possibility of filtering resources here by node path? This might make it easier to locate specific resources for a system under test.

This is a great suggestion! Might not be in the first version, but should be easy to add.

corymhall

Thanks for all the work on this toolkit, it's been super useful in the pulumi-cdk project!

corymhall · 2024-12-16T13:35:33Z

text/0300-programmatic-toolkit.md

+> Instead we might want to consider a more focused approach like an interface for authentication config.
+> This is still in scope for phase 3, but we will run a separate RFC and design for it.
+
+The main purpose of this feature is to expose SDK configuration to an integrator.


This is something that I started exploring for the cdk-diff-action. Users expect tools to have the same authentication flow and behavior as the CDK CLI and it is difficult to try and replicate all of that yourself.

corymhall · 2024-12-16T13:53:34Z

text/0300-programmatic-toolkit.md

+      const app = new cdk.App({ context });
+      const stack = new cdk.Stack(app);
+      return app.synth();
+    } catch(e: unknown) {


Will synth also have error types or will they all be unknown? I think there are still some differences in errors caused by user code and some caused by the toolkit, e.g. context lookups needing to be performed when lookups = false

They will all be typed. The e: unknown is a TypeScript quirk. I can fix the example to be more clear.

Context lookups will NOT be emitted from within this though. They are attached to the Toolkit and not the app.

corymhall · 2024-12-16T14:01:21Z

text/0300-programmatic-toolkit.md

+```ts
+interface IoMessage<T> {
+  time: string;
+  level: 'error' | 'warn' | 'info' | 'debug';


Will all logs regardless of level be sent, or will it still depend on the verbosity you set on the CLI commands? For example will you have to set debug: true in order to get debug logs?

All logs are send. The log level is a concern outside of the Toolkit.

corymhall · 2024-12-16T14:04:47Z

text/0300-programmatic-toolkit.md

+- adding data to a message that previously had no data
+- a backwards compatible change to the type definition of message data, e.g. adding a new property to an interface
+
+#### IoHost


I think this will be the most useful addition for the pulumi-cdk project. Full control over logging will be very useful!

rix0rrr · 2024-12-17T13:55:32Z

text/0300-programmatic-toolkit.md

+msg("Deploying Stack A")
+msg("Stack event for Stack A", event)
+msg("Stack event for Stack A", event)
+msg("Stack event for Stack A", event)
+msg("Stack A deployment completed", stackA)
+msg("Deploying Stack B")
+msg("Stack event for Stack B", event)
+msg("Stack event for Stack B", event)
+msg("Stack B deployment completed", stackB)


One detail that I'm sure we can work in later as well but I do want to call out: this is for the progress monitor, right?

In order to an in-place updating progress monitor we should have begin/end events for stacks, and also for the entire operation, so that we know how and when to update our progress bar.

Yes, that's a good point. The progress monitor will probably be inside the CliIoHost and we need to make sure these messages contain enough information.

rix0rrr · 2024-12-17T13:58:16Z

text/0300-programmatic-toolkit.md

+interface IoMessage<T> {
+  time: string;
+  level: 'error' | 'warn' | 'info' | 'debug';
+  action: 'synth' | 'list' | 'deploy' ... ;


At a certain level, won't this be annoying?

Either we need to pass the "current command" around to all helpers that might emit log messages.

Or we maintain a global "current command" variable in the CdkToolkit class, that we filter all log messages through just so it can attach the right one.

Well if it's that -- then that exact same bookkeeping could also be done by the CdkToolkit client itself, because it knows which method it called.

Which means we don't need to do that work.

rix0rrr · 2024-12-17T14:00:18Z

text/0300-programmatic-toolkit.md

+
+For every message, the toolkit calls `notify` or `requestResponse`, respectively.
+
+A generic `IoEmitter` is available to provide a familiar event emitter interface.


IoHost as specced is at least jsii compatible. .on and .once() less so, or they need to look very different than what JS slingers are used to.

rix0rrr · 2024-12-17T14:02:40Z

text/0300-programmatic-toolkit.md

+The `synth` action takes a Cloud Executable and synthesizes it in to a `CloudAssembly`.
+
+```ts
+const cx: ICloudExecutable =  await cdk.synth(cx, { stacks: ['MyTestStack'] });


I like what you did. The name is maybe just setting some expectations. ICloudAssemblySource ?

### Issue Closes #32568 ### Reason for this change Adding a private, empty package for the new programmatic toolkit. We will start adding preliminary types to this based on the [RFC](aws/aws-cdk-rfcs#654). ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

mrgrain force-pushed the mrgrain/0300-programmatic-toolkit branch from 9f9403f to 74205a4 Compare November 20, 2024 10:45

otaviomacedo reviewed Nov 21, 2024

View reviewed changes

text/0300-programmatic-toolkit.md Show resolved Hide resolved

text/0300-programmatic-toolkit.md Outdated Show resolved Hide resolved

rix0rrr reviewed Nov 21, 2024

View reviewed changes

sumupitchayan reviewed Nov 21, 2024

View reviewed changes

text/0300-programmatic-toolkit.md Outdated Show resolved Hide resolved

sumupitchayan reviewed Nov 21, 2024

View reviewed changes

text/0300-programmatic-toolkit.md Outdated Show resolved Hide resolved

mrgrain force-pushed the mrgrain/0300-programmatic-toolkit branch 4 times, most recently from 137b1af to 3f7072f Compare November 24, 2024 13:35

iliapolo reviewed Nov 25, 2024

View reviewed changes

misterjoshua reviewed Nov 25, 2024

View reviewed changes

mrgrain mentioned this pull request Dec 11, 2024

IoHost logger-only implementation aws/aws-cdk#32345

Open

HBobertz mentioned this pull request Dec 12, 2024

chore: Unconnected IoHost logger-only implementation aws/aws-cdk#32503

Open

1 task

feat(rfc-300): first draft

be14c40

mrgrain force-pushed the mrgrain/0300-programmatic-toolkit branch from 1cdfb52 to 3bf0435 Compare December 16, 2024 12:57

corymhall reviewed Dec 16, 2024

View reviewed changes

mrgrain force-pushed the mrgrain/0300-programmatic-toolkit branch 6 times, most recently from 92db7a9 to 150c091 Compare December 16, 2024 14:45

First review round.

17bc2d9

mrgrain force-pushed the mrgrain/0300-programmatic-toolkit branch from 150c091 to 17bc2d9 Compare December 16, 2024 16:32

rix0rrr approved these changes Dec 17, 2024

View reviewed changes

mrgrain added status/final-comment-period Pending final approval pr/do-not-merge Let mergify know not to auto merge labels Dec 17, 2024

mrgrain self-assigned this Dec 18, 2024

mrgrain mentioned this pull request Dec 19, 2024

chore(toolkit): add private stub package for toolkit aws/aws-cdk#32587

Merged

1 task


		For every message, the toolkit calls `notify` or `requestResponse`, respectively.

		A generic `IoEmitter` is available to provide a familiar event emitter interface.

RFC300: Programmatic Toolkit #654

Are you sure you want to change the base?

RFC300: Programmatic Toolkit #654

Conversation

mrgrain commented Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guysqr commented Nov 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrgrain Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

misterjoshua Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

corymhall left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrgrain commented Nov 12, 2024 •

edited

Loading

mrgrain Dec 16, 2024 •

edited

Loading

misterjoshua Nov 27, 2024 •

edited

Loading