Implement initial tool for publishing to crates.io #251

jdisanti · 2021-10-13T02:04:07Z

This is an initial implementation of a tool to publish the SDK to crates.io. It has two subcommands:

fix-manifests: This is to be run after importing the latest generated code from smithy-rs. It will add versions to all path dependencies for all crates.
publish: This will be run when ready to actually publish to crates.io. It figures out the correct order to publish the crates in, chunks them into batches, and runs them with some throttled concurrency. It currently doesn't retry on failure, but this can be added later.

I tested everything works correctly except for the final publish, which, can't be tested until we actually publish. Doing a dry-run publish succeeds for the first few crates that don't have any path dependencies.

Sample output of the publish command:

Oct 12 18:44:08.509  INFO publisher::subcommand::publish: Discovering crates to publish...
Oct 12 18:44:09.432  INFO publisher::subcommand::publish: Crates discovered.
Publish plan:
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/protocol-test-helpers"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/smithy-async"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/smithy-types"]: cargo publish --dry-run
  wait
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/smithy-eventstream"]: cargo publish --dry-run
  wait
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/aws-sigv4"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/aws-types"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/smithy-http"]: cargo publish --dry-run
  wait
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/smithy-http-tower"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/smithy-json"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/smithy-query"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/smithy-xml"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/aws-endpoint"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/aws-auth"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/aws-http"]: cargo publish --dry-run
  wait
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/smithy-client"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/aws-sig-auth"]: cargo publish --dry-run
  wait
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/aws-hyper"]: cargo publish --dry-run
  wait
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/sts"]: cargo publish --dry-run
  wait
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/aws-config"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/accessanalyzer"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/account"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/acm"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/acmpca"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/alexaforbusiness"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/amp"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/amplify"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/amplifybackend"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/apigateway"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/apigatewaymanagement"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/apigatewayv2"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/appconfig"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/appflow"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/appintegrations"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/applicationautoscaling"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/applicationcostprofiler"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/applicationdiscovery"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/applicationinsights"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/appmesh"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/apprunner"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/appstream"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/appsync"]: cargo publish --dry-run
  [in "/home/jdisanti/src/aws-sdk-rust/sdk/athena"]: cargo publish --dry-run

... snip ...

Continuing will publish to crates.io. Do you wish to continue? yes
Oct 12 18:44:10.765  INFO publisher::subcommand::publish: Executing `[in "/home/jdisanti/src/aws-sdk-rust/sdk/protocol-test-helpers"]: cargo publish --dry-run`...
Oct 12 18:44:10.766  INFO publisher::subcommand::publish: Executing `[in "/home/jdisanti/src/aws-sdk-rust/sdk/smithy-async"]: cargo publish --dry-run`...
Oct 12 18:44:10.766  INFO publisher::subcommand::publish: Executing `[in "/home/jdisanti/src/aws-sdk-rust/sdk/smithy-types"]: cargo publish --dry-run`...
Oct 12 18:44:44.273  INFO publisher::subcommand::publish: Success: `[in "/home/jdisanti/src/aws-sdk-rust/sdk/smithy-types"]: cargo publish --dry-run`
Oct 12 18:44:49.941  INFO publisher::subcommand::publish: Success: `[in "/home/jdisanti/src/aws-sdk-rust/sdk/smithy-async"]: cargo publish --dry-run`
Oct 12 18:44:52.607  INFO publisher::subcommand::publish: Success: `[in "/home/jdisanti/src/aws-sdk-rust/sdk/protocol-test-helpers"]: cargo publish --dry-run`
Oct 12 18:44:52.608  INFO publisher::subcommand::publish: Executing `[in "/home/jdisanti/src/aws-sdk-rust/sdk/smithy-eventstream"]: cargo publish --dry-run`...
Error: Cargo publish failed:
Plan: [in "/home/jdisanti/src/aws-sdk-rust/sdk/smithy-eventstream"]: cargo publish --dry-run
Status: exit status: 101
Stdout: 
Stderr:     Updating crates.io index
warning: manifest has no description, license, license-file, documentation, homepage or repository.
See https://doc.rust-lang.org/cargo/reference/manifest.html#package-metadata for more info.
   Packaging smithy-eventstream v0.1.0 (/home/jdisanti/src/aws-sdk-rust/sdk/smithy-eventstream)
   Verifying smithy-eventstream v0.1.0 (/home/jdisanti/src/aws-sdk-rust/sdk/smithy-eventstream)
error: failed to verify package tarball

Caused by:
  no matching package named `smithy-types` found
  location searched: registry `https://github.com/rust-lang/crates.io-index`
  perhaps you meant: smithy_types
  required by package `smithy-eventstream v0.1.0 (/home/jdisanti/src/aws-sdk-rust/sdk/target/package/smithy-eventstream-0.1.0)`

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Velfi · 2021-10-13T14:09:28Z

tools/publisher/src/main.rs

+pub const REPO_NAME: &'static str = "aws-sdk-rust";
+pub const REPO_CRATE_PATH: &'static str = "sdk";


noting the static lifetime is no longer required by the compiler, &str is fine

It's going to take me a long time to break that habit 😅

tools/publisher/src/subcommand/fix_manifests.rs

Velfi · 2021-10-13T14:20:26Z

tools/publisher/src/subcommand/publish.rs

+use tracing::info;
+
+const BACKOFF: Duration = Duration::from_millis(30);
+const MAX_CONCURRENCY: usize = 4;


The actual max is 30 afaict

rust-lang/crates.io#3229

Thanks for looking that up, that's good to know. I think I'm going to cap this at the number of physical CPUs since cargo publish seems to actually invoke rustc.

Velfi · 2021-10-13T14:21:12Z

tools/publisher/src/subcommand/publish.rs

+    info!("Discovering crates to publish...");
+    let repo = discover_repository(REPO_NAME, REPO_CRATE_PATH)?;
+    let batches = discover_package_batches(Fs::Real, &repo.crates_root).await?;
+    info!("Crates discovered.");


it'd be cool if this could print the number of runtime crates and number of sdk crates just as a sanity check helper

I like this idea! Added.

rcoh

LGTM! a really impressive piece of work, excited to see it in action!

rcoh · 2021-10-14T20:39:10Z

tools/publisher/src/cargo.rs

+#[derive(Default)]
+pub struct Cmd {
+    parts: Vec<String>,
+    working_dir: Option<PathBuf>,
+}


wow...is there no library that does this?

I started off using async-process, but it was more complicated than I wanted to access stdout/stderr with that.

rcoh · 2021-10-14T20:39:40Z

tools/publisher/src/cargo.rs

+    }
+
+    /// Runs the command asynchronously.
+    pub async fn spawn(mut self) -> Result<Output> {


any reason to do this async? Seems like we may want to block on these commands running

oh I see, we use a semaphore later. 👍🏻

rcoh · 2021-10-14T20:43:10Z

tools/publisher/src/package.rs

+
+/// Validates that all of the publishable crates use consistent version numbers
+/// across all of their local dependencies.
+fn validate_packages(packages: &Vec<Package>) -> Result<()> {


clippy won't let you do this, &[Package]

Oh, I should run clippy on this... and also add it to CI 😄

rcoh · 2021-10-14T20:44:09Z

tools/publisher/src/repo.rs

+ * SPDX-License-Identifier: Apache-2.0.
+ */
+
+//! Repository discovery.


maybe add the word "git" here and mention that it's on the local file system? wasn't clear to me what kind of repository this was talking about

rcoh · 2021-10-14T20:45:24Z

tools/publisher/src/sort.rs

+enum Error {
+    #[error("dependency cycle detected")]
+    DependencyCycle,
+}


fwiw, this should be impossible because cargo won't allow it

rcoh

LGTM

Implement initial tool for publishing to crates.io

98b0d80

jdisanti requested review from rcoh and Velfi October 13, 2021 02:04

Velfi reviewed Oct 13, 2021

View reviewed changes

Velfi approved these changes Oct 13, 2021

View reviewed changes

rcoh approved these changes Oct 14, 2021

View reviewed changes

jdisanti added 5 commits October 14, 2021 14:43

CR feedback

5222e2e

Fix clippy lints

c740a1d

Add tools to CI

cc155ad

Only run SDK CI when changing the SDK

97697bb

Revert "Only run SDK CI when changing the SDK"

7ef6cbe

rcoh approved these changes Oct 15, 2021

View reviewed changes

rcoh merged commit c5d6c8d into awslabs:main Oct 18, 2021

jdisanti deleted the publishing-tool branch October 19, 2021 17:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement initial tool for publishing to crates.io #251

Implement initial tool for publishing to crates.io #251

jdisanti commented Oct 13, 2021

Velfi Oct 13, 2021

jdisanti Oct 14, 2021

Velfi Oct 13, 2021

jdisanti Oct 14, 2021

Velfi Oct 13, 2021

jdisanti Oct 14, 2021

rcoh left a comment

rcoh Oct 14, 2021

jdisanti Oct 14, 2021

rcoh Oct 14, 2021

rcoh Oct 14, 2021

rcoh Oct 14, 2021

jdisanti Oct 14, 2021

rcoh Oct 14, 2021

rcoh Oct 14, 2021

rcoh left a comment

		pub const REPO_NAME: &'static str = "aws-sdk-rust";
		pub const REPO_CRATE_PATH: &'static str = "sdk";

Implement initial tool for publishing to crates.io #251

Implement initial tool for publishing to crates.io #251

Conversation

jdisanti commented Oct 13, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rcoh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rcoh left a comment

Choose a reason for hiding this comment