Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"expires" field from credential helper causes re-invocation of helper too often #23429

Closed
nataliejameson opened this issue Aug 26, 2024 · 4 comments
Assignees
Labels
P2 We'll consider working on this in future. (Assignee optional) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug

Comments

@nataliejameson
Copy link

Description of the bug:

It looks like in #21429 bazel started respecting the "expires" field from the json of credential_helper. We noticed after upgrading to bazel 7 that this caused our helper to be invoked hundreds to thousands of times in a build. Removing the "expires" field from the emitted json led to only being called a few times during a build. Looking at that PR, it looks like the ordering of Duration.between()'s parameters here are incorrect. Testing locally, if I reversed those parameters, there were not a ton of invocations as we were seeing before.

Which category does this issue belong to?

Remote Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Add the following to .bazelrc

# build --credential_helper="%workspace%/cred_helper_no_expires.sh"
# build --credential_helper="%workspace%/cred_helper_expires.sh"
build --credential_helper_timeout=30s
build --credential_helper_cache_duration=5m

Have cred_helper_no_expires.sh have something like:

#!/bin/bash
date >> /tmp/bazel_invocations
echo '{"headers": {"Authorization": ["<some token here>"]}}'

Have cred_helper_expires.sh have something like:

#!/bin/bash
date >> /tmp/bazel_invocations
echo '{"expires": "2024-08-26T21:44:34+00:00", "headers": {"Authorization": ["<some token here>"]}}'

Run a clean, then run a build with both the expires and the no_expires helper, removing /tmp/bazel_invocations before each call, and then after the build do wc -l on that file. The one with an expires field should have been invoked many more times.

This fix appears to fix the behavior:

diff --git a/src/main/java/com/google/devtools/build/lib/authandtls/credentialhelper/CredentialCacheExpiry.java b/src/main/java/com/google/devtools/build/lib/authandtls/credentialhelper/CredentialCacheExpiry.java
index 2a8f385260..f09cdcb040 100644
--- a/src/main/java/com/google/devtools/build/lib/authandtls/credentialhelper/CredentialCacheExpiry.java
+++ b/src/main/java/com/google/devtools/build/lib/authandtls/credentialhelper/CredentialCacheExpiry.java
@@ -41,7 +41,7 @@ final class CredentialCacheExpiry implements Expiry<URI, GetCredentialsResponse>
     }

     var now = Instant.now();
-    return Duration.between(expires.get(), now);
+    return Duration.between(now, expires.get());
   }

   @Override

Which operating system are you running Bazel on?

macos, linux

What is the output of bazel info release?

release 7.2.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

@github-actions github-actions bot added the team-Remote-Exec Issues and PRs for the Execution (Remote) team label Aug 26, 2024
@coeuvre coeuvre added P2 We'll consider working on this in future. (Assignee optional) and removed untriaged labels Aug 27, 2024
@tjgq
Copy link
Contributor

tjgq commented Aug 27, 2024

Oof, that's embarrassing. Thanks for investigating! I'll submit the fix (and see if I can write a test for it).

@tjgq
Copy link
Contributor

tjgq commented Aug 27, 2024

@bazel-io flag 7.4.0

@bazel-io bazel-io added the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Aug 27, 2024
@iancha1992
Copy link
Member

@bazel-io fork 7.4.0

@bazel-io bazel-io removed the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Aug 27, 2024
bazel-io pushed a commit to bazel-io/bazel that referenced this issue Aug 28, 2024
Also use the supplied current time instead of calling Instant.now().

I don't know how to meaningfully test this; there's so little going on that the test would essentially mirror the implementation.

Fixes bazelbuild#23429.

PiperOrigin-RevId: 668351088
Change-Id: I12f1575e5280330c61361e4cf1b7d9f9231f16eb
github-merge-queue bot pushed a commit that referenced this issue Aug 28, 2024
)

Also use the supplied current time instead of calling Instant.now().

I don't know how to meaningfully test this; there's so little going on
that the test would essentially mirror the implementation.

Fixes #23429.

PiperOrigin-RevId: 668351088
Change-Id: I12f1575e5280330c61361e4cf1b7d9f9231f16eb

Commit
5209ce7

Co-authored-by: Googler <[email protected]>
@iancha1992
Copy link
Member

A fix for this issue has been included in Bazel 7.4.0 RC1. Please test out the release candidate and report any issues as soon as possible.
If you're using Bazelisk, you can point to the latest RC by setting USE_BAZEL_VERSION=7.4.0rc1. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P2 We'll consider working on this in future. (Assignee optional) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug
Projects
None yet
Development

No branches or pull requests

12 participants
@tjgq @coeuvre @nataliejameson @bazel-io @sgowroji @iancha1992 @satyanandak and others