Caching issues on rules_haskell CI

[aherrmann]

Describe the bug
Builds and tests on rules_haskell CI seem to be mostly missing the cache even when a near 100% cache hit rate would be expected.

To Reproduce
Not a reproduction as such, but two examples:

• PR Make haskell_repl work with remote_download_toplevel #1352
  CI https://buildkite.com/tweag-1/rules-haskell/builds/1465 and https://app.circleci.com/pipelines/github/tweag/rules_haskell/219/workflows/ae71ee59-ed0d-4d41-83c0-d44f587770c9/jobs/9301/steps
• PR Do not pass -i flags to REPL #1345
  CI https://buildkite.com/tweag-1/rules-haskell/builds/1467 and https://app.circleci.com/pipelines/github/tweag/rules_haskell/221/workflows/b65184ae-d9ec-407d-81ff-79d01cee4306/jobs/9305/steps
  Both these PR's changes should not invalidate most cached items. Yet, when mergify rebased on latest master before merge their CI runs built most targets from scratch instead of fetching them from cache.

Expected behavior
I would have expected CI runs on both examples above to mostly fetch from cache.

Environment

• OS name + version: See respective CI pipeline
• Bazel version: 2.1.0
• Version of the rules: See respective PR

Additional context
I have seen cache misses like these on rules_haskell CI for a while now, while not experiencing such cache misses on projects using rules_haskell.

Something that stands out on the buildkite CI runs is that the log is filled with error messages of the form

WARNING: Writing to Remote Cache:
403 Forbidden
<?xml version='1.0' encoding='UTF-8'?><Error><Code>RetentionPolicyNotMet</Code><Message>Object overwrite or deletion is not allowed due to retention policy</Message><Details>Object 'tweag-bazel-cache-buildkite/cas/a4f5b8cffb3889f90cb7fe44ed954eb9e62772a44c2430cafa28d3cd853df3b4' is subject to bucket's retention policy and cannot be deleted, overwritten or archived until 2020-06-20T19:44:29.358643482-07:00</Details></Error>

[Xophmeister]

Is this still relevant? AIUI, CI has moved to GitHub Actions approx. 1 year ago; there's been no activity on BuildKite or CircleCI in the meantime. Have cache misses been observed in Bazel's upstream BuildKite? The scheduled builds complete very quickly, so presumably not...

[aherrmann]

Agreed, this seems no longer relevant. I believe rules_haskell's own CI encounters cache misses with BuildBuddy, but, to my understanding (and corroborated by the discussion here) this is likely caused by cached artifacts being GC'ed on BuildBuddy's side.