stack update failing on CI with file returned by server too large

[avdv]

Describe the bug
When bazel is calling stack update, it is failing in the following way:

INFO: Repository rules_haskell_stack_update instantiated at:
  /home/runner/work/rules_haskell/rules_haskell/WORKSPACE:59:16: in <toplevel>
  /home/runner/work/rules_haskell/rules_haskell/extensions/rules_haskell_dependencies.bzl:23:38: in repositories
  /home/runner/work/rules_haskell/rules_haskell/tools/repositories.bzl:16:23: in rules_haskell_worker_dependencies
  /home/runner/work/rules_haskell/rules_haskell/haskell/cabal.bzl:2653:14: in stack_snapshot
  /home/runner/.cache/bazel/_bazel_runner/c91220d87f26118b19246641d5266d21/external/bazel_tools/tools/build_defs/repo/utils.bzl:233:18: in maybe
Repository rule _stack_update defined at:
  /home/runner/work/rules_haskell/rules_haskell/haskell/cabal.bzl:2322:32: in <toplevel>
Analyzing: 150 targets (84 packages loaded, 311 targets configured)
ERROR: An error occurred during the fetch of repository 'rules_haskell_stack_update':
   Traceback (most recent call last):
	File "/home/runner/work/rules_haskell/rules_haskell/haskell/cabal.bzl", line 2318, column 28, in _stack_update_impl
		_execute_or_fail_loudly(repository_ctx, [stack_cmd, "update"])
	File "/home/runner/work/rules_haskell/rules_haskell/haskell/private/workspace_utils.bzl", line 27, column 13, in execute_or_fail_loudly
		fail("\n".join(["Command failed: " + " ".join(arguments), exec_result.stderr]))
Error in fail: Command failed: /home/runner/.cache/bazel/_bazel_runner/c91220d87f26118b19246641d5266d21/external/rules_haskell_stack/stack update
Selected mirror https://hackage.haskell.org/
Downloading root
Waiting to acquire cache lock on /home/runner/.stack/pantry/hackage/hackage-security-lock
Acquired cache lock on /home/runner/.stack/pantry/hackage/hackage-security-lock
Released cache lock on /home/runner/.stack/pantry/hackage/hackage-security-lock
Selected mirror https://hackage.haskell.org/
Downloading timestamp
Downloading snapshot
Downloading mirrors
Cannot update index (no local copy)
Downloading index
file returned by server too large: <repo>/01-index.tar.gz (expected exactly 116396026 bytes)
ERROR: /home/runner/work/rules_haskell/rules_haskell/WORKSPACE:59:16: fetching _stack_update rule //external:rules_haskell_stack_update: Traceback (most recent call last):
	File "/home/runner/work/rules_haskell/rules_haskell/haskell/cabal.bzl", line 2318, column 28, in _stack_update_impl
		_execute_or_fail_loudly(repository_ctx, [stack_cmd, "update"])
	File "/home/runner/work/rules_haskell/rules_haskell/haskell/private/workspace_utils.bzl", line 27, column 13, in execute_or_fail_loudly
		fail("\n".join(["Command failed: " + " ".join(arguments), exec_result.stderr]))
Error in fail: Command failed: /home/runner/.cache/bazel/_bazel_runner/c91220d87f26118b19246641d5266d21/external/rules_haskell_stack/stack update
Selected mirror https://hackage.haskell.org/
Downloading root
Waiting to acquire cache lock on /home/runner/.stack/pantry/hackage/hackage-security-lock
Acquired cache lock on /home/runner/.stack/pantry/hackage/hackage-security-lock
Released cache lock on /home/runner/.stack/pantry/hackage/hackage-security-lock
Selected mirror https://hackage.haskell.org/
Downloading timestamp
Downloading snapshot
Downloading mirrors
Cannot update index (no local copy)
Downloading index
file returned by server too large: <repo>/01-index.tar.gz (expected exactly 116396026 bytes)
Analyzing: 150 targets (84 packages loaded, 311 targets configured)
ERROR: /home/runner/work/rules_haskell/rules_haskell/tests/BUILD.bazel:20:24: no such package '@stackage//': no such package '@rules_haskell_stack_update//': Command failed: /home/runner/.cache/bazel/_bazel_runner/c91220d87f26118b19246641d5266d21/external/rules_haskell_stack/stack update
Selected mirror https://hackage.haskell.org/
Downloading root
Waiting to acquire cache lock on /home/runner/.stack/pantry/hackage/hackage-security-lock
Acquired cache lock on /home/runner/.stack/pantry/hackage/hackage-security-lock
Released cache lock on /home/runner/.stack/pantry/hackage/hackage-security-lock
Selected mirror https://hackage.haskell.org/
Downloading timestamp
Downloading snapshot
Downloading mirrors
Cannot update index (no local copy)
Downloading index
file returned by server too large: <repo>/01-index.tar.gz (expected exactly 116396026 bytes)
 and referenced by '//tests:protobuf-toolchain-impl'
ERROR: Analysis of target '//tests:protobuf-toolchain-impl' failed; build aborted: Analysis failed
INFO: Elapsed time: 23.470s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (84 packages loaded, 311 targets configured)

See here, here and here.

To Reproduce
This occurs in our CI jobs quite frequently. It might actually be some kind of race condition or stale caches.

Not sure if we can do something about it, maybe a stack upgrade would help? Maybe we could retry the command a few times if it fails with that error?

Expected behavior
This should work more reliably.

Environment

• OS name + version: Github action runners, ubuntu-latest, windows-latest
• Bazel version: 6.2.0
• Version of the rules: -

Additional context
See haskell/hackage-server#1236 and commercialhaskell/stack#5770