Support multiple backends in a prioritized order

[brice-thomas2209]

Summary

Enhance the plugin to support multiple backends in a prioritized order. This allows checking a local cache first before falling-back to remote storage (e.g., S3), improving speed and efficiency by reducing unnecessary network requests.

Motivation

Currently, the plugin supports only a single backend at a time, which leads to inefficiencies depending on the chosen backend:

• Local backend limitation (fs): If the cache is stored only locally, every time a new Buildkite agent starts on a fresh machine, it must generate a completely new cache, incurring unnecessary compute time and slowing down builds.
• Remote backend limitation (s3): If the cache is stored only in a remote backend like S3, every agent has access to it, but retrieving the cache is significantly slower than accessing a local filesystem cache, increasing build times.

Neither of these options alone is ideal. The optimal approach is a hybrid model—checking for a local cache first and falling back to a remote backend if it's not found. This ensures:

• Fast access when possible: Agents reuse existing local caches without needing to fetch from S3.
• Persistent caching across agents: When a new agent starts, it can still retrieve the cache from S3 instead of regenerating it.
• Lower network costs: Local cache hits reduce the number of remote fetches.

By introducing multiple backends with fallback support, this proposal allows teams to get the best of both worlds: speed and persistence.

Proposed Implementation

Introduce support for specifying multiple backends in a prioritized order.

• The plugin will attempt to retrieve the cache from each backend sequentially.
• If a backend does not have the cache, it moves to the next backend.
• The first backend to return a valid cache will be used.
• If no backend has the cache, it proceeds as normal (e.g., generating a new cache).

Example Configuration

steps:
  - label: ':nodejs: Install dependencies'
    command: npm ci
    plugins:
      - cache#v1.4.0:
          manifest: package-lock.json
          path: node_modules
          restore: pipeline
          backend:
            - fs
            - s3
          save:
            - file
            - branch

• This would first check for the cache locally.
• If not found, it would attempt to retrieve it from S3.

Backward Compatibility

• The backend key will still support a single string value (backend: s3).
• If an array is provided, the plugin will use the first available backend in order.
• If only one or no backend is listed, the behavior remains unchanged.

Would love to hear thoughts from the maintainers and the community or if there is an existing pattern of achieving the above. Thanks 😀