The metadata output has gotten big for multilevel images in v5.6.0

[danielhollas]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• The documentation does not mention anything about my problem
• There are no open or closed issues that are related to my problem

Description

Hi 👋

The recently released version 5.6.0 broke our CI because the format of the metadata GHA output changed. I understand that this is probably caused by dependencies of this action, but I was not sure where else to open the issue --- happy to be redirected elsewhere.

Concretely, one of our steps started to fail since we've been parsing the metadata output and we were passing it via an environment variable like this:

  - name: Set output variables    
    run: |    
       .github/workflows/extract-image-names.sh | tee -a "${GITHUB_OUTPUT}" | awk -F'=' '{print $2}' | jq    
    env:    
        BAKE_METADATA: ${{ steps.build-upload.outputs.metadata }}

Because of the size, we started to hit this error:

Error: An error occurred trying to start process '/usr/bin/bash' with working directory '/home/runner/work/aiidalab-docker-stack/aiidalab-docker-stack'. Argument list too long

We've been able to workaround this by saving the metadata output into a file first like this

  - name: Set output image names     
    run: |    
        cat << EOF > bake_metadata.json    
        ${{ steps.build-upload.outputs.metadata }}    
        EOF    
        images=$(.github/workflows/extract-image-names.sh bake_metadata.json)    
        echo "images=${images}" >> "${GITHUB_OUTPUT}" 

Nevertheless, even with this workaround, it might be nice to try to prune the metadata output a bit to make it a bit more readable in the GHA logs. Specifically, there is now a section {"buildx.build.provenance": {"invocation": {...}} that is huge for multilevel builds.

Here's the output from the base image, nothing out of the ordinary

    "base": {
      "buildx.build.provenance": {
        "buildType": "https://mobyproject.org/buildkit@v1",
        "materials": [
          {
            "uri": "pkg:docker/jupyter/[email protected]?platform=linux%2Famd64",
            "digest": {
              "sha256": "f7022a83184c73b04fdb16d3f2f2b6b014c819ac086b87496f1c0641a0614f3a"
            }
          }
        ],
        "invocation": {
          "configSource": {
            "entryPoint": "Dockerfile"
          },
          "parameters": {
            "frontend": "dockerfile.v0",
            "args": {
              "build-arg:AIIDA_VERSION": "2.5.1",
              "build-arg:BASE": "jupyter/minimal-notebook:python-3.9.13"
            },
            "locals": [
              {
                "name": "context"
              },
              {
                "name": "dockerfile"
              }
            ]
          },
          "environment": {
            "platform": "linux/amd64"
          }
        }
      },
      "buildx.build.ref": "builder-df0f1d9d-0da6-4602-89bf-f0dd8e3e962d/builder-df0f1d9d-0da6-4602-89bf-f0dd8e3e962d0/2yfr5apv8yx146zzg287f66an",
      "containerimage.config.digest": "sha256:dc3491813cf2fa44b8ed0dc6a914b221a7d1668544193496a516de248224a29e",
      "containerimage.descriptor": {
        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
        "digest": "sha256:a4ecb05d8859237d79ff229f84d1835f9d639aba923c3e8292318c42763b7267",
        "size": 6356,
        "platform": {
          "architecture": "amd64",
          "os": "linux"
        }
      },
      "containerimage.digest": "sha256:a4ecb05d8859237d79ff229f84d1835f9d639aba923c3e8292318c42763b7267",
      "image.name": "ghcr.io/aiidalab/base"
    },

Here's the second part from the image built on top of base image

    "base-with-services": {
      "buildx.build.provenance": {
        "buildType": "https://mobyproject.org/buildkit@v1",
        "materials": [
          {
            "uri": "pkg:docker/docker/dockerfile@1",
            "digest": {
              "sha256": "fe40cf4e92cd0c467be2cfc30657a680ae2398318afd50b0c80585784c604f28"
            }
          },
          {
            "uri": "pkg:docker/jupyter/[email protected]?digest=sha256:f7022a83184c73b04fdb16d3f2f2b6b014c819ac086b87496f1c0641a0614f3a&platform=linux%2Famd64",
            "digest": {
              "sha256": "f7022a83184c73b04fdb16d3f2f2b6b014c819ac086b87496f1c0641a0614f3a"
            }
          }
        ],
        "invocation": {
          "configSource": {
            "entryPoint": "Dockerfile"
          },
          "parameters": {
            "frontend": "gateway.v0",
            "args": {
              "build-arg:AIIDA_VERSION": "2.5.1",
              "build-arg:PGSQL_VERSION": "15",
              "build-arg:RMQ_VERSION": "3.9.13",
              "cmdline": "docker/dockerfile:1",
              "context:base": "input:0-base",
              "frontend.caps": "moby.buildkit.frontend.contexts+forward",
              "input-metadata:0-base": "{\"containerimage.config\":\"eyJhcmNoaXRlY3R1cmUiOiJhbWQ2NCIsIm9zIjoibGludXgiLCJyb290ZnMiOnsidHlwZSI6ImxheWVycyIsImRpZmZfaWRzIjpbInNoYTI1NjoxN2Y2MjNhZjAxZTI3N2M1ZmZlNjc3OWFmODE2NDkwN2RlMDJkOWFmN2EwZTE2MTY2MmZjNzM1ZGQ2NGYxMTdiIiwic2hhMjU2OjIxMjU2Y2ZiZjVjZDc4MjNjNjk3OTNiZmZlMmEyMmZjNWFiN2U2Mzg0MzZiNzY3MTFjMmIwZTNkYjgwNjExMTUiLCJzaGEyNTY6NWVhNjgxOTRjM2JmMmRjM2U0YzIxMDFiMDdlMWEwMGMyYmI4MmEyOGUyZDk0ZTYwNzNiNTFhOTNhZGE0NzE3YiIsInNoYTI1Njo1ZjcwYmYxOGEwODYwMDcwMTZlOTQ4YjA0YWVkM2I4MjEwM2EzNmJlYTQxNzU1YjZjZGRmYWYxMGFjZTNjNmVmIiwic2hhMjU2OjRhZmE4YzMzY2QyYzZkYTJhNmEyMzllMTcxZGJjMTgyNWVmMzIyYWM3ZmRjMWUwZWFmZGZiZjYyYzkwMTEzNDIiLCJzaGEyNTY6MGNhMzhlZjY2ZGZkZDI2ODRjMjhiMGE4MTQ2YTc2MGQ3YTdhYmJhNmQzYWZlZGRmMjk4NGM4MWFiZWNjZGJiNCIsInNoYTI1NjpjNzZlZTJkMDU0YjBkZWVlZDgzOTc0ZGE5M2Q0ZDBjMzg2NDJmY2JmMGNiYjQ2NTRhZjAzNjUzNzE0MWNmOWFhIiwic2hhMjU2OjRiOGE0YzlkNjM1OTNmMDNjMWI4ZjU1ZGQ4MDE5OGM2YTc3NWQ4ZTFkNjg0MGUzNzRmZTNlMTZlZjg0Y2IxM2IiLCJzaGEyNTY6NWY3MGJmMThhMDg2MDA3MDE2ZTk0OGIwNGFlZDNiODIxMDNhMzZiZWE0MTc1NWI2Y2RkZmFmMTBhY2UzYzZlZiIsInNoYTI1Njo3NWM2OGY2YjA4YjJhNGU2NjMxYTA1MDMwZDA0NTliMjA5ZjZmODdiYTU4ZmI1NWExNzliM2MxYzI1ZWFjYTQ2Iiwic2

(I elided the full output)

It would be great if the "input-metadata:0-base": "{\"containerimage.config\": section was not there.

Repository URL

https://github.com/aiidalab/aiidalab-docker-stack

Workflow run URL

https://github.com/aiidalab/aiidalab-docker-stack/actions/runs/10283635135/job/28465139810

YAML workflow

- name: Build and upload to ghcr.io
                id: build-upload    
                uses: docker/[email protected]
                with:
                    push: true
                    # Using provenance to disable default attestation so it will build only desired images:
                    # https://github.com/orgs/community/discussions/45969
                    provenance: false
                    set: |
                        *.platform=${{ inputs.platforms }}
                        *.output=type=registry,push-by-digest=true,name-canonical=true
                        *.cache-to=type=gha,scope=${{ github.workflow }},mode=max
                        *.cache-from=type=gha,scope=${{ github.workflow }}
                    files: |
                        docker-bake.hcl
                        build.json
                        .github/workflows/env.hcl
  
              - name: Set output image names
                id: bake_metadata
                # bake-action metadata output has gotten too big, so we first write it
                # to a file. See https://github.com/aiidalab/aiidalab-docker-stack/issues/491

              - name: Set output variables    
                run: |    
                    .github/workflows/extract-image-names.sh | tee -a "${GITHUB_OUTPUT}" | awk -F'=' '{print $2}' | jq    
                env:    
                    BAKE_METADATA: ${{ steps.build-upload.outputs.metadata }}

Workflow logs

No response

BuildKit logs

No response

Additional info

No response

[crazy-max]

Thanks for reporting, what outputs are being created in Set output variables step?

Can you show what returns .github/workflows/extract-image-names.sh?

I think it would be better to rely on https://github.com/actions/github-script instead of shell for this use case.

Edit:

Thanks for reporting, what outputs are being created in Set output variables step?

Can you show what returns .github/workflows/extract-image-names.sh?

Nevermind found it: https://github.com/aiidalab/aiidalab-docker-stack/blob/4b5e560db34ded20f5173dba53ecfef247c45c74/.github/workflows/extract-image-names.sh#L39-L46

[crazy-max]

I think something like this could to the trick (not tested):

  - name: Set output variables    
    uses: actions/github-script@v7
    with:
      script: |
        for (const key in JSON.parse(process.env.BAKE_METADATA || '{}', 'utf-8')) {
          if (data.hasOwnProperty(key)) {
            const entry = data[key];
            core.setOutput(`${key.toUpperCase().replace(/-/g, "_")}_IMAGE`, `${entry["image.name"]}@${entry["containerimage.digest"]}`);
          }
        }
    env:
      BAKE_METADATA: ${{ steps.build-upload.outputs.metadata }}