Build fail with "parent snapshot does not exist"

[apostasie]

Description

Build will fail with

FATA[0000] apply layer error for "docker.io/library/test:latest": failed to prepare extraction snapshot "extract-810343375-eUiw sha256:2f8da9dda8e00a45de9a98ebd2dbc4394149baf8686dbda33d20899c797c9f5f": parent snapshot sha256:e644ff0c302d7de4c3e03dc718bc4b77ebeb324f7c8f5483b463fa230027b02c does not exist: not found

This is random and may take some time.

To repro:

while true; do nerdctl build --debug-full --no-cache -t test .; done

Steps to reproduce the issue

na

Describe the results you received and expected

Loaded image: docker.io/library/test:latest
DEBU[0000] stateDir: /run/user/501/containerd-rootless
DEBU[0000] RootlessKit detach-netns mode: true
DEBU[0000] rootless parent main: executing "/usr/bin/nsenter" with [-r/ -w/tmp --preserve-credentials -m -U -t 972422 -F nerdctl build --debug-full --no-cache -t test .]
DEBU[0000] Choosing the buildkit host "unix:///run/user/501/buildkit-default/buildkitd.sock", candidates=[unix:///run/user/501/buildkit-default/buildkitd.sock unix:///run/user/501/buildkit/buildkitd.sock]
DEBU[0000] Choosing the buildkit host "unix:///run/user/501/buildkit/buildkitd.sock", candidates=[unix:///run/user/501/buildkit-default/buildkitd.sock unix:///run/user/501/buildkit/buildkitd.sock]
DEBU[0000] Chosen buildkit host "unix:///run/user/501/buildkit/buildkitd.sock"
DEBU[0000] worker labels: map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:lima-v2 org.mobyproject.buildkit.worker.network:cni org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs]
DEBU[0000] running /usr/local/bin/buildctl [--addr=unix:///run/user/501/buildkit/buildkitd.sock build --progress=auto --frontend=dockerfile.v0 --local=context=. --output=type=docker,name=docker.io/library/test:latest --local=dockerfile=/tmp --opt=filename=Dockerfile --no-cache]
[+] Building 0.5s (6/6)
[+] Building 0.5s (6/6) FINISHED
 => [internal] load build definition from Dockerfile                                                                     0.0s
 => => transferring dockerfile: 124B                                                                                     0.0s
 => [internal] load metadata for docker.io/library/nginx:latest                                                          0.1s
 => [internal] load .dockerignore                                                                                        0.0s
 => => transferring context: 2B                                                                                          0.0s
 => CACHED [1/2] FROM docker.io/library/nginx:latest@sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9  0.0s
 => => resolve docker.io/library/nginx:latest@sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3    0.0s
 => [2/2] RUN echo 'Hello Nerdctl From Containerd' > /usr/share/nginx/html/index.html                                    0.0s
 => exporting to docker image format                                                                                     0.3s
 => => exporting layers                                                                                                  0.0s
 => => exporting manifest sha256:2cab31a5770a6e4b1b883e12003f7db13dbec31f1453ffd9c5cf40f595786007                        0.0s
 => => exporting config sha256:d17d7567c3198103f72ee65d06acb0b63a5bc524a9afcef847fd647fb6eeec83                          0.0s
 => => sending tarball                                                                                                   0.3s
FATA[0000] apply layer error for "docker.io/library/test:latest": failed to prepare extraction snapshot "extract-153144429-zNRp sha256:29099085e37cb571e3c6e6f499df323f4965d1bd7cd8139415fc03fedb121077": parent snapshot sha256:3b487b874707d138c99a5dc4c6ffdb9a19b45937998141fa43aefab26d607347 does not exist: not found
fail

What version of nerdctl are you using?

Client:
 Version:	v2.0.0-rc.1-132-g814418aa.m
 OS/Arch:	linux/arm64
 Git commit:	814418aa3becd4231ca39e400832bc86dc5481d1.m
 buildctl:
  Version:	v0.15.2
  GitCommit:	9e14164a1099d3e41b58fc879cbdd6f2b2edb04e

Server:
 containerd:
  Version:	v2.0.0-rc.3-211-g7b948faa0.m
  GitCommit:	7b948faa084506e72967d88b5d35423211f13b07.m
 runc:
  Version:	1.2.0-rc.3+dev
  GitCommit:	v1.2.0-rc.3-1-g961b8031

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

No response

[apostasie]

@AkihiroSuda suggesting we make this a blocker for 2.0.

This might be yet another variant of #827 #3415 and many others, which apparently would stem from:

containerd/containerd#8973

[AkihiroSuda]

Is this a recent regression?

Probably this should be rather reported to https://github.com/moby/buildkit/issues ?

[apostasie]

Is this a recent regression?

Probably this should be rather reported to https://github.com/moby/buildkit/issues ?

I will test with older versions soon to check that.

My hunch is that this is the same root issue in containerd that is causing the other issues with commit and push, but this is unverified yet.

i am not sure where the fix (or workaround) should go yet.

[apostasie]

The containerd bug is stale and the PR to fix got closed.

we might have to work with this inside nerdctl.