/bin/sh not available when building image in AWS Codebuild with aws-xray-daemon base image

[fosrias]

I have verified that both 3.3.0 and 3.3.1 fail with the same error when using aws-xray-daemon as a base image for the following docker file when running in AWS Codebuild (just show the file using the latest, but setting the version explicitly fails as well).

0.324 container_linux.go:349: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: no such file or directory"

It does not fail in version 3.2.0. This error started showing up in our pipelines after your recent release.

Further, this error does not occur building the docker file on OSX 10.14.6, for reference.

FROM amazon/aws-xray-daemon

LABEL "description"="AWS X-Ray Daemon Dockerfile" "location"="Infrastructure/cdk/lib/aws-xray-daemon"

WORKDIR /

# Add Tini
ENV TINI_VERSION v0.19.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini
ENTRYPOINT ["/tini", "--"]

ADD cfg.yaml /

EXPOSE 2000/udp
EXPOSE 2000/tcp

CMD ["/usr/bin/xray", "-n", "us-west-2", "-b", "0.0.0.0:2000", "-t", "0.0.0.0:2000", "-l", "info"]

The error log is:

[Container] 2021/04/14 16:49:24 Running command bash ./build.sh
--
239 | + export DOCKER_BUILDKIT=1
240 | + DOCKER_BUILDKIT=1
241 | + export BUILDKIT_INLINE_CACHE=BUILDKIT_INLINE_CACHE=1
242 | + BUILDKIT_INLINE_CACHE=BUILDKIT_INLINE_CACHE=1
243 | + export TAG=docker.io/rapticore/amazon-xray-daemon:latest
244 | + TAG=docker.io/rapticore/amazon-xray-daemon:latest
245 | + pwd
246 | /codebuild/output/src053465283/src/Infrastructure/cdk/lib/aws-xray-daemon
247 | + ls -lh
248 | total 16K
249 | -rw-rw-r-- 1 root root  461 Apr 14 16:30 Dockerfile
250 | -rwxrwxr-x 1 root root  508 Apr 14 16:30 build.sh
251 | -rw-rw-r-- 1 root root 1.5K Apr 14 16:30 cfg.yaml
252 | -rwxrwxr-x 1 root root   92 Apr 14 16:30 path.sh
253 | + true
254 | + docker login
255 | + cat
256 | Authenticating with existing credentials...
257 | WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
258 | Configure a credential helper to remove this warning. See
259 | https://docs.docker.com/engine/reference/commandline/login/#credentials-store
260 |  
261 | Login Succeeded
262 | + true
263 | + cat
264 | + docker login https://registry-1.docker.io/v2/
265 | Authenticating with existing credentials...
266 | WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
267 | Configure a credential helper to remove this warning. See
268 | https://docs.docker.com/engine/reference/commandline/login/#credentials-store
269 |  
270 | Login Succeeded
271 | + docker build --build-arg BUILDKIT_INLINE_CACHE=1 --rm=false --cache-from docker.io/rapticore/amazon-xray-daemon:latest -t rapticore/aws-xray-daemon:latest -t docker.io/rapticore/amazon-xray-daemon:latest -f Dockerfile .
272 | #1 [internal] load .dockerignore
273 | #1 transferring context: 2B done
274 | #1 DONE 0.0s
275 |  
276 | #2 [internal] load build definition from Dockerfile
277 | #2 transferring dockerfile: 500B 0.0s done
278 | #2 DONE 0.0s
279 |  
280 | #3 [internal] load metadata for docker.io/amazon/aws-xray-daemon:3.3.1
281 | #3 DONE 1.2s
282 |  
283 | #5 [1/5] FROM docker.io/amazon/aws-xray-daemon:3.3.1@sha256:257520f15ad8300...
284 | #5 DONE 0.0s
285 |  
286 | #9 [internal] load build context
287 | #9 DONE 0.0s
288 |  
289 | #6 https://github.com/krallin/tini/releases/download/v0.19.0/tini
290 | #6 DONE 0.1s
291 |  
292 | #4 importing cache manifest from docker.io/rapticore/amazon-xray-daemon:lat...
293 | #4 DONE 0.6s
294 |  
295 | #6 https://github.com/krallin/tini/releases/download/v0.19.0/tini
296 | #6 DONE 0.0s
297 |  
298 | #9 [internal] load build context
299 | #9 transferring context: 1.52kB done
300 | #9 DONE 0.0s
301 |  
302 | #5 [1/5] FROM docker.io/amazon/aws-xray-daemon:3.3.1@sha256:257520f15ad8300...
303 | #5 resolve docker.io/amazon/aws-xray-daemon:3.3.1@sha256:257520f15ad830044f73d1b194bb857d61865a48b9d6b39e76e35491ea620e93 done
304 | #5 sha256:257520f15ad830044f73d1b194bb857d61865a48b9d6b39e76e35491ea620e93 743B / 743B done
305 | #5 sha256:eb0fd00316c41f07e47a2f42edf6e79601b9116c948147084c82e17a4dafa3d6 1.15kB / 1.15kB done
306 | #5 sha256:3e37efe593d7d3a497e2d864356cef53d2de46da16029efff0db87e8f253f18b 0B / 605B 0.2s
307 | #5 sha256:b74e1e16d401488307c5e5afa74d8dbfdfa74f320efb8eef1e3a582224e1736b 0B / 122.41kB 0.2s
308 | #5 sha256:f9066f994766b7bfaf916a45511091f6b1702904139d861bf4867d3e71fe2ec3 1.42kB / 1.42kB done
309 | #5 sha256:1454e538c265c921848ec6bf7b91f37b90ca3f5eca3afb125cadcfe976d3cbd5 2.69MB / 3.39MB 0.2s
310 | #5 sha256:b74e1e16d401488307c5e5afa74d8dbfdfa74f320efb8eef1e3a582224e1736b 122.41kB / 122.41kB 0.3s done
311 | #5 sha256:1454e538c265c921848ec6bf7b91f37b90ca3f5eca3afb125cadcfe976d3cbd5 3.39MB / 3.39MB 0.3s done
312 | #5 extracting sha256:1454e538c265c921848ec6bf7b91f37b90ca3f5eca3afb125cadcfe976d3cbd5
313 | #5 sha256:958577898abbd97dc9f7a556e3cb671c091246492bd9eaffd5d2675e4d14a869 0B / 807B 0.3s
314 | #5 sha256:3e37efe593d7d3a497e2d864356cef53d2de46da16029efff0db87e8f253f18b 605B / 605B 0.3s done
315 | #5 extracting sha256:1454e538c265c921848ec6bf7b91f37b90ca3f5eca3afb125cadcfe976d3cbd5 0.2s done
316 | #5 sha256:958577898abbd97dc9f7a556e3cb671c091246492bd9eaffd5d2675e4d14a869 807B / 807B 0.4s done
317 | #5 extracting sha256:3e37efe593d7d3a497e2d864356cef53d2de46da16029efff0db87e8f253f18b
318 | #5 extracting sha256:3e37efe593d7d3a497e2d864356cef53d2de46da16029efff0db87e8f253f18b done
319 | #5 extracting sha256:b74e1e16d401488307c5e5afa74d8dbfdfa74f320efb8eef1e3a582224e1736b 0.0s done
320 | #5 extracting sha256:958577898abbd97dc9f7a556e3cb671c091246492bd9eaffd5d2675e4d14a869 done
321 | #5 DONE 0.8s
322 |  
323 | #7 [2/5] ADD https://github.com/krallin/tini/releases/download/v0.19.0/tini...
324 | #7 DONE 0.3s
325 |  
326 | #8 [3/5] RUN chmod +x /tini
327 | #8 0.316 container_linux.go:349: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: no such file or directory"
328 | #8 ERROR: executor failed running [/bin/sh -c chmod +x /tini]: runc did not terminate sucessfully
329 | ------
330 | > [3/5] RUN chmod +x /tini:
331 | ------
332 | failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c chmod +x /tini]: runc did not terminate sucessfully
333 |  
334 | [Container] 2021/04/14 16:49:30 Command did not exit successfully bash ./build.sh exit status 1
335 | [Container] 2021/04/14 16:49:30 Phase complete: BUILD State: FAILED
336 | [Container] 2021/04/14 16:49:30 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: bash ./build.sh. Reason: exit status 1
337 | [Container] 2021/04/14 16:49:30 Entering phase POST_BUILD
338 | [Container] 2021/04/14 16:49:30 Phase complete: POST_BUILD State: SUCCEEDED
339 | [Container] 2021/04/14 16:49:30 Phase context status code:  Message:
340 |  

Error shows up at line 326.

This error actually has nothing to do with the presence of /tini file as I have also run this with a generic bash command before that statement ls -la in a test version of the docker file and got the same error message (partial log):

#12 [1/7] FROM docker.io/amazon/aws-xray-daemon@sha256:257520f15ad830044f73d...
#12 resolve docker.io/amazon/aws-xray-daemon@sha256:257520f15ad830044f73d1b194bb857d61865a48b9d6b39e76e35491ea620e93 done
#12 sha256:3e37efe593d7d3a497e2d864356cef53d2de46da16029efff0db87e8f253f18b 0B / 605B 0.1s
#12 sha256:b74e1e16d401488307c5e5afa74d8dbfdfa74f320efb8eef1e3a582224e1736b 0B / 122.41kB 0.1s
#12 sha256:257520f15ad830044f73d1b194bb857d61865a48b9d6b39e76e35491ea620e93 743B / 743B done
#12 sha256:eb0fd00316c41f07e47a2f42edf6e79601b9116c948147084c82e17a4dafa3d6 1.15kB / 1.15kB done
#12 sha256:f9066f994766b7bfaf916a45511091f6b1702904139d861bf4867d3e71fe2ec3 1.42kB / 1.42kB done
#12 sha256:1454e538c265c921848ec6bf7b91f37b90ca3f5eca3afb125cadcfe976d3cbd5 0B / 3.39MB 0.1s
#12 sha256:3e37efe593d7d3a497e2d864356cef53d2de46da16029efff0db87e8f253f18b 605B / 605B 0.1s done
#12 sha256:1454e538c265c921848ec6bf7b91f37b90ca3f5eca3afb125cadcfe976d3cbd5 261.10kB / 3.39MB 0.2s
#12 sha256:958577898abbd97dc9f7a556e3cb671c091246492bd9eaffd5d2675e4d14a869 0B / 807B 0.2s
#12 extracting sha256:1454e538c265c921848ec6bf7b91f37b90ca3f5eca3afb125cadcfe976d3cbd5
#12 sha256:b74e1e16d401488307c5e5afa74d8dbfdfa74f320efb8eef1e3a582224e1736b 122.41kB / 122.41kB 0.3s done
#12 sha256:1454e538c265c921848ec6bf7b91f37b90ca3f5eca3afb125cadcfe976d3cbd5 3.39MB / 3.39MB 0.3s done
#12 sha256:958577898abbd97dc9f7a556e3cb671c091246492bd9eaffd5d2675e4d14a869 807B / 807B 0.3s done
#12 extracting sha256:1454e538c265c921848ec6bf7b91f37b90ca3f5eca3afb125cadcfe976d3cbd5 0.2s done
#12 extracting sha256:3e37efe593d7d3a497e2d864356cef53d2de46da16029efff0db87e8f253f18b done
#12 extracting sha256:b74e1e16d401488307c5e5afa74d8dbfdfa74f320efb8eef1e3a582224e1736b 0.0s done
#12 extracting sha256:958577898abbd97dc9f7a556e3cb671c091246492bd9eaffd5d2675e4d14a869
#12 extracting sha256:958577898abbd97dc9f7a556e3cb671c091246492bd9eaffd5d2675e4d14a869 done
#12 DONE 0.9s

#5 [2/7] RUN ls -la
#5 0.324 container_linux.go:349: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: no such file or directory"
#5 ERROR: executor failed running [/bin/sh -c ls -la]: runc did not terminate sucessfully
------
 > [2/7] RUN ls -la:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c ls -la]: runc did not terminate sucessfully

[Container] 2021/04/14 11:59:02 Command did not exit successfully bash ./build.sh exit status 1
[Container] 2021/04/14 11:59:02 Phase complete: BUILD State: FAILED
[Container] 2021/04/14 11:59:02 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: bash ./build.sh. Reason: exit status 1
[Container] 2021/04/14 11:59:02 Entering phase POST_BUILD
[Container] 2021/04/14 11:59:02 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2021/04/14 11:59:02 Phase context status code:  Message: 

I am not sure what type of instance Codebuild is using to run this and happy to add that information if you tell me what you need and how to get it from AWS console.

[willarmiros]

Hi @fosrias,

Thank you for raising this. In the latest releases of the daemon, we switched the base image of the amazon/aws-xray-daemon image from amazonlinux to scratch, as you can see in our Dockerfile. This greatly reduced the size of the daemon image from about 150 MB to about 3 MB. However this change also removes all of the executables that existed alongside the daemon binary in its image, such as /bin/sh which you were trying to use.

We didn't sufficiently consider that customers were using the tooling provided by amazonlinux, but we are now aware of that issue and are considering different approaches for structuring the image. We will post updates to this issue as we have them.

For now, I would suggest using a Docker multi-stage build for your Dockerfile. In the build stage, you can use a heavier linux base image to set up your /tini project. After that, you can add another stage with amazon/aws-xray-daemon as the parent image.

[fosrias]

@willarmiros Good idea. Thanks.

[willarmiros]

Closing this since it describes the same issue as #119. We will track the missing tooling over there.