Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add build stage to Dockerfile #155

Merged
merged 1 commit into from
Nov 25, 2024

Conversation

pjonsson
Copy link
Collaborator

The image currently has 840 security
vulnerabilities according to Trivy.
Many of those vulnerabilities are
in the development packages, so
add a build stage to the Dockerfile
so the development packages do not
end up in the final image. This reduces
the final image size by roughly
1/3 of its size.

Since everything is being changed,
also replace wget with curl, so we
get error messages on HTTP failures
(wget -q silences everything including
error printouts, and the behavior
cannot be overridden).

@pjonsson
Copy link
Collaborator Author

@alexgleith according to the commit logs it was you who removed the previously existing build-stage of the Dockerfile. What do you say about re-introducing it? There's a substantial saving in image size, and a fair amount of fewer security vulnerabilities flagged by Trivy.

This PR builds the binary wheels in the build stage of the container and installs them in the "real" container. I'm not sure how to put the wheels in the requirements.txt so it works with pip-compile, or if we perhaps could drop the pip-compile step completely (which would reduce the number of releases required since a rebuild of the image would give security updates).

And as a general remark, I cannot understand how the CI tests did not trigger the eodatasets3 distutils dependency problem until now.

Copy link
Contributor

@alexgleith alexgleith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Does it reduce the image size significantly?

The reason I like single-stage builds is that it keep things simpler... but this looks neat and tidy, and with Python 12 requiring virtual environments, it's a good way to do it.

One idea, and it's kinda complex too, is to use uv to do the installing... it's super fast. Does introduce a new change, though. Example here for doing that: https://github.com/auspatious/ldn-land-productivity/blob/main/Dockerfile

index/Dockerfile Outdated
@@ -37,26 +68,30 @@ RUN apt-get update \
ENV VIRTUAL_ENV=/virtualenv/python3.12 \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could set this earlier, and use it in the previous step.

@pjonsson pjonsson force-pushed the split-docker-file branch 2 times, most recently from f98f867 to 70a2b23 Compare November 21, 2024 22:56
@pjonsson pjonsson requested a review from alexgleith November 21, 2024 23:27
@pjonsson
Copy link
Collaborator Author

According to "docker images", the image is now 966MB instead of 1.43GB, which I think should be considered respectable savings.

I would like to make a release of this repository when the don't-hang-on-empty-stac-files fix has appeared in a release on PyPi, so I'll update uv.lock and version.txt in a separate PR when that time comes.

alexgleith
alexgleith previously approved these changes Nov 24, 2024
Copy link
Contributor

@alexgleith alexgleith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Just two minor comments

index/Dockerfile Outdated Show resolved Hide resolved
index/Dockerfile Outdated Show resolved Hide resolved
The image currently has 840 security
vulnerabilities according to Trivy.
Many of those vulnerabilities are
in the development packages, so
add a build stage to the Dockerfile
so the development packages do not
end up in the final image. This reduces
the final image size by roughly
1/3 of its size.

Since everything is being changed,
also replace wget with curl, so we
get error messages on HTTP failures
(wget -q silences everything including
error printouts, and the behavior
cannot be overridden).
@alexgleith alexgleith merged commit 59ec1f5 into opendatacube:main Nov 25, 2024
3 checks passed
@pjonsson pjonsson deleted the split-docker-file branch November 25, 2024 23:25
@pjonsson
Copy link
Collaborator Author

Seems the build failed for the job that pushes the image. I have no idea what is causing it, but I filed bug report on the action (whoan/docker-build-with-cache-action#161) and made a PR that adds Dependabot for Github actions/Dockerfile updates (whoan/docker-build-with-cache-action#162) in case it is related to the Docker version the action uses.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants