Add CI action to push images to Docker Hub #1917

DanielCosme · 2024-03-14T20:09:53Z

This PR creates a Github Action that builds and push docker images to Docker Hub. For now the trigger to this workflow will be manual and will be tagged only with latest. We expect to do a second pass on this workflow so that we have multiple tags that relate to different versions of AM. We will do this once we have a better understanding on how a dockerised production grade deployment of AM looks like.

Also, the Dockerfile has been reworked a little bit to make the resulting images smaller in the registry. We did this by only using dependencies that are strictly needed for all components of AM. Before all 3 components shared dependencies that where not really needed.

I expect to do an additional pass on this file in the future, once I have a better understanding on how the dependencies work. As for the archivematica-tests target I left it as it was originally.

replaceafill

@DanielCosme great progress! The new workflow for publishing images looks good to me. I haven't tested the Dockerfile in the PR yet, but I left a few questions and suggestions.

replaceafill · 2024-03-20T14:33:31Z

.github/workflows/push-images.yml

+on: workflow_dispatch
+jobs:
+  build:
+    name: "Builds images and push them to Docker Hub"


Suggested change

name: "Builds images and push them to Docker Hub"

name: "Build and push images"

.github/workflows/push-images.yml

replaceafill · 2024-03-20T14:42:30Z

.github/workflows/push-images.yml

+          load: true
+          file: ./hack/Dockerfile
+          target: "archivematica-mcp-server"
+          tags: artefactual/archivematica-mcp-server:latest


Could we add a newline at the end of the file?

replaceafill · 2024-03-20T14:46:51Z

hack/Dockerfile

+		libldap2-dev \
+		libsasl2-dev \


Could these be removed since they're already installed in the base-builder?

replaceafill · 2024-03-20T14:48:55Z

hack/Dockerfile

+	&& apt-get update \
+	&& apt-get install -y --no-install-recommends \
+		coreutils \
+		clamav \


I might be wrong, but I think the clamav package and the:

# Download ClamAV virus signatures RUN freshclam --quiet

instruction below are only necessary in the MCPClient stage.

I'll try it.

Done. Nothing broke in the tests.

replaceafill · 2024-03-20T14:51:29Z

hack/Dockerfile

+RUN set -ex \
+	&& apt-get update \
+	&& apt-get install -y --no-install-recommends \
+		coreutils \


I noticed the ansible-archivematica-src vars set this as a dashboard dependency. Should it be moved there?

I remember. I decided to leave coreutils here just because I don't understand the components and dependencies enough. coreutils contains (sometimes) essential packages for the OS. And since AM is so reliant on OS dependencies I left it in the base-builder step. Should I remove it?

I think you can try moving coreutils to the dashboard stage and let the Acceptance Test workflow to tell you if anything breaks.

coreutils removed. nothing broke :)

replaceafill · 2024-03-20T14:59:51Z

hack/Dockerfile

@@ -283,7 +296,75 @@ ENTRYPOINT ["pyenv", "exec", "python3", "-m", "gunicorn", "--config=/src/src/das

 # -----------------------------------------------------------------------------

-FROM base AS archivematica-tests
+FROM base-builder as archivematica-tests


I think the archivematica-tests stage can be much more simple. It's needed for running tox and building its virtual environments, so it only needs Python, the common development libraries base-builder already has and from a quick local test based on the qa/1.x branch the following dependencies:

gcc for building C dependencies for the virtual environment.

media-types, p7zip-full, rsync and unar for being able to run the Storage Service tests from the hack/Makefile.

Yeah, it works, thank you. Done.

replaceafill · 2024-03-20T15:10:12Z

hack/Dockerfile

+RUN set -ex \
+	&& groupadd --gid ${GROUP_ID} --system archivematica \
+	&& useradd --uid ${USER_ID} --gid ${GROUP_ID} --home-dir /var/archivematica --system archivematica \
+	&& mkdir -p /var/archivematica/sharedDirectory \
+	&& chown -R archivematica:archivematica /var/archivematica
+
+# Download ClamAV virus signatures
+RUN freshclam --quiet
+
+USER archivematica
+
+COPY --chown=${USER_ID}:${GROUP_ID} --from=pyenv-builder --link ${PYENV_DIR} ${PYENV_DIR}
+COPY --chown=${USER_ID}:${GROUP_ID} --link . /src


Not sure if it makes sense, but I think if you end up moving all the OS dependencies out of base to their corresponding stages, this block might be inherited from the base stage instead.

So what I think you are saying is that this block is not really needed because it already happens in base. So by having this block here is just redundant, doing the same work again, right?

That's correct.

sevein · 2024-03-21T18:44:41Z

.github/workflows/push-images.yml

+          file: ./hack/Dockerfile
+          target: "archivematica-dashboard"
+          tags: artefactual/archivematica-dashboard:latest
+      - name: "Build and Push MCP-client"


Suggested change

- name: "Build and Push MCP-client"

- name: "Build and push MCPClient"

sevein · 2024-03-21T18:45:02Z

.github/workflows/push-images.yml

+          file: ./hack/Dockerfile
+          target: "archivematica-mcp-client"
+          tags: artefactual/archivematica-mcp-client:latest
+      - name: "Build and Push MCP-server"


Suggested change

- name: "Build and Push MCP-server"

- name: "Build and push MCPServer"

sevein · 2024-03-21T18:55:14Z

hack/Dockerfile

+ARG USER_ID=1000
+ARG GROUP_ID=1000
+ARG PYENV_DIR=/pyenv


According to the docs, we don't need to redefine the default again when pulling the argument, e.g.:

Suggested change

ARG USER_ID=1000

ARG GROUP_ID=1000

ARG PYENV_DIR=/pyenv

ARG USER_ID

ARG GROUP_ID

ARG PYENV_DIR

DanielCosme · 2024-04-03T19:06:43Z

@replaceafill I think I addressed all feedback. All tests are passing, if you have any further feedback, let me know.

This adds a data migration for replacing the existing command with a fixed version of its Python script.

orjson is significantly faster and comes with native serialization of some types like datetime or UUID.

This commit modifies the MCPServer to encode the `createdDate` of tasks using `datetime` which is natively supported by our JSONDataEncoder. The conversion to the METS-compliant format now occurs during the decoding process on the client side. This change ensures consistent use of RFC 3339 format across all communications.

MySQL uses mbind to bind processes and threads to CPUs which is not allowed by the default configuration of Docker's seccomp profile. This causes MySQL to log the following error repeatedly: mbind: Operation not permitted Adding the `SYS_NICE` capability to the container forces Docker to adjust the default seccomp profile to include the necessary permissions. For more details, visit docker-library/mysql#303. It looks like MySQL fixed the problem in [v8.0.13] but Percona doesn't seem to have received this update. [v8.0.13]: https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-13.html

This replaces the black, pyupgrade, reorder-python-imports and flake8 hooks in pre-commit with ruff. It also upgrades the django-upgrade and markdownlint-cli hooks.

DanielCosme added the AMAUAT Issues relating to the improvement of the AM Automated Acceptance tests label Mar 14, 2024

DanielCosme force-pushed the dev/trim-dockerfile branch from 97499bd to 6bcc277 Compare March 18, 2024 14:27

DanielCosme self-assigned this Mar 18, 2024

DanielCosme requested review from replaceafill and sevein March 18, 2024 14:34

DanielCosme added AMAUAT Issues relating to the improvement of the AM Automated Acceptance tests and removed AMAUAT Issues relating to the improvement of the AM Automated Acceptance tests labels Mar 18, 2024

replaceafill reviewed Mar 20, 2024

View reviewed changes

sevein reviewed Mar 21, 2024

View reviewed changes

DanielCosme added AMAUAT Issues relating to the improvement of the AM Automated Acceptance tests and removed AMAUAT Issues relating to the improvement of the AM Automated Acceptance tests labels Apr 3, 2024

DanielCosme force-pushed the dev/trim-dockerfile branch from 86dcdbc to 4e772f4 Compare April 15, 2024 16:05

DanielCosme added 8 commits April 30, 2024 12:58

Add GHA to build and push images to docker hub

4afa5d1

Make Docker images smaller

6b5dbc7

WIP: feedback from code review

550e541

WIP: fix typos in GH workflow

7221d26

WIP: trigger CI

d207c91

Remove coreutils comment

f6a5367

Revert back Github action trigger to manual

1efafce

Add SSH package to mcp-client

b3f5b83

DanielCosme force-pushed the dev/trim-dockerfile branch from 4e772f4 to b3f5b83 Compare April 30, 2024 16:59

DanielCosme and others added 6 commits May 1, 2024 12:07

WIP: trim build time dependencies from final images

dabcad0

Extend elasticSearchFunctions test coverage

3e1bd3b

Remove comment from JHOVE validation migration

58e30b3

Fix default thumbnail normalization command

ba0db52

This adds a data migration for replacing the existing command with a fixed version of its Python script.

Fix SIP arrangement from ArchivesSpace pane

fb09902

Fix failure reports index

79ab3d5

replaceafill and others added 18 commits May 31, 2024 12:04

Remove references to Binder

34aa3c8

Upgrade Python requirements

159a1ec

Update submodules

44ae43e

Update archivematica-storage-service submodule

b08560e

Bump version to 1.17.0

fb498dc

Upgrade Python requirements

d5344ce

Remove dead code in MCPServer

c73d1b3

Remove unused file filter

1ae0dc7

Use json.loads in OutputClientScriptJob

6927afe

Introduce JSONDataEncoder for Gearman

54f9de7

Use JSONDataEncoder in MCP task backend

ab63027

Use JSONDataEncoder in RPC server

95153d7

Remove use of pickle in Access.target

96e9d3a

Use orjson in JSONDataEncoder

5326b78

orjson is significantly faster and comes with native serialization of some types like datetime or UUID.

Use Ruff for Python linting and formatting

20d633e

This replaces the black, pyupgrade, reorder-python-imports and flake8 hooks in pre-commit with ruff. It also upgrades the django-upgrade and markdownlint-cli hooks.

Allow triggering the test CI workflows manually

6ea792c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CI action to push images to Docker Hub #1917

Add CI action to push images to Docker Hub #1917

DanielCosme commented Mar 14, 2024 •

edited

Loading

replaceafill left a comment

replaceafill Mar 20, 2024

DanielCosme Apr 3, 2024

replaceafill Mar 20, 2024

DanielCosme Apr 3, 2024

replaceafill Mar 20, 2024

DanielCosme Apr 3, 2024

replaceafill Mar 20, 2024

DanielCosme Apr 3, 2024

DanielCosme Apr 3, 2024

replaceafill Mar 20, 2024

DanielCosme Apr 3, 2024 •

edited

Loading

replaceafill Apr 3, 2024

DanielCosme Apr 3, 2024

replaceafill Mar 20, 2024

DanielCosme Apr 3, 2024

replaceafill Mar 20, 2024

DanielCosme Apr 3, 2024

replaceafill Apr 3, 2024

DanielCosme Apr 3, 2024

sevein Mar 21, 2024 •

edited

Loading

DanielCosme Apr 3, 2024

sevein Mar 21, 2024 •

edited

Loading

DanielCosme Apr 3, 2024

sevein Mar 21, 2024

DanielCosme Apr 3, 2024

DanielCosme commented Apr 3, 2024 •

edited

Loading

	name: "Builds images and push them to Docker Hub"
	name: "Build and push images"

	- name: "Build and Push MCP-client"
	- name: "Build and push MCPClient"

	- name: "Build and Push MCP-server"
	- name: "Build and push MCPServer"

Add CI action to push images to Docker Hub #1917

Are you sure you want to change the base?

Add CI action to push images to Docker Hub #1917

Conversation

DanielCosme commented Mar 14, 2024 • edited Loading

replaceafill left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DanielCosme Apr 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sevein Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sevein Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DanielCosme commented Apr 3, 2024 • edited Loading

DanielCosme commented Mar 14, 2024 •

edited

Loading

DanielCosme Apr 3, 2024 •

edited

Loading

sevein Mar 21, 2024 •

edited

Loading

sevein Mar 21, 2024 •

edited

Loading

DanielCosme commented Apr 3, 2024 •

edited

Loading