Skip to content

Commit

Permalink
Introduce Java-Python base image for builder server (#7419)
Browse files Browse the repository at this point in the history
  • Loading branch information
Joe Reuter committed Jun 28, 2023
1 parent 171a377 commit b125210
Show file tree
Hide file tree
Showing 5 changed files with 39 additions and 14 deletions.
1 change: 1 addition & 0 deletions airbyte-base-java-image/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,5 +20,6 @@ To release a new version of this base image, use the following steps:
```bash
FROM airbyte/airbyte-base-java-image:<NEW VERSION>
```
5. Update the [airbyte-base-java-python-image](../airbyte-base-java-python-image/) image to use the new version of this image.

[dockerhub]: https://hub.docker.com/repository/docker/airbyte/airbyte-base-java-image/general
13 changes: 13 additions & 0 deletions airbyte-base-java-python-image/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
FROM airbyte/airbyte-base-java-image:2.0.3

RUN yum update -y && \
yum groupinstall -y "Development Tools" && \
yum install -y openssl11-devel bzip2-devel libffi-devel zlib-devel sqlite-devel xz-devel

ENV PYTHON_VERSION=3.9.11

# Set up python
RUN git clone https://github.com/pyenv/pyenv.git ~/.pyenv
ENV PYENV_ROOT /root/.pyenv
ENV PATH ${PYENV_ROOT}/shims:${PYENV_ROOT}/bin:$PATH
RUN pyenv install ${PYTHON_VERSION} && pyenv global ${PYTHON_VERSION}
23 changes: 23 additions & 0 deletions airbyte-base-java-python-image/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Base Docker Image for Java + Python

This Docker image provides the base for any Java-based Airbyte module that also needs to run python code. It is based on the [airbyte/airbyte-base-java-image](../airbyte-base-java-image/) image.

# Releasing

To release a new version of this base image, use the following steps:

1. Log in to [Dockerhub](https://hub.docker.com/) via the Docker CLI (`docker login`).
2. Run `docker buildx create --use` to enable Docker `buildx` if you have not used it previously.
3. Run the following to build and push a new version of this image (replace `<new_version>` with a new version!) :
```
docker buildx build --push \
--tag airbyte/airbyte-base-java-python-image:<new_version> \
--platform linux/amd64,linux/arm64 .
```
To see existing versions, [view the image on Dockerhub](https://hub.docker.com/r/airbyte/airbyte-base-java-python-image).
4. Update base Docker image tag to the new version in all Dockerfiles that depend on the base image:
```bash
FROM airbyte/airbyte-base-java-python-image:<NEW VERSION>
```

[dockerhub]: https://hub.docker.com/repository/docker/airbyte/airbyte-base-java-python-image/general
14 changes: 2 additions & 12 deletions airbyte-connector-atelier-server/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,18 +1,8 @@
ARG JDK_IMAGE=airbyte/airbyte-base-java-image:2.0.3
FROM ${JDK_IMAGE} AS connector-atelier-server

RUN yum update -y && \
yum groupinstall -y "Development Tools" && \
yum install -y openssl11-devel bzip2-devel libffi-devel zlib-devel sqlite-devel xz-devel
ARG BASE_IMAGE=airbyte/airbyte-base-java-python-image:1.0
FROM ${BASE_IMAGE} AS connector-atelier-server

# Set up CDK requirements
ARG PYTHON_VERSION=3.9.11
ARG CDK_VERSION=0.43.0
# Set up python
RUN git clone https://github.com/pyenv/pyenv.git ~/.pyenv
ENV PYENV_ROOT /root/.pyenv
ENV PATH ${PYENV_ROOT}/shims:${PYENV_ROOT}/bin:$PATH
RUN pyenv install ${PYTHON_VERSION} && pyenv global ${PYTHON_VERSION}
ENV CDK_PYTHON=${PYENV_ROOT}/versions/${PYTHON_VERSION}/bin/python
ENV CDK_ENTRYPOINT ${PYENV_ROOT}/versions/${PYTHON_VERSION}/lib/python3.9/site-packages/airbyte_cdk/connector_builder/main.py
# Set up CDK
Expand Down
2 changes: 0 additions & 2 deletions airbyte-connector-atelier-server/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,6 @@ tasks.named("buildDockerImage") {
project.projectDir.parentFile,
'airbyte-connector-builder-resources/CDK_VERSION').text.trim()
buildArgs['CDK_VERSION'] = cdkVersion
// Current Python version used by the CDK image
buildArgs['PYTHON_VERSION'] = System.getenv('PYTHON_VERSION') ?: '3.9.11'

dependsOn copyPythonDependencies
dependsOn generateOpenApiServer
Expand Down

0 comments on commit b125210

Please sign in to comment.