diff --git a/README.md b/README.md index 8765d1a9..fb830350 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ The following pipeline will run `test.sh` inside a `app` service container using steps: - command: test.sh plugins: - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: run: app ``` @@ -28,7 +28,7 @@ through if you need: steps: - command: test.sh plugins: - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: run: app config: docker-compose.tests.yml env: @@ -41,7 +41,7 @@ or multiple config files: steps: - command: test.sh plugins: - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: run: app config: - docker-compose.yml @@ -56,10 +56,22 @@ env: steps: - command: test.sh plugins: - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: run: app ``` +If you want to control how your command is passed to docker-compose, you can use the command parameter on the plugin directly: + +```yml +steps: + - plugins: + - docker-compose#v3.12.0: + run: app + command: ["custom", "command", "values"] +``` + +## Authenticated registries + You can leverage the [docker-login plugin](https://github.com/buildkite-plugins/docker-login-buildkite-plugin) in tandem for authenticating with a registry. For example, the following will build and push an image to a private repo, and pull from that private repo in subsequent run commands: ```yml @@ -67,7 +79,7 @@ steps: - plugins: - docker-login#v2.0.1: username: xyz - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: build: app image-repository: index.docker.io/myorg/myrepo - wait @@ -75,19 +87,11 @@ steps: plugins: - docker-login#v2.0.1: username: xyz - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: run: app ``` -If you want to control how your command is passed to docker-compose, you can use the command parameter on the plugin directly: - -```yml -steps: - - plugins: - - docker-compose#v3.11.1: - run: app - command: ["custom", "command", "values"] -``` +Note, you will need to add the configuration to all steps in which you use this plugin. ## Artifacts @@ -100,7 +104,7 @@ steps: - command: generate-dist.sh artifact_paths: "dist/*" plugins: - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: run: app ``` @@ -118,7 +122,7 @@ steps: - command: generate-dist.sh artifact_paths: "dist/*" plugins: - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: run: app volumes: - "./dist:/app/dist" @@ -139,7 +143,7 @@ this plugin offers a `environment` block of its own: steps: - command: generate-dist.sh plugins: - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: run: app env: - BUILDKITE_BUILD_NUMBER @@ -159,24 +163,24 @@ Alternatively, if you want to set build arguments when pre-building an image, th steps: - command: generate-dist.sh plugins: - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: build: app image-repository: index.docker.io/myorg/myrepo args: - MY_CUSTOM_ARG=panda ``` -Note that the values in the list must be a KEY=VALUE pair. +Note that the values in the list must be a `KEY=VALUE` pair. ## Pre-building the image -To speed up run steps that use the same service/image (such as steps that run in parallel), you can add a pre-build step to your pipeline: +If you have multiple steps that use the same service/image (such as steps that run in parallel), you can use this plugin in a specific `build` step to your pipeline. That will set specific metadata in the pipeline for this plugin to use in `run` steps afterwards: ```yml steps: - label: ":docker: Build" plugins: - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: build: app image-repository: index.docker.io/myorg/myrepo @@ -186,11 +190,11 @@ steps: command: test.sh parallelism: 25 plugins: - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: run: app ``` -All `run` steps for the service `app` will automatically pull and use the pre-built image. +All `run` steps for the service `app` will automatically pull and use the pre-built image. Without this, each `Test %n` job would build its own instead. ## Building multiple images @@ -202,7 +206,7 @@ steps: agents: queue: docker-builder plugins: - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: build: - app - tests @@ -214,7 +218,7 @@ steps: command: test.sh parallelism: 25 plugins: - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: run: tests ``` @@ -226,19 +230,7 @@ If you want to push your Docker images ready for deployment, you can use the `pu steps: - label: ":docker: Push" plugins: - - docker-compose#v3.11.1: - push: app -``` - -If you need to authenticate to the repository to push (e.g. when pushing to Docker Hub), use the Docker Login plugin: - -```yml -steps: - - label: ":docker: Push" - plugins: - - docker-login#v2.0.1: - username: xyz - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: push: app ``` @@ -248,9 +240,7 @@ To push multiple images, you can use a list: steps: - label: ":docker: Push" plugins: - - docker-login#v2.0.1: - username: xyz - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: push: - first-service - second-service @@ -262,9 +252,7 @@ If you want to push to a specific location (that's not defined as the `image` in steps: - label: ":docker: Push" plugins: - - docker-login#v2.0.1: - username: xyz - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: push: - app:index.docker.io/myorg/myrepo/myapp - app:index.docker.io/myorg/myrepo/myapp:latest @@ -278,14 +266,14 @@ A newly spawned agent won't contain any of the docker caches for the first run w steps: - label: ":docker: Build an image" plugins: - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: build: app image-repository: index.docker.io/myorg/myrepo cache-from: app:index.docker.io/myorg/myrepo/myapp:latest - wait - label: ":docker: Push to final repository" plugins: - - docker-compose#v3.11.1: + - docker-compose#v3.12.0: push: - app:index.docker.io/myorg/myrepo/myapp - app:index.docker.io/myorg/myrepo/myapp:latest @@ -299,7 +287,7 @@ This plugin allows for the value of `cache-from` to be a string or a list. If it steps: - label: ":docker Build an image" plugins: - - docker-compose#v3.9.0: + - docker-compose#v3.12.0: build: app image-repository: index.docker.io/myorg/myrepo cache-from: @@ -308,7 +296,7 @@ steps: - wait - label: ":docker: Push to final repository" plugins: - - docker-compose#v3.9.0: + - docker-compose#v3.12.0: push: - app:index.docker.io/myorg/myrepo/myapp - app:index.docker.io/myorg/myrepo/myapp:my-branch @@ -322,7 +310,7 @@ Adding a grouping tag to the end of a cache-from list item allows this plugin to steps: - label: ":docker: Build Intermediate Image" plugins: - - docker-compose#v3.9.0: + - docker-compose#v3.12.0: build: myservice_intermediate # docker-compose.yml is the same as myservice but has `target: intermediate` image-name: buildkite-build-${BUILDKITE_BUILD_NUMBER} image-repository: index.docker.io/myorg/myrepo/myservice_intermediate @@ -332,7 +320,7 @@ steps: - wait - label: ":docker: Build Final Image" plugins: - - docker-compose#v3.9.0: + - docker-compose#v3.12.0: build: myservice image-name: buildkite-build-${BUILDKITE_BUILD_NUMBER} image-repository: index.docker.io/myorg/myrepo @@ -348,19 +336,55 @@ are another (with a default name). The first successfully downloaded image in ea ## Configuration -### `build` +### Main Commands + +You will need to specify at least one of the following to use this extension. -The name of a service to build and store, allowing following pipeline steps to run faster as they won't need to build the image. The step’s `command` will be ignored and does not need to be specified. +#### `build` + +The name of a service to build and store, allowing following pipeline steps to run faster as they won't need to build the image. The step's `command` will be ignored and does not need to be specified. Either a single service or multiple services can be provided as an array. -### `run` +#### `run` The name of the service the command should be run within. If the docker-compose command would usually be `docker-compose run app test.sh` then the value would be `app`. -### `push` +#### `push` + +A list of services to push in the format `service:image:tag`. If an image has been pre-built with the build step, that image will be re-tagged, otherwise docker-compose's built-in push operation will be used. + +#### Known issues + +##### Run & Push + +A basic pipeline similar to the following: + +```yaml +steps: + - label: ":docker: Run & Push" + plugins: + - docker-compose#v3.12.0: + run: myservice + push: myservice +``` + +Will cause the image to be built twice (once before running and once before pushing) unless there was a previous `build` step that set the appropriate metadata. + +##### Run & Push + +A basic pipeline similar to the following: + +```yaml +steps: + - label: ":docker: Build & Push" + plugins: + - docker-compose#v3.12.0: + build: myservice + push: myservice +``` -A list of services to push in the format `service:image:tag`. If an image has been pre-built with the build step, that image will be re-tagged, otherwise docker-compose's built in push operation will be used. +Will cause the image to be pushed twice (once by the build step and another by the push step) ### `pull` (optional, run only) diff --git a/hooks/command b/hooks/command index 8e0692b1..41c857ca 100755 --- a/hooks/command +++ b/hooks/command @@ -12,14 +12,6 @@ commands=() [[ -n "$(plugin_read_list BUILD)" ]] && commands+=("BUILD") [[ -n "$(plugin_read_list RUN)" ]] && commands+=("RUN") - -# Check we've only got one of BUILD or RUN -if [[ ${#commands[@]} -gt 1 ]] ; then - echo "+++ Docker Compose plugin error" - echo "Only one of build or run is supported. More than one was used." - exit 1 -fi - [[ -n "$(plugin_read_list PUSH)" ]] && commands+=("PUSH") # Don't convert paths on gitbash on windows diff --git a/plugin.yml b/plugin.yml index b0a1d41f..1086916a 100644 --- a/plugin.yml +++ b/plugin.yml @@ -78,7 +78,7 @@ configuration: type: boolean entrypoint: type: string - oneOf: + anyOf: - required: - run - required: diff --git a/tests/multiple-commands.bats b/tests/multiple-commands.bats new file mode 100644 index 00000000..39aa0736 --- /dev/null +++ b/tests/multiple-commands.bats @@ -0,0 +1,157 @@ +#!/usr/bin/env bats + +load '/usr/local/lib/bats/load.bash' +load '../lib/shared' +load '../lib/metadata' + +# export DOCKER_COMPOSE_STUB_DEBUG=/dev/tty +# export BUILDKITE_AGENT_STUB_DEBUG=/dev/tty +# export BATS_MOCK_TMPDIR=$PWD + +# General pipeline variables +export BUILDKITE_BUILD_NUMBER=1 +export BUILDKITE_COMMAND="pwd" +export BUILDKITE_JOB_ID=12 +export BUILDKITE_PIPELINE_SLUG=test + + +@test "Build and run" { + export BUILDKITE_PLUGIN_DOCKER_COMPOSE_RUN=myservice + export BUILDKITE_PLUGIN_DOCKER_COMPOSE_BUILD=myservice + + # necessary for build + export BUILDKITE_PLUGIN_DOCKER_COMPOSE_IMAGE_REPOSITORY=my.repository/llamas + + stub docker-compose \ + "-f docker-compose.yml -p buildkite12 -f docker-compose.buildkite-1-override.yml build --pull myservice : echo built myservice" \ + "-f docker-compose.yml -p buildkite12 -f docker-compose.buildkite-1-override.yml push myservice : echo pushed myservice" \ + "-f docker-compose.yml -p buildkite12 -f docker-compose.buildkite-1-override.yml pull myservice : echo pulled myservice" \ + "-f docker-compose.yml -p buildkite12 -f docker-compose.buildkite-1-override.yml up -d --scale myservice=0 myservice : echo ran dependencies" \ + "-f docker-compose.yml -p buildkite12 -f docker-compose.buildkite-1-override.yml run --name buildkite12_myservice_build_1 --rm myservice /bin/sh -e -c 'pwd' : echo ran myservice" + + # these commands simulate metadata for a specific value by using an intermediate-file + stub buildkite-agent \ + "meta-data set docker-compose-plugin-built-image-tag-myservice \* : echo \$4 > /tmp/build-run-metadata" \ + "meta-data exists docker-compose-plugin-built-image-tag-myservice : test -f /tmp/build-run-metadata" \ + "meta-data get docker-compose-plugin-built-image-tag-myservice : cat /tmp/build-run-metadata" + + run $PWD/hooks/command + + assert_success + assert_output --partial "Building services myservice" + assert_output --partial "Pushing built images to my.repository/llamas" + assert_output --partial "Found a pre-built image for myservice" + assert_output --partial "Starting dependencies" + assert_output --partial "ran myservice" + + unstub docker-compose + unstub buildkite-agent +} + +@test "Build and push" { + export BUILDKITE_PLUGIN_DOCKER_COMPOSE_BUILD=myservice + export BUILDKITE_PLUGIN_DOCKER_COMPOSE_PUSH=myservice + + # necessary for build + export BUILDKITE_PLUGIN_DOCKER_COMPOSE_IMAGE_REPOSITORY=my.repository/llamas + + stub docker-compose \ + "-f docker-compose.yml -p buildkite12 -f docker-compose.buildkite-1-override.yml build --pull myservice : echo built myservice" \ + "-f docker-compose.yml -p buildkite12 -f docker-compose.buildkite-1-override.yml push myservice : echo build-pushed myservice" \ + "-f docker-compose.yml -p buildkite12 config : echo ''" \ + "-f docker-compose.yml -p buildkite12 push myservice : echo push-pushed myservice" + + # these commands simulate metadata for a specific value by using an intermediate-file + stub buildkite-agent \ + "meta-data set docker-compose-plugin-built-image-tag-myservice \* : echo \$4 > /tmp/build-push-metadata" \ + "meta-data exists docker-compose-plugin-built-image-tag-myservice : test -f /tmp/build-push-metadata" \ + "meta-data get docker-compose-plugin-built-image-tag-myservice : cat /tmp/build-push-metadata" + + stub docker \ + "pull my.repository/llamas:test-myservice-build-1 : echo pulled pre-built image" \ + "tag my.repository/llamas:test-myservice-build-1 buildkite12_myservice : echo re-tagged pre-built image" + + run $PWD/hooks/command + + assert_success + + assert_output --partial "Building services myservice" + assert_output --partial "Pushing built images to my.repository/llamas" + assert_output --partial "Pulling pre-built service myservice" + assert_output --partial "Tagging pre-built service myservice" + assert_output --partial "Pushing images for myservice" + + unstub docker-compose + unstub buildkite-agent +} + +@test "Run and push without pre-built image" { + export BUILDKITE_PLUGIN_DOCKER_COMPOSE_RUN=myservice + export BUILDKITE_PLUGIN_DOCKER_COMPOSE_PUSH=myservice + + stub docker-compose \ + "-f docker-compose.yml -p buildkite12 build --pull myservice : echo built myservice" \ + "-f docker-compose.yml -p buildkite12 up -d --scale myservice=0 myservice : echo ran dependencies" \ + "-f docker-compose.yml -p buildkite12 run --name buildkite12_myservice_build_1 --rm myservice /bin/sh -e -c 'pwd' : echo ran myservice" \ + "-f docker-compose.yml -p buildkite12 config : echo ''" \ + "-f docker-compose.yml -p buildkite12 build myservice : echo built-2 myservice" \ + "-f docker-compose.yml -p buildkite12 push myservice : echo pushed myservice" + + # these make sure that the image is not pre-built + stub buildkite-agent \ + "meta-data exists docker-compose-plugin-built-image-tag-myservice : exit 1" \ + "meta-data exists docker-compose-plugin-built-image-tag-myservice : exit 1" + + run $PWD/hooks/command + + assert_success + + assert_output --partial "Building Docker Compose Service: myservice" + assert_output --partial "No pre-built image found from a previous " + assert_output --partial "Starting dependencies" + assert_output --partial "ran myservice" + assert_output --partial "Building myservice" + assert_output --partial "Pushing images for myservice" + + unstub docker-compose + unstub buildkite-agent +} + + +@test "Run and push with pre-built image" { + export BUILDKITE_PLUGIN_DOCKER_COMPOSE_RUN=myservice + export BUILDKITE_PLUGIN_DOCKER_COMPOSE_PUSH=myservice + + stub docker-compose \ + "-f docker-compose.yml -p buildkite12 -f docker-compose.buildkite-1-override.yml pull myservice : echo pulled myservice" \ + "-f docker-compose.yml -p buildkite12 -f docker-compose.buildkite-1-override.yml up -d --scale myservice=0 myservice : echo ran dependencies" \ + "-f docker-compose.yml -p buildkite12 -f docker-compose.buildkite-1-override.yml run --name buildkite12_myservice_build_1 --rm myservice /bin/sh -e -c 'pwd' : echo ran myservice" \ + "-f docker-compose.yml -p buildkite12 config : echo ''" \ + "-f docker-compose.yml -p buildkite12 push myservice : echo pushed myservice" + + # these make sure that the image is not pre-built + stub buildkite-agent \ + "meta-data exists docker-compose-plugin-built-image-tag-myservice : exit 0" \ + "meta-data get docker-compose-plugin-built-image-tag-myservice : echo myservice-tag" \ + "meta-data exists docker-compose-plugin-built-image-tag-myservice : exit 0" \ + "meta-data get docker-compose-plugin-built-image-tag-myservice : echo myservice-tag" + + stub docker \ + "pull myservice-tag : echo pulled pre-built image" \ + "tag myservice-tag buildkite12_myservice : echo re-tagged pre-built image" + + run $PWD/hooks/command + + assert_success + + refute_output --partial "Building services myservice" + assert_output --partial "Found a pre-built image for myservice" + assert_output --partial "Pulling services myservice" + assert_output --partial "Starting dependencies" + assert_output --partial "Pulling pre-built service myservice" + assert_output --partial "Pushing images for myservice" + + unstub docker-compose + unstub buildkite-agent +} +