|
| 1 | +# Buildkite Rails Docker Parallel Pipeline Example |
| 2 | + |
| 3 | +[](https://buildkite.com/buildkite/rails-docker-parallel-example/builds/latest?branch=main) |
| 4 | +[](https://buildkite.com/new) |
| 5 | + |
| 6 | +This repository is an example [Buildkite](https://buildkite.com/) pipeline that demonstrates how to run a large number of parallel test jobs for a [Rails](https://rubyonrails.org/) application using [Docker](https://www.docker.com/) and [Knapsack](https://github.com/ArturT/knapsack). |
| 7 | + |
| 8 | +👉 **See this example in action:** [buildkite/rails-docker-parallel-example](https://buildkite.com/buildkite/rails-docker-parallel-example/builds/latest?branch=main) |
| 9 | + |
| 10 | +[](https://buildkite.com/new) |
| 11 | + |
| 12 | +<a href="https://buildkite.com/buildkite/rails-docker-parallel-example/builds/latest?branch=main"> |
| 13 | + <img width="1491" alt="Screenshot of Buildkite Rails Docker Parallel example pipeline" src=".buildkite/screenshot.png" /> |
| 14 | +</a> |
| 15 | + |
| 16 | +<!-- docs:start --> |
| 17 | + |
| 18 | +## How it works |
| 19 | +In the [pipeline configuration file](.buildkite/pipeline.yml) the `parallelism` property for the test step is set to 20. When a build, the step will appear 20 times in the pipeline, each with different environment variables exposed so you can divvy up your test suite accordingly. You can set the parallelism as high as you need to bring down your build times. |
| 20 | + |
| 21 | +This example: |
| 22 | +- Uses Docker Compose to orchestrate the environment (Redis, Postgres, Memcached) |
| 23 | +- Runs tests in parallel across 20 containers using Buildkite’s `parallelism` feature |
| 24 | +- Leverages [Knapsack](https://github.com/ArturT/knapsack) for even test distribution |
| 25 | +- Is suitable for use with autoscaling environments like the [Buildkite AWS Stack](https://github.com/buildkite/elastic-ci-stack-for-aws) |
| 26 | + |
| 27 | +Key files: |
| 28 | +- [`docker-compose.yml`](docker-compose.yml): defines app services |
| 29 | +- [`Dockerfile`](Dockerfile): sets up the Ruby environment and database waiters |
| 30 | +- [`pipeline.yml`](.buildkite/pipeline.yml): configures a parallel test step |
| 31 | +- [`scripts/ci/parallel_specs.sh`](scripts/ci/parallel_specs.sh): runs specs in parallel |
| 32 | + |
| 33 | +### Parallel test execution |
| 34 | + |
| 35 | +In [`pipeline.yml`](.buildkite/pipeline.yml), the test step is configured with: |
| 36 | + |
| 37 | +```yaml |
| 38 | +parallelism: 20 |
| 39 | +``` |
| 40 | +
|
| 41 | +This causes the step to run 20 times in parallel. Each instance is exposed to environment variables like `BUILDKITE_PARALLEL_JOB` and `BUILDKITE_PARALLEL_JOB_COUNT`, which can be used to split the test suite dynamically. |
| 42 | + |
| 43 | +## Alternate setups |
| 44 | + |
| 45 | +> 💡 Not using Docker? Try the [Rails Parallel Example](https://github.com/buildkite/rails-parallel-example) |
| 46 | + |
| 47 | +> 💡 Want dynamic splitting/allocation? See the [Knapsack Pro version](https://github.com/KnapsackPro/buildkite-rails-docker-parallel-example-with-knapsack_pro) |
| 48 | + |
| 49 | + |
| 50 | +## Further Reading |
| 51 | + |
| 52 | +See the [parallelizing builds guide](https://buildkite.com/docs/guides/parallelizing-builds) for more information to create parallelized and distributed builds with Buildkite. |
| 53 | + |
| 54 | +See the [Containerized Builds with Docker guide](https://buildkite.com/docs/guides/docker-containerized-builds) for information on how to use Docker Compose with Buildkite pipelines. |
| 55 | + |
| 56 | +Consult the [Knapsack documentation](https://github.com/ArturT/knapsack) for configuring your database and dependent services to support running parallel steps on the one machine. |
| 57 | + |
| 58 | +<!-- docs:end --> |
| 59 | + |
| 60 | +## License |
| 61 | + |
| 62 | +See [LICENSE.md](LICENSE.md) (MIT) |
0 commit comments