Buildkite Rails Docker Parallel Pipeline Example

This repository is an example Buildkite pipeline that demonstrates how to run a large number of parallel test jobs for a Rails application using Docker and Knapsack.

👉 See this example in action: buildkite/rails-docker-parallel-example

See the full Getting Started Guide for step-by-step instructions on how to get this running, or try it yourself:

How it works

In the pipeline configuration file the parallelism property for the test step is set to 20. When a build, the step will appear 20 times in the pipeline, each with different environment variables exposed so you can divvy up your test suite accordingly. You can set the parallelism as high as you need to bring down your build times.

This example:

• Uses Docker Compose to orchestrate the environment (Redis, Postgres, Memcached)
• Runs tests in parallel across 20 containers using Buildkite’s parallelism feature
• Leverages Knapsack for even test distribution
• Is suitable for use with autoscaling environments like the Buildkite AWS Stack

Key files:

• docker-compose.yml: defines app services
• Dockerfile: sets up the Ruby environment and database waiters
• pipeline.yml: configures a parallel test step
• scripts/ci/parallel_specs.sh: runs specs in parallel

Parallel test execution

In pipeline.yml, the test step is configured with:

parallelism: 20

This causes the step to run 20 times in parallel. Each instance is exposed to environment variables like BUILDKITE_PARALLEL_JOB and BUILDKITE_PARALLEL_JOB_COUNT, which can be used to split the test suite dynamically.

Alternate setups

💡 Not using Docker? Try the Rails Parallel Example

💡 Want dynamic splitting/allocation? See the Knapsack Pro version

Further Reading

See the parallelizing builds guide for more information to create parallelized and distributed builds with Buildkite.

See the Containerized Builds with Docker guide for information on how to use Docker Compose with Buildkite pipelines.

Consult the Knapsack documentation for configuring your database and dependent services to support running parallel steps on the one machine.

License

See LICENSE.md (MIT)