Here's a collection of thoughts regarding how to chose a CI/CD tool that fits your needs.
- is the overall concept documented and easy-to-understand
- does usage of tool provide enough blog posts / SO posts / tutorials / videos
- Packages (
.rpm,.deb...) available (and maintained) - Docker images available (and maintained)
- Proper update workflow (zero downtime)
- Installer for (public) cloud environments (BOSH manifest, Terrarform, Ansible, Chef, ...)
- Can setup and configuration be fully automated (e.g. Concourse via BOSH)
- Regarding configuration: GUI-only config or files (that can be put in version control...)
- HA setup, zero downtime
- do you need an on-prem solution
- could it be SaaS solution
- do you "only" build stuff, or also deploy stuff
- how big is the community around the tool
- popularity of tool (last commits, stars on Github, issues opened vs. closed...)
- do you need (commercial) support
- LDAP integration
- multi-tenancy capabilities
- Granularity of roles / permissions
- can user / roles management be stored in files or only via GUI
- SSO with Github, Gitlab, ...
- Internal credentials store
- Connection to external store (credhub, vault, ...)
- are credentials securely injected into your jobs
- can credentials be masqueraded in log outputs (to wrap up log exposure generated by ugly configured gradle plugins for example ...)
- support for openID
- Scalability of runners (auto-scale)
- for various backends (aws auto-scaling-groups, kubernetes, dockermachine, .... find more)
- allow dynamic registration of new runners
- allow save deregistration of scaled out runners!
- Integration of Dockerimages for single build-steps (be not forced to bake all the tools into a single docker)
- running on arbitrary machines (even local laptop)
- uni-directional vs. bi-directional communication between master and agents
- do runners cover all OSes that you need (i.e. Windows, Mac)
- can runners easily be set up (with your build tools)
- are environments separated (each tech stack has its own runner)
- do jobs survive a restart of the master
- Caching concept - many build jobs need massive ammount of dependencies. Not downloading these dependencies every time is relevant for build-steps speed up.
- Support of Pipelines
- Pipline per commit
- Pipelines as code
- DSL (Groovy...) or YAML
- fan-in / fan-out
- can tasks be shared in libraries
- can pipelines be generated via CLI tool / automated
- are manual approval / input steps provided
- pipeline of pipelines (cool concept found in GoCD :)
- Quality of statemachine concepts - found some suprising situations where jobs was not expected to be executed.
- Respect order of commits & serialization points in pipelines (ensure just one concurrent deployment of a single stage x service)
- grouping jobs in folders / groups
- templates for job configuration re-use
- Artifact storage and retrival for a job (S3, on CICD Server, on artifact server ...), support for artifact lifecycle
- Deployment overview per environment: If some of your jobs do a deployment, you may want to be able to redo the deployment. A overview of available and current deployments per environment is very useful.
- can tool be accessed / configured / extended via API / CLI
- what features are exposed as API
- what about multi-tenancy / permission management of API
- feedback on every commit (builds / doesn't build), visible in SCM
- check merge / pull requests
- automatically create jobs for branches
- scan ORGs / groups for projects, automatically create jobs (e.g. Jenkins Github integration)
- grant permissions to build jobs based on SCM permissions
- does tool integrate well into your ecosystem
- .NET --> TFS?
- Cloud Foundry --> Concourse?
- does ecosystem provide ready-to-use setup (e.g. spring pipelines)
- Linking / tracing of artifacts from ticket to scm commit to release to deployment
- Integration in ticketing system like JIRA - show build state, group builds, close / open tickets
- are useful, configurable dashboards provided
- is monitoring of disk usage, CPU, ... provided (e.g. integration in Prometheus...)
- what about rotating logs, deleting old artifacts, clean up of broken builds
- integration in chat servers like Slack, HipChat, Mattermost or RocketChat
- email notifications - possibility to manage groups of people that are notified
- reports on code quality, test results, performance tests
- given a broken build: how fast can you figure out, why it broke
- plugins / vs. plugin hell
- how to check new plugins before upgrade
- how well are extension points documented
- can product be extended without plugins (e.g. Concourse resources)