waymond - Autoscale Anything Anywhere All at once! 👀
waymond is
- An open-source autoscaler.
- Aiming to provide autoscaling for a wide variety of infrastructure.
- Modular and extensible.
- Built with Go.
There are a lot of autoscalers out there like AWS EC2 Autoscaling, kubernetes autoscalers, nomad autoscaler, etc. Most of them are suited towards specific type of targets (example: nomad autoscaler could only be used for nomad). There is a good deal of overlap between those autoscalers in-terms of what and how an autoscale can happen.
The original idea for waymond came up while trying to autoscale CI/CD workloads in self-hosted infrastructure. Example: Autoscale the number of CI agents running as systemd processes inside a big EC2 VM and when we run out of limits there, try to bring up new EC2 VMs that run one CI agent per machine for a given CI job queue. At the sametime, autoscale the agents running in a kubernetes cluster when jobs are arriving in a different CI job queue.
waymond tries to support a variety of autoscaling targets from operating system processes to kubernetes clusters and everything in-between like traditional VMs. One of the main goals of the project is to make it very easy to autoscale mixed type of targets. Truly anything and anywhere!
Triggers are the components which could trigger an event, which could ultimately yield to autoscaling. The decision of when and how to auto-scale will flow from here.
Scalers are the components which perform autoscaling operation. They access the target APIs (like docker or AWS API) to achieve a desired state in those systems.
Connectors are the components which connect two objects to facilitate the flow of events between them. Example: A connector could connect a cron trigger to docker scaler. This will ensure that the desired number of docker containers are running in a machine periodically.
Connectors can connect a trigger to a trigger (and form a chain of triggers) that ultimately is connected to a scaler. Connectors are also a good place to do data transformation of data from a trigger to a format of data that a scaler can understand.
All the components triggers, scalers, and connectors are internally connected via a simple event bus (don't be scared it is just a Go channel and some helper functions 😄). This event-based architecture will help any of the above mentioned components to capture and act on events in a seamless way.
Download the binary for your OS and architecture from the latest release. Extract the compressed package and move it an executable PATH.
# example: for linux amd64
wget https://github.com/scriptnull/waymond/releases/latest/download/waymond_Linux_x86_64.tar.gz
tar -xf waymond_Linux_x86_64.tar.gz
mv waymond /usr/bin/waymondThe only way to run waymond right now is
waymond -config waymond.tomlBut the project is looking to improve the CLI experience. So, please take a look here if you would like to contribute.
waymond accepts a toml configuration file when it boots up to configure all its components.
#
# For every 1 minute, waymond will check if there are two redis docker containers running and run them if not
#
[[trigger]]
type = "cron"
id = "global_cron"
expression = "*/1 * * * *"
[[scaler]]
type = "docker"
id = "local_redis_containers"
image_name = "redis"
image_tag = "latest"
count = 2
[[connect]]
type = "direct"
id = "run_redis_via_cron"
from = "trigger.global_cron"
to = "scaler.local_redis_containers"| type | status | description |
|---|---|---|
| cron | Available | Trigger events based on cron expressions |
| http_endpoint | Looking for contribution | Starts a HTTP server in waymond and triggers event for every HTTP request |
| http_client | Looking for contribution | Creates a HTTP client in waymond and triggers event based on the HTTP response |
| buildkite | In progress | Trigger event based on the CI job queue length in Buildkite |
Propose a new trigger here.
| type | status | description |
|---|---|---|
| docker | Available | Autoscales docker containers |
| docker_compose | Looking for contribution | Autoscales containers in a docker compose setup |
| aws_ec2 | Looking for contribution | Autoscales AWS EC2 machines |
| aws_ec2_fleet | Looking for contribution | Autoscales AWS EC2 machines via EC2 Fleet |
| aws_ec2_asg | Planned | Autoscales AWS EC2 machines via Autoscaling groups |
Propose a new scaler here.
| type | status | description |
|---|---|---|
| direct | Available | Directly pass events between Trigger-Scaler or Trigger-Trigger |
| regex | Planned | Like direct, but uses regular expressions to transform data sent between the components |
Propose a new connector here.
Software pre-requisites
If you are a user of nix package manager, you can enter the nix-shell to automatically download all the above dependencies.
Build
Run just build and ./waymond binary should be ready for use.
The project is currently in very early stages and it would be awesome if you could join us! If you are looking to talk to a human about the waymond project, feel free join the telegram group. If you are interested in requesting new features or report bugs, please do so in the issue tracker. If you are looking to contribute for the first time, try checking the issues tagged with good-first-issue.