diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1e32e11..3937c35 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -67,3 +67,25 @@ Instead, pass `-D /local` to the [`runner.sh`](./runner.sh) script. This will mount the [`runner`](./runner/) directory into the microVM at `/local` and run the scripts that it contains from there instead. Which "entrypoint" to use is driven by the `RUNNER_ENTRYPOINT` variable in [`runner.sh`](./runner.sh). + +## Cleanup + +During development, many images might be created. To clean them away, you can +run one of the following commands. + +When using the `krunvm` runtime: + +```bash +buildah rmi $(buildah images --format '{{.ID}}') +``` + +When using `podman+krun`: + +```bash +podman image rm $(podman images -q) +``` + +> [!WARNING] +> These commands will remove all unused images on your system. Make sure you +> don't need any of these images for other projects before running the cleanup. +> You may need to rebuild images for this project after cleanup. diff --git a/README.md b/README.md index 08e949a..cad2900 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,24 @@ # krunvm-based GitHub Runner(s) This project creates [self-hosted][self] (ephemeral) GitHub [runners] based on -[krunvm]. [krunvm] creates [microVM]s, so the project enables fully isolated +[libkrun]. [libkrun] creates [microVM]s, so the project enables fully isolated [runners] inside your infrastruture. MicroVMs boot fast, providing an experience -close to running containers. [krunvm] creates and starts VMs based on the +close to running containers. [libkrun] creates and starts VMs based on the multi-platform OCI images created for this project -- [ubuntu] (default) or -[fedora]. +[fedora]. The project will create [microVM]s using either [krunvm] or +[krun][crun] and [podman]. ![Demo](./demo/demo.gif) [self]: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners [runners]: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners + [libkrun]: https://github.com/containers/libkrun [krunvm]: https://github.com/containers/krunvm [microVM]: https://github.com/infracloudio/awesome-microvm [ubuntu]: https://github.com/efrecon/gh-runner-krunvm/pkgs/container/runner-krunvm-ubuntu [fedora]: https://github.com/efrecon/gh-runner-krunvm/pkgs/container/runner-krunvm-fedora + [crun]: https://github.com/containers/crun + [podman]: https://github.com/containers/podman ## Example @@ -68,6 +72,7 @@ starting with `RUNNER_` will affect the behaviour of each [runner] (loop). critical software tools. + Good compatibility with the regular GitHub [runners]: same user ID, member of the `docker` group, password-less `sudo`, etc. ++ Supports both [krunvm] or the [krun][crun] runtime under [podman]. + In theory, the main [ubuntu] and [fedora] images should be able to be used in more traditional container-based solutions -- perhaps [sysbox]? Reports and/or changes are welcome. @@ -85,17 +90,20 @@ UNIX binary utilities. PRs are welcome to make the project work on MacOS, if it does not already. Apart from the standard UNIX binary utilities, you will need the following -installed on the host. Installation is easiest on Fedora +installed on the host. Installation is easiest on Fedora (see original [issue] +for installation on older versions). + `curl` + `jq` -+ `buildah` -+ `krunvm` (and its [requirements]) ++ A compatible runtime, i.e. either: + + `krun` and [`podman`][podman]. + + `krunvm`, its [requirements] and `buildah` -Note: You do not need `podman`. +Note: When opting for `krunvm`, you do not need `podman`. [built]: ./.github/workflows/ci.yml [requirements]: https://github.com/containers/krunvm#installation + [issue]: https://github.com/efrecon/gh-runner-krunvm/issues/22 ## GitHub Token @@ -129,9 +137,9 @@ permissions. The [orchestrator] creates as many loops of ephemeral runners as requested. These loops are implemented as part of the [runner.sh][runner] script: the script will create a microVM based on the default image (see below), memory and -vCPU requirement. It will then start that microVM using `krunvm` and that will -start an (ephemeral) GitHub [runner][self]. As soon as a job has been executed -on that runner, the microVM will end and a new will be created. +vCPU requirement. It will then start that microVM using `krunvm` or `podman` and +the VM will start an (ephemeral) GitHub [runner][self]. As soon as a job has +been executed on that runner, the microVM will end and a new will be created. The OCI image is built in two parts: diff --git a/lib/common.sh b/lib/common.sh index 9fbab23..110c7b6 100644 --- a/lib/common.sh +++ b/lib/common.sh @@ -51,9 +51,33 @@ usage() { } check_command() { + OPTIND=1 + _hard=1 + _warn=0 + while getopts "sw-" _opt; do + case "$_opt" in + s) # Soft check, return an error code instead of exiting + _hard=0;; + w) # Print a warning when soft checking + _warn=1;; + -) # End of options, everything after is the command + break;; + ?) + error "$_opt is an unrecognised option";; + esac + done + shift $((OPTIND-1)) + if [ -z "$1" ]; then + error "No command specified for checking" + fi trace "Checking $1 is an accessible command" if ! command -v "$1" >/dev/null 2>&1; then - error "Command not found: $1" + if is_true "$_hard"; then + error "Command not found: $1" + elif is_true "$_warn"; then + warn "Command not found: $1" + fi + return 1 fi } @@ -70,11 +94,6 @@ get_env() ( fi ) -run_krunvm() { - debug "Running krunvm $*" - buildah unshare krunvm "$@" -} - tac() { awk '{ buffer[NR] = $0; } END { for(i=NR; i>0; i--) { print buffer[i] } }' } diff --git a/lib/microvm.sh b/lib/microvm.sh new file mode 100644 index 0000000..4700de3 --- /dev/null +++ b/lib/microvm.sh @@ -0,0 +1,296 @@ +#!/bin/sh + +# This implements an API to manage microVMs using krunvm and podman. + + +# Runtime to use for microVMs: podman+krun, krunvm. When empty, it will be +# automatically selected. +: "${KRUNVM_RUNNER_RUNTIME:=""}" + +# Directory used for VM->PID mapping when using krunvm +: "${KRUNVM_RUNNER_STORAGE:=""}" + +# Run krunvm with the provided arguments, behind a buildah unshare. +run_krunvm() { + debug "Running krunvm $*" + buildah unshare krunvm "$@" +} + +_podman_runtime() { + _runtime=${KRUNVM_RUNNER_RUNTIME#podman+} + [ -z "$_runtime" ] && _runtime="krun" + printf %s\\n "$_runtime" +} + + +# Automatically select a microVM runtime based on the available commands. Set +# the KRUNVM_RUNNER_RUNTIME variable. +_microvm_runtime_auto() { + if check_command -s -- krun; then + check_command podman + KRUNVM_RUNNER_RUNTIME="podman+krun" + elif check_command -s -- krunvm; then + check_command buildah + KRUNVM_RUNNER_RUNTIME="krunvm" + else + error "No suitable runtime found. Please install 'krun' or 'krunvm'" + fi + info "Automatically selected $KRUNVM_RUNNER_RUNTIME to handle microVMs" +} + + +# Set the microVM runtime to use. When no argument is provided, it will try to +# automatically detect it based on the available commands. +# shellcheck disable=SC2120 +microvm_runtime() { + # Pick runtime provided as an argument, when available. + [ "$#" -gt 0 ] && KRUNVM_RUNNER_RUNTIME="$1" + + # When no runtime is provided, try to auto-detect it. + [ -z "${KRUNVM_RUNNER_RUNTIME:-""}" ] && _microvm_runtime_auto + + # Enforce podman+krun as soon as anything starting podman is provided. + [ "${KRUNVM_RUNNER_RUNTIME#podman}" != "$KRUNVM_RUNNER_RUNTIME" ] && KRUNVM_RUNNER_RUNTIME="podman+krun" + + # Check if the runtime is valid. + case "$KRUNVM_RUNNER_RUNTIME" in + podman*) + check_command podman + check_command "$(_podman_runtime)" + ;; + krunvm) + check_command krunvm + check_command buildah + if [ -z "$KRUNVM_RUNNER_STORAGE" ]; then + KRUNVM_RUNNER_STORAGE="$(mktemp -d)" + fi + ;; + *) + error "Unknown microVM runtime: $KRUNVM_RUNNER_RUNTIME" + ;; + esac +} + + +# List all microVMs. +microvm_list() { + [ -z "$KRUNVM_RUNNER_RUNTIME" ] && microvm_runtime + case "$KRUNVM_RUNNER_RUNTIME" in + podman*) + podman ps -a --format "{{.Names}}" + ;; + krunvm) + run_krunvm list + ;; + esac +} + + +_krunvm_create() { + KRUNVM_RUNNER_IMAGE=$1 + verbose "Creating microVM '${KRUNVM_RUNNER_NAME}', $KRUNVM_RUNNER_CPUS vCPUs, ${KRUNVM_RUNNER_MEM}M memory" + # Note: reset arguments! + set -- \ + --cpus "$KRUNVM_RUNNER_CPUS" \ + --mem "$KRUNVM_RUNNER_MEM" \ + --dns "$KRUNVM_RUNNER_DNS" \ + --name "$KRUNVM_RUNNER_NAME" + if [ -n "$KRUNVM_RUNNER_VOLS" ]; then + while IFS= read -r mount || [ -n "$mount" ]; do + if [ -n "$mount" ]; then + set -- "$@" --volume "$mount" + fi + done < "${KRUNVM_RUNNER_STORAGE}/${KRUNVM_RUNNER_NAME}.pid" + verbose "Started microVM '$KRUNVM_RUNNER_NAME' with PID $KRUNVM_RUNNER_PID" + wait "$KRUNVM_RUNNER_PID" + KRUNVM_RUNNER_PID= + ;; + *) + error "Unknown microVM runtime: $KRUNVM_RUNNER_RUNTIME" + ;; + esac +} + + +microvm_wait() { + [ -z "$KRUNVM_RUNNER_RUNTIME" ] && microvm_runtime + + if [ "$#" -lt 1 ]; then + error "No name specified" + fi + + case "$KRUNVM_RUNNER_RUNTIME" in + podman*) + podman wait "$1";; + krunvm) + KRUNVM_RUNNER_PID=$(cat "${KRUNVM_RUNNER_STORAGE}/$1.pid") + if [ -n "$KRUNVM_RUNNER_PID" ]; then + # shellcheck disable=SC2046 # We want to wait for all children + waitpid $(ps_tree "$KRUNVM_RUNNER_PID"|tac) + KRUNVM_RUNNER_PID= + fi + ;; + *) + error "Unknown microVM runtime: $KRUNVM_RUNNER_RUNTIME" + ;; + esac +} + + +# NOTE: we won't be using this much, since we terminate through the .trm file in most cases. +microvm_stop() { + [ -z "$KRUNVM_RUNNER_RUNTIME" ] && microvm_runtime + + if [ "$#" -lt 1 ]; then + error "No name specified" + fi + + case "$KRUNVM_RUNNER_RUNTIME" in + podman*) + # TODO: Specify how long to wait between TERM and KILL? + podman stop "$1";; + krunvm) + KRUNVM_RUNNER_PID=$(cat "${KRUNVM_RUNNER_STORAGE}/$1.pid") + if [ -n "$KRUNVM_RUNNER_PID" ]; then + kill_tree "$KRUNVM_RUNNER_PID" + # shellcheck disable=SC2046 # We want to wait for all children + microvm_wait "$1" + rm -f "${KRUNVM_RUNNER_STORAGE}/$1.pid" || true + fi + ;; + *) + error "Unknown microVM runtime: $KRUNVM_RUNNER_RUNTIME" + ;; + esac +} + +microvm_delete() { + [ -z "$KRUNVM_RUNNER_RUNTIME" ] && microvm_runtime + + if [ "$#" -lt 1 ]; then + error "No name specified" + fi + + case "$KRUNVM_RUNNER_RUNTIME" in + podman*) + verbose "Removing container '$1'" + podman rm -f "$1";; + krunvm) + verbose "Removing microVM '$1'" + run_krunvm delete "$1" + ;; + *) + error "Unknown microVM runtime: $KRUNVM_RUNNER_RUNTIME" + ;; + esac +} + +microvm_pull() { + [ -z "$KRUNVM_RUNNER_RUNTIME" ] && microvm_runtime + + if [ "$#" -lt 1 ]; then + error "No image name specified" + fi + + verbose "Pulling image(s) '$*'" + case "$KRUNVM_RUNNER_RUNTIME" in + podman*) + podman pull "$@";; + krunvm) + buildah pull "$@";; + *) + error "Unknown microVM runtime: $KRUNVM_RUNNER_RUNTIME" + ;; + esac +} + +microvm_cleanup() { + [ -n "$KRUNVM_RUNNER_STORAGE" ] && rm -rf "$KRUNVM_RUNNER_STORAGE" +} diff --git a/orchestrator.sh b/orchestrator.sh index 424b468..e2029c4 100755 --- a/orchestrator.sh +++ b/orchestrator.sh @@ -31,6 +31,8 @@ ORCHESTRATOR_ROOTDIR=$( cd -P -- "$(dirname -- "$(command -v -- "$(abspath "$0") # shellcheck source=lib/common.sh . "$ORCHESTRATOR_ROOTDIR/lib/common.sh" +# shellcheck source=lib/microvm.sh +. "$ORCHESTRATOR_ROOTDIR/lib/microvm.sh" # Level of verbosity, the higher the more verbose. All messages are sent to the # stderr. @@ -56,11 +58,14 @@ ORCHESTRATOR_ISOLATION=${ORCHESTRATOR_ISOLATION:-"1"} # has been turned on. ORCHESTRATOR_SLEEP=${ORCHESTRATOR_SLEEP:-"30"} +# Runtime to use when managing microVMs. +ORCHESTRATOR_RUNTIME=${ORCHESTRATOR_RUNTIME:-""} + # shellcheck disable=SC2034 # Used in sourced scripts -KRUNVM_RUNNER_DESCR="Run krunvm-based GitHub runners on a single host" +KRUNVM_RUNNER_DESCR="Run libkrun-based GitHub runners on a single host" -while getopts "s:Il:n:p:vh-" opt; do +while getopts "s:Il:n:p:R:vh-" opt; do case "$opt" in s) # Number of seconds to sleep between microVM creation at start, when no isolation ORCHESTRATOR_SLEEP="$OPTARG";; @@ -72,6 +77,8 @@ while getopts "s:Il:n:p:vh-" opt; do ORCHESTRATOR_RUNNERS="$OPTARG";; p) # Prefix to use for the VM name ORCHESTRATOR_PREFIX="$OPTARG";; + R) # Runtime to use when managing microVMs, podman+krun or krunvm. Empty==first available + ORCHESTRATOR_RUNTIME="$OPTARG";; v) # Increase verbosity, will otherwise log on errors/warnings only ORCHESTRATOR_VERBOSE=$((ORCHESTRATOR_VERBOSE+1));; h) # Print help and exit @@ -99,23 +106,25 @@ cleanup() { # shellcheck disable=SC2086 # We want to wait for all pids waitpid $ORCHESTRATOR_PIDS - if run_krunvm list | grep -qE "^${ORCHESTRATOR_PREFIX}-"; then - while IFS= read -r vm; do + while IFS= read -r vm; do + if [ -n "$vm" ]; then verbose "Removing microVM $vm" - run_krunvm delete "$vm" - done < "${RUNNER_ENVIRONMENT}/${RUNNER_ID}.trm" - elif [ -n "$RUNNER_PID" ]; then - kill_tree "$RUNNER_PID" - fi - if [ "$RUNNER_PID" ]; then - # shellcheck disable=SC2046 # We want to wait for all children - waitpid $(ps_tree "$RUNNER_PID"|tac) - else - warning "No PID to wait for" - fi - elif [ -n "$RUNNER_PID" ]; then - kill_tree "$RUNNER_PID" - # shellcheck disable=SC2046 # We want to wait for all children - waitpid $(ps_tree "$RUNNER_PID"|tac) - fi - elif [ -n "$RUNNER_PID" ]; then - kill_tree "$RUNNER_PID" - # shellcheck disable=SC2046 # We want to wait for all children - waitpid $(ps_tree "$RUNNER_PID"|tac) + # Request for termination through .trm file, whenever possible. Otherwise, + # just stop the VM. + if [ -n "$RUNNER_ENVIRONMENT" ] \ + && [ -f "${RUNNER_ENVIRONMENT}/${1}.tkn" ] \ + && [ -n "${RUNNER_SECRET:-}" ]; then + verbose "Requesting termination via ${RUNNER_ENVIRONMENT}/${1}.trm" + printf %s\\n "$RUNNER_SECRET" > "${RUNNER_ENVIRONMENT}/${1}.trm" + microvm_wait "${RUNNER_PREFIX}-$1" + else + microvm_stop "${RUNNER_PREFIX}-$1" fi } + cleanup() { trap '' EXIT - if [ -n "${RUNNER_PID:-}" ]; then - vm_terminate - fi + if [ -n "${RUNNER_ID:-}" ]; then + vm_terminate "$RUNNER_ID" vm_delete "$RUNNER_ID" fi + + microvm_cleanup } trap cleanup EXIT @@ -339,9 +334,19 @@ trap cleanup EXIT iteration=0 while true; do + # Prefetch, since this might take time and we want to be ready to count away + # download time from the termination setting. + microvm_pull "$RUNNER_IMAGE" + + # Terminate in xx seconds. This is mostly used for demo purposes, but might + # help keeping the machines "warm" and actualised (as per the pull above). + if [ -n "$RUNNER_TERMINATE" ]; then + verbose "Terminating runner in $RUNNER_TERMINATE seconds" + sleep "$RUNNER_TERMINATE" && cleanup & + fi + RUNNER_ID="${loop}-$(random_string)" - vm_create "${RUNNER_ID}" - vm_start "${RUNNER_ID}" + vm_run "${RUNNER_ID}" vm_delete "${RUNNER_ID}" RUNNER_ID=