IPv6-friendly Kubernetes cluster running on commodity hardware in a locker (yes, an actual locker) in our basement.
This repository contains Infrastructure as Code (IaC) and GitOps config files for managing my hobby cluster in the basement. Inspired by popular repos like toboshii/home-ops, with a few additional considerations:
- 🛠️ Unconventional hardware: As much as I enjoy automating the software infrastructure, I also really like building custom hardware to power it all. I spend maybe half the time in front of the ⌨️ keyboard and half the time using 🪚🪛 power tools.
- 🌳 Low footprint: All of the nodes are either old machines I am no longer using, or used machines I bought for next to nothing. Many use passive cooling, and there are a fair bit of x86 (mostly i686) CPUs involved.
To use a custom domain, there is a one-time bootstrapping procedure to go through.
I'll be going with dorn.haus, but any domain should do.
I'll be using Cloudflare services later on, most notably DNS, so I always start by registering the domain with Cloudflare.
An additional perk with Cloudflare is the free email forwarding of wildcard addresses, allowing incoming emails without having to register with at enail provider or managing an exchange server.
But the main reason for registering early is to get an SSL certificate. My ISP likes to block incoming traffic on port 80 from time to time, making it impossible to get/renew certificates using Certbot with the HTTP challenge.
An easy way to get started is to manually get an initial certificate:
certbot certonly --preferred-challenges dns --manual -d dorn.haus
Then manually add & remove the TXT record in the Cloudflare UI.
I then set up a simple Nginx reverse-proxy and NAT port 443. This will be needed to serve the OpenID challenge via Keycloak (next step).
Fire up a Keycloak development server using Podman to create an initial user, [email protected]. I do this at home
while NAT'ing myself to the outside world, as well as making sure my LetsEncrypt cert is still functional.
export PASSWORD="$(pwgen -1sy 12)"
echo "admin password: $PASSWORD (temporary)"
podman run -p 8080:8080 \
docker run \
--name keycloak \
-p 8443:8443 \
-p 9000:9000 \
-e KEYCLOAK_ADMIN=admin \
-e KEYCLOAK_ADMIN_PASSWORD="$PASSWORD" \
-e PROXY_ADDRESS_FORWARDING=true \
-v /path/to/certs:/etc/certs \
quay.io/keycloak/keycloak start \
--proxy-headers forwarded \
--hostname=https://dorn.haus/keycloak/ \
--https-certificate-file=/etc/certs/cert.pem \
--https-certificate-key-file=/etc/certs/privkey.pem \
--log-level=INFO \
--verbose
Next, configure WebFinger in Nginx (see bootstrap/nginx.conf). Create a new Keycloak realm, add a user (e.g.
[email protected]), and add the Tailscale client. The Tailscale Client ID & secret will be needed when connecting.
Once connected, we can sign in to Tailscale using Keycloak as the OIDC provider, and create a personal tailnet for our domain, for free.
Currently the machines in the cluster are connected to to the router that Swisscom provides us, through cheap 1 Gbps switches that only do L2 forwarding. This router advertises two IPv6 prefixes:
- A
scope global,dynamicprefix that belongs to the2000::/3range. - A
scope globalstatic prefix in thefd00::/8range. This appears to be the prefixfdaa:bbcc:ddee:0/64on these modems.
The router has IPv6 pinholing configured to access the load balancers from the outside. Cloudflare sits in front of the them and provides IPv4 connectivity.
For now, most networks run in dual-stack mode, all networks being part of the 10./8 & fd10::/8 subnets, which are
both routable locally.
The easiest way to get the required dependencies is to have nix and direnv configured. Entering the repo will
execute the .envrc file, which in turn will activate devenv to build the required dependencies.
Without direnv, one would need to manually run devenv shell to enter the development shell.
Much of this was inspired by a number of similar repos:
There is an existing repository where I already have most of these configs, however it contains various secrets that are not properly extracted out. This is an attempt to migrate exsting configs and redact any secrets.