Skip to content

Latest commit

 

History

History
535 lines (426 loc) · 21.2 KB

DEPLOY.org

File metadata and controls

535 lines (426 loc) · 21.2 KB
Raw

Deployment (rolling provisioning and orchestration)

Table of Contents

Introduction

Rolling deployment and orchestration is a holy grail in devop-land. In this document we pursue deployment with Guix, configuration with Guix and orchestration with Guix (WIP). This will be done with bare metal, VMs and containers. We manage enough machines to want to automate installation and updates and soon it may be necessary to build a compute cluster again. So, it is time to get ready.

In the past I have attempted multiple ways to define systems. Around 2000 I got exposed to Cfengine which aims to bring system deployment under control. Later I tried Puppet, Chef and others. Finally I rolled my own Cfengine clone named CfRuby and the even have a rewrite in a simpler tool called deploy which I am using today on our machines. All these systems suffer because they have ‘state’ in two places (/etc can be considered its own state) and these tools overwrite targets destructively. With Guix we can do better than that.

Guix system definition

GNU Guix has a method for defining systems (see operating-system). The idea is that deployment is integrated at the system definition stage.

Guix deploy

Guix deploy was recently introduced (2019) by Jakob L. Kreuze. The short of it is that a GNU Guix system that has openssh can be updated using remote ssh commands. Interestingly it shares the infrastructure Guix already provided for defining systems (see operating-system). An example of ‘guix deploy’ by the author can be found here/

If you read the code for gnu/system.scm in the Guix source tree you can see a system update leads to the creation of a new generation of /etc directory with files contained (just search for ‘sudo’ to see how it is done).

(define %sudoers-specification
  ;; Default /etc/sudoers contents: 'root' and all members of the 'wheel'
  ;; group can do anything.  See
  ;; <http://www.sudo.ws/sudo/man/1.8.10/sudoers.man.html>.
  (plain-file "sudoers" "\
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL\n"))

where plain-file writes into the current ‘build’ directory. Parametrizing plain-file also works (of course)

(define (local-host-aliases host-name)
  "Return aliases for HOST-NAME, to be used in /etc/hosts."
  (string-append "127.0.0.1 localhost " host-name "\n"
                 "::1       localhost " host-name "\n"))

In fact, Guix is a champion of abstraction and comes with a load of methods to manage system files.

Also note that Guix transactional goodness shows up here! Not only do you track changes through git, you can actually roll back to a previous version of /etc. This is very different from the way Cfengine, Puppet and others work. With those tools, to guarantee consistency, you always have to rebuild the full thing from scratch. With GNU Guix to roll back is only the switch of a symlink which takes seconds.

A similar system can be used to orchestrate containers as explained here.

Now this is all very nice if you are running a 100% Guix distribution. In contrast, all my machines run Debian with Guix on top (though I have pure Guix containers). GNU Guix as a distribution certainly has its attractions, but the reality is that I share machines with others and they are not ready to make a 100% shift. The cool thing about Guix is that you can shift gradually because Guix packages run on any Linux distribution, whether it is Red Hat, SuSE or Debian/Ubuntu/Mint. Guix rocks in that way and OpenSuSE even has built-in Guix package support nowadays.

Guix partial deploy using packages

Rather than control the whole /etc, what we want is partial system configuration at the package level. Say you want to configure an nginx web server, you can define a new package that depends on nginx that contains the configuration files. On myserver the new package nginx-myserver simply creates the contents of an /etc/nginx directory inside the store relevant to the package.

The only tricky aspect is to tell nginx where to find its configuration and (possibly) add that to systemd. Guix does not write files outside the store in /gnu/store. Nevertheless it can create profiles which are paths that symlink into the store. These profiles can be anywhere - even /etc. So maybe Guix can symlink /etc/nginx. And we can setup

guix package -i nginx-myserver -p /etc/nginx

Similarly Guix can symlink /etc/systemd/system for a systemd configuration (even if systemd was installed by Debian though you will have to check carefully what is installed!). We can give full control to Guix for packages that conform to these rules. Setting a profile directly in /etc may not be a great idea because not only it will be cluttered by generations of nginx, but also the profiles contain other dirs, such as bin, lib and shared. Probably better to install the profile in, say, /usr/local/guix-etc and run a script to link up the latest configuration.

Next we want to parametrize so we can target multiple servers in a simple way. This should work fine in standard Guix packages too by using named packages for each server and using package inheritance where applicable. Guix parametrized packages may help too, even though it is currently a proposal. Guix manifests can be used to define bundles and Guix channels can be used to embed these packages so they can be installed by Guix. Now the script becomes the install of a manifest followed by updating symlinks that are not generations:

#!/bin/bash
guix package --manifest=myserver -p /usr/local/guix-etc
for link in /usr/local/guix-etc/etc/* ; do
  # skip generation link
  # force link -> /etc/link
done

Funny thing to note: except for running the final script the infrastructure is pretty much there! Another thing to notice is that this method is not limited to machine installs, it can also be used to configure a HOME directory.

Creating a configuration package

After installing Guix on a Debian using the Guix binary install (and following the instructions) you should have a running Guix daemon with group guixbuild and you should have added a key with guix archive.

Next, I create a channel by telling Guix to use my git repo

To easily use the packages from this repo, simply add it to your `channels` list in ~/.config/guix/channels.scm as described here:

(cons*
  (channel
    (name 'gn-bioinformatics)
    (url "https://git.genenetwork.org/pjotrp/guix-bioinformatics.git")
    (branch "master"))
  %default-channels)

and run

guix pull

My package defined as

; Configure a default version of the nginx web server
(define-public nginx-config
  (let ((commit "e2ac61bfa472f23eb8e0c6863395a79c94a3d68a")
        (revision "1"))
    (package
     (name "nginx-gn-config")
     (version (git-version "0.0.1" revision commit))
     (source (origin
              (method git-fetch)
              (uri (git-reference
                    (url "http://git.genenetwork.org/pjotrp/guix-bioinformatics.git")
                    (commit commit)))
              (file-name (git-file-name name version))
              (sha256
               (base32
                "1pczs7farkcklvh96q1psjgv86mlwv93c3lzmc1mwp9m299g4qdr"))))
     (build-system trivial-build-system)
     (native-inputs `(("unzip" ,unzip)
                      ("source" ,source)))
     (propagated-inputs `(("nginx" ,nginx)))
     (arguments
      `(#:modules ((guix build utils))
        #:builder
        (begin
          (use-modules (guix build utils))
          (let ((target (string-append (assoc-ref %outputs "out")
                                       "/etc/nginx"))
                (nginx-etc (string-append (assoc-ref %build-inputs "nginx")
                                            "/share/nginx/conf")))
            (mkdir-p target)
            ; copy up,, original nginx configuration /gnu/store/nginx-ver/share/nginx/conf/*
            ; (copy-recursively nginx-etc target)
            (copy-file (string-append nginx-etc "/nginx.conf")
                       (string-append target "/nginx.conf"))
            #t))))
     (home-page "http://git.genenetwork.org/pjotrp/guix-bioinformatics")
     (synopsis "Nginx configuration")
     (description "None.")
     (license license:expat))))

should be visible

guix package -A nginx
nginx          1.17.6 out /export/local/wrk/iwrk/opensource/guix/guix-master/gnu/packages/web.scm:211:2
nginx-gn-config 0.0.1 out ../guix-bioinformatics/gn/deploy/machines.scm:22:2

which can be installed with using a

guix package -i nginx-gn-config -p ~/opt/myserver
tree ~/opt/myserver
/home/wrk/opt/myserver
├── etc
│   ├── nginx -> /gnu/store/zl7plvii6979gxxm7dwqynnmxamdygal-nginx-gn-config-0.0.1-1.e2ac61b/etc/nginx
│   └── profile
├── manifest
├── sbin -> /gnu/store/6q44kjf59rgkvn0ip8m0454ybszhjpy0-nginx-1.17.6/sbin
└── share
    ├── doc -> /gnu/store/6q44kjf59rgkvn0ip8m0454ybszhjpy0-nginx-1.17.6/share/doc
    ├── info -> /gnu/store/z3jphxl8isvsgylxsyrnmjjlqd7p9lkx-info-dir/share/info
    ├── man
    │   ├── index.db -> /gnu/store/9ym9wi3b4imjlqc2mby813q9dsa43k5x-manual-database/share/man/index.db
    │   └── man8 -> /gnu/store/6q44kjf59rgkvn0ip8m0454ybszhjpy0-nginx-1.17.6/share/man/man8
    └── nginx -> /gnu/store/6q44kjf59rgkvn0ip8m0454ybszhjpy0-nginx-1.17.6/share/nginx

and

tree /gnu/store/zl7plvii6979gxxm7dwqynnmxamdygal-nginx-gn-config-0.0.1-1.e2ac61b/etc/nginx
/gnu/store/zl7plvii6979gxxm7dwqynnmxamdygal-nginx-gn-config-0.0.1-1.e2ac61b/etc/nginx
├── fastcgi.conf
├── fastcgi_params
├── koi-utf
├── koi-win
├── mime.types
├── nginx.conf
├── scgi_params
├── uwsgi_params
└── win-utf

Holds the default configuration which can be started with

~/opt/myserver/sbin/nginx -c ~/opt/myserver/etc/nginx/nginx.conf

or by symlinking the conf file from /etc. You get the idea. In this case we can create a wrapper script that would pick up this configuration file and there is no need to symlink from /etc at all.

To plug in our own nginx.conf hosted in our git repo we can replace

(nginx-etc (string-append (assoc-ref %build-inputs "nginx") "/share/nginx/conf"))

with something like

(nginx-etc (string-append (assoc-ref %build-inputs "source") "/gn/deploy/myserver/nginx.conf"))

but that won’t make use of GNU Guix machine configuration capabilities. So we need to plug that in now.

Note that to develop the package it may be useful to use the GUIX_PACKAGE_PATH instead of a channel and guix pull and build against local checkout

env GUIX_PACKAGE_PATH=./guix-bioinformatics guix build nginx-gn-config -K

In this section we show how you can configure a package by creating another package. This may be sufficient for many cases. On a path to a full Guix solution it would be interesting if we can use the system configuration stuff that comes with GNU Guix.

Provisioning a container

Guix has a command for creating containers (which is the same for VM, a bare machine and even a Docker image) with system settings:

guix system container myserver.scm --network

(note that this does not require Docker!). Building the container returns a script which you can run. E.g.

/gnu/store/v056qnyvaz8rrjbhhk64xmlw302s2vn5-run-container

it says

populating /etc from /gnu/store/i2cjw7m1wfiil11h5jg6iipdcrmz6wia-etc...

and if you included a shell you can login with something like

nsenter -a -t 23718 sh

and with network and ssh configured so if you set up an account you can do

ssh -p 2222 pjotr@localhost

Tools are symlinked from the store in

/run/current-system/profile/bin/

which runs a container in a Linux namespace, a much lighter weight alternative to Docker. You can verify that the container has full access to the store, has a special /etc and does not expose the underlying $HOME etc. Note: don’t run ‘guix’ in the namespace, only in full containers or VM because the store is shared even though guix gc --verify=repair may fix it.

Just for the heck of it I added ruby

(packages (append (list
                   screen ruby)
                  %base-packages))
pjotr@komputilo ~$ which ruby
/run/current-system/profile/bin/ruby
pjotr@komputilo ~$ ruby -v
ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]

Interestingly the configuration file myserver.scm is the same for a container as for a VM or a bare bones machine. GNU Guix strips the additional stuff that you need to run one of those. You can move between the three targets freely.

Note that, after testing and if you need it, with the same myserver.scm you can create a Docker container or a (USB) disk image.

guix system docker myserver.scm

Guix system

How does guix system work? One of the great features of GNU Guix is that it is all Scheme Lisp. This means it is not too hard to digest what is going on.

Above

populating /etc from /gnu/store/i2cjw7m1wfiil11h5jg6iipdcrmz6wia-etc...

is done by gnu/build/activation.scm. Essentially the procedure activate-etc symlinks from the profile

(rm-f "/etc/ssl")
(symlink "/run/current-system/profile/etc/ssl" "/etc/ssl")

This is a standard Guix profile. In the next step add ssh and nginx and have it list to respectively port 2222 and port 8080 with this commit. Now you point the webserver on your machine to http://localhost:8080/.

Note that for the ssh service the procedure openssh-activation creates the /etc/ssh directory and its contents, including key generation. Meanwhile sshd is configured to start up with a configuration in

cat /gnu/store/wf6zrbx1s9b5kidb489r5xx40zhp2w2v-sshd_config
# Generated by 'openssh-service'.
Port 2222
PermitRootLogin no
PermitEmptyPasswords no
PasswordAuthentication yes
PubkeyAuthentication yes
X11Forwarding no
AllowAgentForwarding yes
AllowTcpForwarding yes
GatewayPorts no
PidFile /var/run/sshd.pid
ChallengeResponseAuthentication no
UsePAM yes
PrintLastLog yes
LogLevel INFO
AuthorizedKeysFile  .ssh/authorized_keys .ssh/authorized_keys2 /etc/ssh/authorized_keys.d/%u
Subsystem       sftp    internal-sftp

Similarly the /etc/hosts file is created by operating-system-etc-service. In other words, /etc in the machine is a combination of symlinks and generated/copied files depending on what it relates to. And services can directly invoke paths such as the sshd configuration.

The components are here. To benefit from services we can use a machine definition that creates a profile that we can use in a container on any linux distribution, including Debian.

Guix partial deploy on Debian

In this section we want to use Guix system definitions outside Guix containers - i.e., for software running on bare metal on top of a non-Guix linux distribution, such as Debian, Ubuntu or Fedora. One thing you should understand is that GNU Guix builds through a daemon with restricted access. This is by design. The guix-daemon can only build, write and install software hosted in /gnu/store. For containers and VM this is not a problem because it actually builds those in the store for you to run!

On Debian the guix-daemon can not reach outside the store to, for example, /etc. The only thing it can do - as noted earlier - is create a so called profile that consists of symlinks into the store.

What we can do is use GNU Guix to create a machine profile and run a program after that hooks the profile components into Debian, similarly to what we did for packages earlier.

All we need to do is tell GNU Guix to create a machine profile (which as it happens is listed in the container runner (i.e., /gnu/store/rrqj1pzyw2ajk1gh9xrd29lnda75msf3-run-container), make it available and make sure it does not garbage collected, and then symlink from /etc.

GNU Shepherd

Rather than using systemd it may be an option to use GNU Shepherd which comes with Guix. It is possible to fire up the shepherd next to (or even by) systemd since these are independent control daemons (shepherd can even run without root privilege). Having both running is an option to slowly migrate existing services to our new partial deploy system. Shepherd can start and stop services, resolve orchestration dependencies, and even be a watchdog.

Starting shepherd as a normal user it created a stub file ~/.config/shepherd/init.scm containing

;; init.scm -- default shepherd configuration file.
(register-services)
;; Send shepherd into the background
(action 'shepherd 'daemonize)
;; Services to start when shepherd starts:
(for-each start '())

Based on Guix examples in the /gnu/services directory and docs I wrote a small service that fires up a web service on a non-privileged port

With Shepherd we can consider supporting system services in containers as described here.

Using Shepherd on Debian

We run Shepherd in user land to deploy software on bare metal and in Guix containers. A shepherd user is created and the shepherd software is running from a Guix profile, e.g. /home/shepherd/.guix-profile/bin.

shepherd@penguin2:~$ herd status
Started:
 + bnw                                                                                                      + covid19-pubseq
 + genenetwork1                                                                                             + hrdp-project
 + ipfs                                                                                                     + mcron
 + power                                                                                                    + ratspub
 + rn6app
 + root
 + singlecell                                                                                              Stopped:
 - gitea
 - virtuoso
One-shot:
 * test-upgrade

Classes

One thing I might also like to have is a concept of classes such as Cfengine and Cfruby incorporated. A typical class can be a mailserver or a webserver. There can also be ssh and firewall classes. To make a machine we could state

myserver: mailserver ssh firewall

which would configure the machine. Compound classes may exist so

mailclient: postfix mutt
mylaptop: mailclient

where mylaptop expands to class ‘mailclient postfix mutt’. Essentially classes are a simple list of symbols that expand and describe the machine.

Conclusion

Where Cfengine makes it a point to talk about ‘convergence’ of system configuration to a ‘sane state’, GNU Guix takes a different approach and talks about a ‘functional’ paradigm where package dependencies, deployment and system configuration are treated as one. A Guix installation is a defined output defined by a function and it is always transactional, predictable and ‘sane’. Provided the inputs are well defined. There is no concept of mixing state between the provisioning system and what is already in /etc, at least for the packages that Guix manages.

Here I am mixing Debian with Guix packages to define a system configuration in functional way. It is less rigorous than a pure Guix installation because Debian itself is not rigorous, i.e., a Debian system does not have a well-defined state (for one, it matters at what time you install a piece of software). By applying the techniques presented here, an existing Debian/Ubuntu/Red hat/SuSE installation can be gradually morphed into a Guix one, gaining more control over dependencies and configuration one package at a time!

Notes

See https://github.com/mbakke/ganeti-instance-guix