Hanlon-Microkernel v3.0.1

This release contains bug fixes as follows:

• Minor change in Dockerfile to specify the version of Facter to use when building the microkernel container

First Docker-based Microkernel release

This release of the Hanlon-Microkernel project represents the first (major) release of a Docker-based Hanlon Microkernel. Prior to this release, the Hanlon Microkernel was essentially a customized version of a Tiny Core Linux ISO (with all of the complexities that are necessary to give users the ability to rebuild their own customized (remastered) version of this ISO. With the changes in this release, the Hanlon Microkernel project has been greatly simplified (and, we hope, been made much simpler for users to interact with and contribute to).

The Microkernel (Docker) image that can be built from this repository will, by its very nature, be much simpler to add to and maintain. Building a new image is as simple as running a standard docker build command from the repository, and we've even provided examples of how to do this (along with a discussion of the structure of the Dockerfile in this project in the new Building-a-MK-Container.md file that appears in the root directory of this repository. In short, you can build a new instance of the Microkernel (Docker) image using a command like the following:

docker build -t new_mk_image:3.0.0 .

which will create a tagged version of the Microkernel (it's important to tag the image with a semantic version number). That image can then save as a tarfile using a command that looks something like this:

docker save new_mk_image > new_mk_image_save.tar

Once saved, you can then use that tarfile (or it's bzipped or gzipped form) as an input when adding a new Microkernel instance to Hanlon. Details of the new hanlon image add ... command (or it's RESTful equivalent) are available in the Hanlon project's Wiki.

Of course, the biggest advantage to the new Docker-based Microkernel is that we can now provide a standard Hanlon Microkernel (tagged by version) directly to users via DockerHub. Details on this will follow in a subsequent blog posting (planned to be pushed out shortly).

Finally, as an aside, if you'd like to use the git describe ... command to tag your docker image during the docker build process, the command you'd use would look something like this:

docker build -t new_mk_image:`git describe --tags --dirty --always | sed -e 's@-@_@' | sed -e 's/^v//'` .

That will construct an appropriate tag for your Microkernel (Docker) image based on the latest tag assigned to the repository being used to build the Microkernel image, the number of commits made to the repository since that tag was assigned, the current commit ID of the repository (if there are commits that have been made since the repository has been tagged), and even a -dirty suffix if there are uncommitted changes in the repository at the time that the Microkernel image is being built.

Bug Fixes for recent KVM releases and newer TCL builds

The changes in this release patch an error that was detected when the v2.0.0 Microkernel was used to boot a recent KVM instance by one of our users (the sudo dmidecode -s system-uuid returned multi-line input on newer versions of KVM, resulting in an error that prevented the Microkernel from checking in with Hanlon successfully). During the process of rebuilding a test Microkernel instance, we realized that location of the debian_wheezy_main_i386_Packages.gz file that we download from the Tiny Core Linux site during the bundle build process had changed. This release also resolves that issue (without that fix, the nodes are never assigned automatic tags by Hanlon because the lshw utility is not installed during the bundle build process).

A major update to the Microkernel kernel

This release of the Hanlon-Microkernel has a number of improvements over previous releases; among them are the following new features/enhancements:

• We've modified the build process so that the new Microkernel is based on a much more recent version of TinyCore Linux; previous releases were based on TCL-4.x builds which were in turn based on a 3.0.x version of the Linux kernel, whereas this release is based on the latest, TCL-5.4, builds which are in turn based on a 3.8.13 version of the Linux kernel. This should provide much better support for newer hardware directly out of the Microkernel.
• We changed the mechanism for bundling additional kernel modules and TinyCore Linux extensions significantly. Previously, adding kernel modules or additional TCEs that were needed during the Microkernel boot process required that the user customize the bundle build script itself (and perhaps the underlying Microkernel codebase), essentially requiring that the user customize the Hanlon-Microkernel project itself in order to build a custom Microkernel. This version of the Hanlon-Microkernel adds two additional flags that can be used to specify additional kernel modules (-a, --addtnl-kmods=FILE) or TinyCore Linux Extensions (-l, --local-extents=FILE) that each point to a file on the local filesytem (the format for these files is outlined, below) that list the tcz or Debian Wheeze packages containing the kernel modules or extensions to add to the bundle. This means that extending the Microkernel to support new hardware or capabilities at boot time is now much simpler, requiring only the creation of a few files in the local filesystem.
• Added code to support the collection of IPMI-related meta-data by the Microkernel (and reporting of those facts back to the Hanlon server) if the ipmitool is available locally on the Microkernel. This is in preparation for adding the ability to control the power state of a node to the node slice's RESTful API in the near future (provided there is a Baseboard Management Controller, or BMC, attached to the node).
• Fixed a bug in the SMBIOS UUID reported back to Hanlon by the Microkernel. This resulted from the fact that the lshw utility used by the Microkernel to gather this value from the underlying hardware was, on some hardware, reporting an incorrect value for this field (the hexidecimal values representing the underlying SMBIOS UUID were returned in the wrong order by lshw). This resulted in an infinite sequence of boots into the Microkernel as soon as a node that had identified itself using this SMBIOS UUID was bound to an underlying model by a policy, since the SMBIOS UUID value returned by the Microkernel didn't match the (actual) value reported to Hanlon when requesting an iPXE-boot script to use for that node. Fortunately, the resolution was simply to use the dmidecode utilty to obtain this information instead of the lshw utility, since the dmidecode utility didn't suffer from this problem.
• Fixed a bug in the script used when starting the udhcpc client in the Microkernel; with this bug fix in place a default route is now being setup for the network, allowing for access to the world outside of the Hanlon subnet.

With these changes in place, it should now be much easier to build your own custom Microkernel, and the Microkernel that you build should be useable with more modern hardware than was previously supported.

One final note: here, we show an example of the new bundle build process and the two new files that we've created in order to extend the standard Microkernel to support a custom networking card (an Emulex 10Gbe CNA) and the drivers and utilities used to collect IPMI-related facts within the Microkernel and report those facts back to Hanlon:

$ ./build-bundle-file.sh -b additional-build-files/builtin-extensions.lst \
    -m additional-build-files/mirror-extensions.lst \
    -l additional-build-files/local-extensions.lst \
    -a additional-build-files/addtnl-driver-mods.lst -d 2>&1 | tee t.t

as you can see, we've assigned files to the two new options added to this shell script (the last two options in the command); the contents of those files are as follows:

$ cat additional-build-files/local-extensions.lst
extra-extensions/open-vm-tools-modules-3.8.13-tinycore.tcz
extra-extensions/OpenIPMI-2.0.21.tcz
extra-extensions/freeipmi-1.4.5.tcz
extra-extensions/ipmitool-1.8.14.tcz
$ 

and

$ cat additional-build-files/addtnl-driver-mods.lst
extra-driver-files/be2net.tcz               be2net.ko           net/ethernet/emulex/benet/be2net.ko.gz
extra-driver-files/ipmi-kernel-mods.tcz     ipmi_msghandler.ko  char/ipmi/ipmi_msghandler.ko.gz
extra-driver-files/ipmi-kernel-mods.tcz     ipmi_si.ko          char/ipmi/ipmi_si.ko.gz
extra-driver-files/ipmi-kernel-mods.tcz     ipmi_devintf.ko     char/ipmi/ipmi_devintf.ko.gz
extra-driver-files/ipmi-kernel-mods.tcz     ipmi_poweroff.ko    char/ipmi/ipmi_poweroff.ko.gz
extra-driver-files/ipmi-kernel-mods.tcz     ipmi_watchdog.ko    char/ipmi/ipmi_watchdog.ko.gz
$ 

The first file is simply a list of the locations of the additional extensions from the local filesystem that should be included in the bundle (and installed during the bundle boot process). This includes an extension containing the open_vm_tools and several IPMI-related tools that will be used by the Microkernel. The second file contains a list of the kernel modules that should be installed during the bundle boot process. As you can see, the file has a format that lists one line per kernel module, with three fields on each line (the location of the tcz or Debian Wheezy file containing the kernel module in question, the name of the module itself, and the location where it should be installed under the standard filesystem location used in TinyCore Linux (/usr/local/lib/modules/uname -r/kernel/drivers). Once these two files have been defined, and the appropriate files are placed into the directories shown in the files, above:

$ ls extra-extensions/
freeipmi-1.4.5.tcz          ipmitool-1.8.14.tcz.md5.txt  open-vm-tools-modules-3.8.13-tinycore.tcz
freeipmi-1.4.5.tcz.md5.txt  OpenIPMI-2.0.21.tcz          open-vm-tools-modules-3.8.13-tinycore.tcz.list
ipmitool-1.8.14.tcz         OpenIPMI-2.0.21.tcz.md5.txt  open-vm-tools-modules-3.8.13-tinycore.tcz.md5.txt
$ 

and

$ ls extra-driver-files/
be2net.deb  be2net.tcz  be2net.tcz.md5.txt  ipmi-kernel-mods.tcz  ipmi-kernel-mods.tcz.md5.txt
$ 

building the bundle file used to build our customized Hanlon-Microkernel (with support for those Emulex CNAs and support for gathering IPMI-related meta-data and reporting it back to Hanlon) is a simple, one-line command.

Supports new 'shutdown' command

This version of the Hanlon Microkernel includes support for a new poweroff command that can be sent back to the Microkernel by newer versions of Hanlon (post-v1.2.0). This feature was added to support some of the new 'noop' models that are being added to Hanlon; with this new command supported, models can now include a 'powered-off' end state (which is used in the new discover_only model that was just added to Hanlon). As was the case with previous versions of the Hanlon Microkernel, there are three pre-built ISO files provided (below) as part of this release. These three ISOs are intended to be used in three different ways:

• hnl_mk_debug-image.1.1.0.iso -- This (debug) microkernel will auto-login the tc user at the console once it completes booting. It is suitable for developers who are debugging issues with the microkernel boot/checkin process and want quick access to the microkernel once it has completed booting. Remote access (via SSH using the builtin tc password) is also enabled. This microkernel should not be used in production environments. It is meant to be used only for development purposes.
• hnl_mk_dev-image.1.1.0.iso -- This (development) microkernel differs from the debug microkernel only in that the tc user is not automatically logged into the console when the microkernel completes the boot process. Access to the microkernel by the tc user is still allowed (both locally on the console and remotely via SSH) using the builtin tc password. As was the case with the debug microkernel, this microkernel should not be used in production environments. It is meant to be used only for development/test purposes.
• hnl_mk_prod-image.1.1.0.iso -- This (production) microkernel is meant to be used in production environments. Access to the microkernel (both locally via the console and remotely via SSH) are disabled. In fact, in this microkernel the SSH daemon isn't even included as part of the microkernel image.

Please use the ISO that is most suitable for the environment in which you are deploying/using Hanlon (and the Hanlon Microkernel).

Allows for use of (SMBIOS) UUID as a node ID

This release of the Hanlon Microkernel (v1.1.0) is designed to work with the same release (v1.1.0 and later) of Hanlon in order to enable the use of the (SMBIOS) UUID of a node as its the hardware_id. It enables the use of this UUID (and continues to support the old MAC-address-based scheme for identifying nodes) by including two new parameters (the uuid and mac_id) instead of the single parameter used previously (the hw_id) in its checkin and registration requests.

As was the case with the old hw_id parameter, these two new parameters are reported as query string parameters in the URI used for the checkin requests and as fields in a (JSON) hash map that is included in the body of the registration requests. With these changes in place, Hanlon is now able to identify nodes using either scheme (SMBIOS-UUID-based or MAC-address-based) as needed.

Also, as was the case with previous versions of the Hanlon Microkernel, three are three pre-built ISO files provided (below) as part of this release. These three ISOs are intended to be used in three different ways:

• hnl_mk_debug-image.1.1.0.iso -- This (debug) microkernel will auto-login the tc user at the console once it completes booting. It is suitable for developers who are debugging issues with the microkernel boot/checkin process and what quick access to the microkernel once it has completed booting. Remote access (via SSH using the builtin tc password) is also enabled. This microkernel should not be used in production environments. It is meant to be used only for development purposes.
• hnl_mk_dev-image.1.1.0.iso -- This (development) microkernel differs from the debug microkernel only in that the tc user is not automatically logged into the console when the microkernel completes the boot process. Access to the microkernel by the tc user is still allowed (both locally on the console and remotely via SSH) using the builtin tc password. As was the case with the debug microkernel, this microkernel should not be used in production environments. It is meant to be used only for development/test purposes.
• hnl_mk_prod-image.1.1.0.iso -- This (production) microkernel is meant to be used in production environments. Access to the microkernel (both locally via the console and remotely via SSH) are disabled. In fact, in this microkernel the SSH daemon isn't even included as part of the microkernel image.

Please use the ISO that is most suitable for the environment in which you are deploying/using Hanlon (and the Hanlon Microkernel).

First production-ready release of the Hanlon Microkernel

This is the first public release of the Hanlon Microkernel. It is based on the earlier Razor Microkernel, but adds in support for several new features. Specifically, this version of the Microkernel supports configurations where the Hanlon server is deployed to a server listening on a non-standard port or who's RESTful API is listening under a non-standard context and also supports deployments where the Hanlon server IP address is different from the IP address of the associated (PXE-boot) TFTP server. In addition, the three pre-compiled binaries that add these new features (and that the bundle build process depends on) are now downloaded from the project's release area instead of being downloaded from a personal Dropbox repository (which is where they were downloaded from during development of the Hanlon Microkernel) or from the Puppet Labs download site (which is where the Razor Microkernel obtained the 'mk-open-vm-tools.tar.gz' file that it depended on.

For those who are interested, the files downloaded from the project "releases" area by the bundle build proces in this version of the Hanlon-Microkernel are as follows:

• mk-open-vm-tools.tar.gz: this file contains a version of the open-vm-tools utilities compiled and packaged for use with Tiny Core Linux; it used to be downloaded from the Puppet Labs download site in the Razor Microkernel days.
• mk-custom-busybox.tar.gz: this file contains a custom version of the busybox kernel whose DHCP client as been customized to support three additional DHCP options; specifically:
  • hanlon_server (option 224), which provides a mechanism for defining an IP address for the Hanlon server that is different from the next-server IP address (which must point at the server that provides the bootstrap kernel for the PXE boot process, typically a TFTP server)
  • hanlon_port (option 225), which provides a mechanism for passing the port that Hanlon is listening on through the DHCP response sent back to the Microkernel (important if the Hanlon server is listening on a port other than the default port, 8026)
  • hanlon_base_uri (option 226), which provides a mechanism for passing the base URI (or context) under which the Hanlon RESTful API is accessible (important if Hanlon is deployed to a server as a WAR file and this context changes from the default /hanlon/api context defined in Hanlon and the Hanlon-Microkernel)
• mk-ipmi-mods-and-tools.tar.gz: this file contains the drivers and utilities used by the Microkernel to access the BMC of an underlying node (if one exists on the node) and report back the associated IPMI-related facts to Hanlon

Pre-production release of the Hanlon-Microkernel project

This is the first tagged release of the Hanlon Microkernel; will require some additional work to get the build working off of the binaries that are downloaded from here instead of being downloaded from my personal Dropbox account. Should be able to make those changes and tag a v1.0 release shortly.