Amazon Machine Image (AMI) Packaging Manager

[nderjung]

Feature request summary

This issue tracks the addition of a new Package Manager into KraftKit which should enable users to manage and distribute Unikraft unikernels as Amazon Machine Images (AMIs).

Overview

AMIs are ID-referencable objects in AWS' infrastructure and are artifacts used as the OS during the "launching" of new EC2 VM instances. The goal of this project is to simplify the creation of such objects directly at the CLI with kraft, at package-time, and ultimately enabling users to easily distribute their application unikernels into AWS' infrastructure.

The process of creating a new AMI is unfortunately relatively convoluted and requires an existing EC2 instance to perform a snapshot of an EBS volume. This means that the entire process of creating an AMI is not only non-local but also requires an RPC-operated program in conjunction with adjustments to KraftKit itself, acting as command-and-control.

At a high-level, when the user specifies that they wish to create an AMI of their project (which consists of a successfully constructed unikernel binary), KraftKit must launch and instantiate a remote program on AWS infrastructure, perform an upload, format and write to the partition (along with additional bootloader), perform the snapshot and then tag the AMI, whilst also having to gracefully teardown everything on completion or on error. There is a lot which can go wrong in between the start and end of this process and as a result a lot of care and additional thought must go into safeguarding, handling unexpected (network) interruptions (context cancellation) and ensuring proper clean up.

This project is split into two components: the local adjustments representing the command-and-control aspects of creating an AMI and an independent AMI creating program which is launched on an EC2 program and receives instructions from the KraftKit user.

Command-and-Control

In order to introduce the ability manage AMIs in KraftKit, a new package manager implementation must be introduced. The package manager interface in KraftKit consists of a number of methods designed around the distribution of built unikernel images (or components of a unikernel, such as a library):

kraftkit/packmanager/manager.go

Lines 21 to 61 in a7c267d

	type PackageManager interface {
	// Update retrieves and stores locally a cache of the upstream registry.
	Update(context.Context) error
	// Pack turns the provided component into the distributable package. Since
	// components can comprise of other components, it is possible to return more
	// than one package. It is possible to disable this and "flatten" a component
	// into a single package by setting a relevant `pack.PackOption`.
	Pack(context.Context, component.Component, ...PackOption) ([]pack.Package, error)
	// Unpack turns a given package into a usable component. Since a package can
	// compromise of a multiple components, it is possible to return multiple
	// components.
	Unpack(context.Context, pack.Package, ...UnpackOption) ([]component.Component, error)
	// Catalog returns all packages known to the manager via given query
	Catalog(context.Context, ...QueryOption) ([]pack.Package, error)
	// Set the list of sources for the package manager
	SetSources(context.Context, ...string) error
	// Add a source to the package manager
	AddSource(context.Context, string) error
	// Delete package(s) from the package manager
	Delete(context.Context, ...QueryOption) error
	// Remove a source from the package manager
	RemoveSource(context.Context, string) error
	// IsCompatible checks whether the provided source is compatible with the
	// package manager
	IsCompatible(context.Context, string, ...QueryOption) (PackageManager, bool, error)
	// From is used to retrieve a sub-package manager. For now, this is a small
	// hack used for the umbrella.
	From(pack.PackageFormat) (PackageManager, error)
	// Format returns the name of the implementation.
	Format() pack.PackageFormat
	}

Upon implementation, aptly placed within kraftkit.sh/ami, and following relevant bootstrap registration, the process of creating an AMI should be as simple as calling the following in the context of a project repository:

kraft pkg --as ami --name my-ami-name .

At a glance, this usage will ultimately start a new remote EC2 instance
and install and run an external, remotely accessible AMI snapshotter program. Together, the two will upload and perform the creation of the AMI.

Package Manager Interface Method Notes

Update(context.Context) error

The Update method is typically used to locally cache information about remote entities which concern the implementing package manager. For example, with the OCI package manager, it makes a reference to the OCI Index and Manifest without saving layers, making it possible to skip lookups and simply download the layer later.

For an initial implementation of the AMI package manager, we need not concern ourselves with an implementation for Update an AMI. However, the goal with the Update method would be to ultimately save a reference to the list of AMI images (and their metadata) which belong to the user locally such that it can be referenced quickly and offline.

Pack(context.Context, component.Component, ...PackOption) ([]pack.Package, error)

The Pack method is where most of the "command-and-control" operation occurs. It's here that remote calls to AWS' API will be performed to:

1. Start a new EC2 instance;
2. Install the AMI image snapshotter program (see below);
3. Start the AMI image snapshotter program and start receiving the unikernel binary and receive upload progress;
4. Receive logs and status updates as he AMI image snapshotter performs its main task; and,
5. Cleans up.

Unpack(context.Context, pack.Package, ...UnpackOption) ([]component.Component, error) 

This method should ultimately download the AMI. To perform this action, the AMI needs to be exported as a VM image to an S3 bucket and then transferred locally and unpacked.

Catalog(context.Context, ...QueryOption) ([]pack.Package, error) 

This method should ultimately return a list of AMI objects for the given user account.

AddSource(context.Context, string) error 

This can be a no-op (return nil) and ignored.

Delete(context.Context, ...QueryOption) error

This method should delete the provided AMI image.

RemoveSource(context.Context, string) error

This can be a no-op (return nil) and ignored.

IsCompatible(context.Context, string, ...QueryOption) (PackageManager, bool, error)

Must determine if the provided input is an AMI. This is a simple check with the provided string is either an AMI ID or name and part of the user account.

From(pack.PackageFormat) (PackageManager, error)

This can be a no-op (return nil) and ignored.

Format() pack.PackageFormat

This should return the const:

const AMIFormat pack.PackageFormat = "ami"

Remote AMI Snapshotter

Since the process of creating an AMI is performed entirely remotely, a new, separate program is necessary for performing this procedure. The program, aptly called unikraft-ami-snapshotter, should be placed inside of the tools and must be a standalone Go program.

Generally, the execution of the program should:

1. Perform some pre-flight initialization steps which determine whether it is in the correct context (has access to an EBS volume);
2. Listen on a remote port in server mode to receive instructions, preferable over gRPC and having interface methods defined in Protobuf using buf such that KraftKit can reference these and make Go struct calls;

Roughly, the methods which the program should be able to handle are:

message UploadRequest {
  string file_name = 1;
  bytes chunk      = 2;
}

message UploadResponse {
  string file_name = 1;
  uint32 size      = 2;
}

message CreateRequest { /* TODO */ }
message CreateResponse { /* TODO */ }
message CleanupRequest { /* TODO */ }
message CleanupResponse { /* TODO */ }

service AMISnapshotter {
  rpc Upload(stream UploadRequest) returns (UploadResponse);
  rpc Create(CreateRequest) returns (CreateResponse);
  rpc Cleanup(CleanupRequest) returns (CleanupResponse);
}

The instantiation of this program within the context of an EC2 instance is also required and can be performed by encoding a bash script which downloads and runs the program as part of the RunInstance method's
UserData attribute. This program will be executed upon instance creation.

The unikraft-ami-snappshotter must receive its own GitHub Actions pipeline and be included as part of the release. This should make retrieving it as straightforward as:

wget -O unikraft-ami-snapshotter.tar.gz https://github.com/unikraft/kraftkit/releases/download/v0.8.1/unikraft-ami-snapshotter_0.8.1_linux_amd64.tar.gz
tar xzvf unikraft-ami-snapshotter.tar.gz
./unikraft-ami-snapshotter

See also:

• https://github.com/aws/aws-sdk-go-v2/blob/main/service/ec2/api_op_CreateImage.go

Additional considerations/notes

• It will be necessary to pass a "on progress" call back method as a new package option called:

  func OnProgress(cb func(float64)) PackOption

  The cb (callback) method can be used as part of the upload process. The value (float64) should be propagated to a tui.ParaProgress in relevant packaging subcommand files.

• The user's credentials for their AWS account is required as part of the AMI package manager and should be retrieved automatically by reading the default location (~/.aws/credentials). However, command-line flags and environmental variables should also be usable and supplied at the global context. This can be done by setting new attributes in config.KraftKit.