Skip to content

Files

Latest commit

anti-socialanti-social
Use vagga on github CI (#573)
Jan 14, 2022
5e020a0 · Jan 14, 2022

History

History
766 lines (543 loc) · 20.8 KB

build_commands.rst

File metadata and controls

766 lines (543 loc) · 20.8 KB

Container Building Guide

Build commands are tagged values in your container definition. For example:

containers:
  ubuntu:
    setup:
    - !Ubuntu xenial
    - !Install [python]

This contains two build commands !Ubuntu and !Install. They mostly run sequentially, but some of them are interesting, for example !BuildDeps installs package right now, but also removes package at the end of the build to keep container smaller and cleaner.

See :ref:`build_steps` for additional details on specific commands. There is also an :ref:`genindex`

Generic Installers

To run arbitrary shell command use !Sh:

setup:
- !Ubuntu xenial
- !Sh "apt-get update && apt-get install -y python"

If you have more than one-liner you may use YAMLy literal syntax for it:

setup:
- !Ubuntu xenial
- !Sh |
   wget somepackage.tar.gz
   tar -xzf somepackage.tar.gz
   cd somepackage
   make && make install

Warning

The !Sh command is run by /bin/sh -exc. With the flags meaning -e -- exit if any command fails, -x -- print command before executing, -c -- execute command. You may undo -ex by inserting set +ex at the start of the script. But it's not recommended.

To run !Sh you need /bin/sh. If you don't have shell in container you may use !Cmd that runs command directly:

setup:
# ...
- !Cmd [/usr/bin/python, '-c', 'print "hello from build"']

To install a package of any (supported) linux distribution just use !Install command:

containers:

  ubuntu:
    setup:
    - !Ubuntu xenial
    - !Install [python]

  ubuntu-trusty:
    setup:
    - !Ubuntu trusty
    - !Install [python]

  alpine:
    setup:
    - !Alpine v3.5
    - !Install [python]

Occasionally you need some additional packages to use for container building, but not on final machine. Use !BuildDeps for them:

setup:
- !Ubuntu xenial
- !Install [python]
- !BuildDeps [python-dev, gcc]
- !Sh "make && make install"

The python-dev and gcc packages from above will be removed after building whole container.

To add some environment arguments to subsequent build commands use !Env:

setup:
# ...
- !Env
  VAR1: value1
  VAR2: value2
- !Sh "echo $VAR1 / $VAR2"

Note

The !Env command doesn't add environment variables for processes run after build. Use environ setting for that.

Sometimes you want to rebuild container when some file changes. For example if you have used the file in the build. There is a !Depends command which does nothing per se, but add a dependency. The path must be relative to your project directory (the dir where vagga.yaml is). For example:

setup:
# ...
- !Depends requirements.txt
- !Sh "pip install -r requirements.txt"

To download and unpack tar archive use !Tar command:

setup:
- !Tar
  url: http://something.example.com/some-project-1.0.tar.gz
  sha256: acd1234...
  path: /
  subdir: some-project-1.0

Only url field is mandatory. If url starts with dot . it's treated as filename inside project directory. The path is target path to unpack into, and subdir is a dir inside tar file. By default path is root of new filesystem. The subdir is a dir inside the tar file, if omitted whole tar archive will be unpacked.

You can use !Tar command to download and unpack the root filesystem from scratch.

There is a shortcut to download tar file and build and install from there, which is !TarInstall:

setup:
- !TarInstall
  url: https://static.rust-lang.org/dist/rust-1.10.0-x86_64-unknown-linux-gnu.tar.gz
  sha256: abcd1234...
  subdir: rust-1.10.0-x86_64-unknown-linux-gnu
  script: ./install.sh --prefix=/usr

Only the url is mandatory here too. Similarly, if url starts with dot . it's treated as filename inside project directory. The script is by default ./configure --prefix=/usr; make; make install. It's run in subdir of unpacked archive. If subdir is omitted it's run in the only subdirectory of the archive. If archive contains more than one directory and subdir is empty, it's an error, however you may use . as subdir.

To remove some data from the image after building use !Remove command:

setup:
# ...
- !Remove /var/cache/something

To clean directory but ensure that directory exists use !EmptyDir command:

setup:
# ...
- !EmptyDir /tmp

Note

The /tmp directory is declared as !EmptyDir implicitly for all containers.

To ensure that directory exists use !EnsureDir command. It's very often used for future mount points:

setup:
# ...
- !EnsureDir /sys
- !EnsureDir /dev
- !EnsureDir /proc

Note

The /sys, /dev and /proc directories are created automatically for all containers.

Sometimes you want to keep some cache between builds of container or similar containers. Use !CacheDirs for that:

setup:
# ...
- !CacheDirs { "/var/cache/apt": "apt-cache" }

Multiple directories may be specified at once.

Warning

In this example, "apt-cache" is the name of the directory on your host. Unless changed in the :ref:`settings`, the directory can be found in .vagga/.cache/apt-cache. It is shared both between all the containers and all the different builders (not only same versions of the single container). In case the user enabled shared-cache, the folder will also be shared between containers of different projects.

Sometimes you just want to write a file in target system:

setup:
# ...
- !Text
  /etc/locale.conf: |
     LANG=en_US.UTF-8
     LC_TIME=uk_UA.UTF-8

Note

You can use any YAML'y syntax for file body just the "literal" one which starts with a pipe | character is the most handy one

Ubuntu

To install base ubuntu system use:

setup:
- !Ubuntu xenial

Potentially any ubuntu long term support release instead of xenial should work. To install a non LTS release, use:

setup:
- !UbuntuRelease { codename: wily }

To install any ubuntu package use generic !Install command:

setup:
- !Ubuntu xenial
- !Install python

Many interesting ubuntu packages are in the "universe" repository, you may add it by series of !UbuntuRepo commands (see below), but there is shortcut !UbuntuUniverse:

setup:
- !Ubuntu xenial
- !UbuntuUniverse
- !Install [checkinstall]

The !UbuntuRepo command adds additional repository. For example, to add marathon repository you may write:

setup:
- !Ubuntu xenial
- !UbuntuRepo
  url: http://repos.mesosphere.io/ubuntu
  suite: xenial
  components: [main]
- !Install [mesos, marathon]

This effectively adds the repository and installs mesos and marathon packages.

Note

Probably the key for repository should be added to be able to install packages.

Alpine

To install base alpine system use:

setup:
- !Alpine v3.5

Potentially any alpine version instead of v3.4 should work.

To install any alpine package use generic !Install command:

setup:
- !Alpine v3.5
- !Install [python]

Npm Installer

You can build somewhat default nodejs environment using !NpmInstall command. For example:

setup:
- !Ubuntu xenial
- !NpmInstall [babel]

All node packages are installed as --global which should be expected. If no distribution is specified before the !NpmInstall command, the implicit !Alpine v3.5 (in fact the latest version) will be executed.

setup:
- !NpmInstall [babel]

So above should just work as expected if you don't need any special needs. E.g. it's usually perfectly okay if you only use node to build static scripts.

The following npm features are supported:

  • Specify package@version to install specific version (recommended)
  • Use git+https:// url for the package. In this case git will be installed for the duration of the build automatically
  • Bare package_name (should be used only for one-off environments)

Other forms may work, but are unsupported for now.

Note

The npm and additional utilities (like build-essential and git) will be removed after end of container building. You must !Install them explicitly if you rely on them later.

Python Installer

There are two separate commands for installing packages for python2 and python3. Here is a brief example:

setup:
- !Ubuntu xenial
- !Py2Install [sphinx]

We always fetch latest pip for installing dependencies. The python-dev headers are installed for the time of the build too. Both python-dev and pip are removed when installation is finished.

The following pip package specification formats are supported:

  • The package_name==version to install specific version (recommended)
  • Bare package_name (should be used only for one-off environments)
  • The git+ and hg+ links (the git and mercurial are installed as build dependency automatically), since vagga 0.4 git+https and hg+https are supported too (required installing ca-certificates manually before)

All other forms may work but not supported. Specifying command-line arguments instead of package names is not supported. To configure pip use !PipConfig directive. In the example there are full list of parameters:

setup:
- !Ubuntu xenial
- !PipConfig
  index-urls: ["http://internal.pypi.local"]
  find-links: ["http://internal.additional-packages.local"]
  dependencies: true
- !Py2Install [sphinx]

They should be self-descriptive. Note unlike in pip command line we use single list both for primary and "extra" indexes. See pip documentation for more info about options

Note

By default dependencies is false. Which means pip is run with --no-deps option. Which is recommended way for setting up isolated environments anyway. Even setuptools are not installed by default. To see list of dependencies and their versions you may use pip freeze command.

Better way to specify python dependencies is to use "requirements.txt":

setup:
- !Ubuntu xenial
- !Py3Requirements "requirements.txt"

This works the same as Py3Install including auto-installing of version control packages and changes tracking. I.e. It will rebuild container when "requirements.txt" change. So ideally in python projects you may use two lines above and that's it.

The Py2Requirements command exists too.

Note

The "requirements.txt" is checked semantically. I.e. empty lines and comments are ignored. In current implementation the order of items is significant but we might remove this restriction in the future.

PHP/Composer Installer

Composer packages can be installed either explicitly or from composer.json. For example:

setup:
- !Ubuntu xenial
- !ComposerInstall [laravel/installer]

The packages will be installed using Composer's global require at /usr/local/lib/composer/vendor. This is only useful for installing packages that provide binaries used to bootstrap your project (like the Laravel installer, for instance):

setup:
- !Ubuntu xenial
- !ComposerInstall [laravel/installer]
- !Sh laravel new src

Alternatively, you can use Composer's crate-project command:

setup:
- !Ubuntu xenial
- !ComposerInstall # just to have composer available
- !Sh composer create-project --prefer-dist laravel/laravel src

Note

In the examples above, it is used src (/work/src) instead of . (/work) because Composer only accepts creating a new project in an empty directory.

For your project dependencies, you should install packages from your composer.json. For example:

setup:
- !Ubuntu xenial
- !ComposerDependencies

This command will install packages (including dev) from composer.json into /usr/local/lib/composer/vendor using Composer's install command.

Note

The /usr/local/lib/composer directory will be automatically added to PHP's include_path.

Warning

Most PHP frameworks expect to find the vendor directory at the same path as your project in order to require autoload.php, so you may need to fix your application entry point (in a Laravel 5 project, for example, you should edit bootstrap/autoload.php and change the line require __DIR__.'/../vendor/autoload.php'; to require 'vendor/autoload.php';.

You can also specify some options available from Composer command line, for example:

setup:
- !Ubuntu xenial
- !ComposerDependencies
  working_dir: src # run command inside src directory
  dev: false # do not install dev dependencies
  optimize_autoloader: true

If you want to use hhvm, you can disable the installation of the php runtime:

setup:
- !Ubuntu xenial
- !ComposerConfig
  install_runtime: false
  runtime_exe: /usr/bin/hhvm

Note

When setting the runtime_exe option, be sure to specify the full path of the binary (e.g /usr/bin/hhvm).

Note

Vagga will try to create a symlink from runtime_exe into /usr/bin/php. If that location already exists, Vagga will not overwrite it.

Note that you will have to manually install hhvm and set the include_path:

setup:
- !Ubuntu xenial
- !Repo universe
- !Install [hhvm]
- !ComposerConfig
  install_runtime: false
  runtime_exe: /usr/bin/hhvm
- !Sh echo 'include_path=.:/usr/local/lib/composer' >> /etc/hhvm/php.ini ❶
environ:
  HHVM_REPO_CENTRAL_PATH: /run/hhvm.hhbc ❷
  • ❶ -- setup include_path in hhvm config
  • ❷ -- tell hhvm to store the build cache database in a writeable directory

Alpine v3.5 added support for php7 in their "community" repository while keeping php5 as the default runtime. In order to use php7, you have to specify all the packages required by composer (and any other php packages you may need):

setup:
- !Alpine v3.5
- !Repo community
- !Install
  - php7
  - php7-openssl
  - php7-phar
  - php7-json
  - php7-pdo
  - php7-dom
  - php7-zip
- !ComposerConfig
  install_runtime: false
  runtime_exe: /usr/bin/php7

Note

Composer executable and additional utilities (like build-essential and git) will be removed after end of container building. You must !Download or !Install them explicitly if you rely on them later.

Warning

PHP/Composer support was recently added to vagga, some things may change as we gain experience with the tool.

Ruby Installer

Ruby gems can be installed either by providing a list of gems or from a Gemfile using bundler. For example:

setup:
- !Ubuntu xenial
- !GemInstall [rake]

We will update gem to the latest version (unless specified not to) for installing gems. The ruby-dev headers are installed for the time of the build too and are removed when installation is finished.

The following gem package specification formats are supported:

  • The package_name:version to install specific version (recommended)
  • Bare package_name (should be used only for one-off environments)
setup:
- !Ubuntu xenial
- !Install [zlib1g]
- !BuildDeps [zlib1g-dev]
- !Env
  HOME: /tmp
- !GemInstall [rails]
- !Sh rails new . --skip-bundle

Bundler is also available for installing gems from Gemfile. For example:

setup:
- !Ubuntu xenial
- !GemBundle

You can also specify some options to Bundler, for example:

setup:
- !Ubuntu xenial
- !GemBundle
  gemfile: src/Gemfile # use this Gemfile
  without: [development, test] # groups to exclude when installing gems
  trust_policy: HighSecurity

It is possible to avoid installing ruby if you are providing it yourself:

setup:
- !Ubuntu xenial
- !GemSettings
  install_ruby: false
  gem_exe: /usr/bin/gem

Warning

Ruby/Gem support was recently added to vagga, some things may change as we gain experience with the tool.

Dependent Containers

Sometimes you want to build on top of another container. For example, container for running tests might be based on production container, but it might add some test utils. Use !Container command for that:

containers:
  base:
    setup:
    - !Ubuntu xenial
    - !Py3Install [django]
  test:
    setup:
    - !Container base
    - !Py3Install [nose]

It's also sometimes useful to freeze some part of container and test next build steps on top of it. For example:

containers:
  temporary:
    setup:
    - !Ubuntu xenial
    - !TarInstall
      url: http://download.zeromq.org/zeromq-4.1.4.tar.gz
  web:
    setup:
    - !Container temporary
    - !Py3Install [pyzmq]

In this case when you try multiple different versions of pyzmq, the zeromq itself will not be rebuilt. When you're done, you can append build steps and remove the temporary container.

Sometimes you need to generate (part of) vagga.yaml itself. For some things you may just use shell scripting. For example:

container:
  setup:
  - !Ubuntu xenial
  - !Env { VERSION: 0.1.0 }
  - !Sh "apt-get install somepackage==$VERSION"

Note

Environment of user building container is always ignored during build process (but may be used when running command).

In more complex scenarios you may want to generate real vagga.yaml. You may use that with ancillary container and !SubConfig command. For example, here is how we use a docker2vagga script to transform Dockerfile to vagga config:

docker-parser: ❶
  setup:
  - !Alpine v3.5
  - !Install [python]
  - !Depends Dockerfile ❷
  - !Depends docker2vagga.py ❷
  - !Sh 'python ./docker2vagga.py > /docker.yaml' ❸

somecontainer:
  setup:
  - !SubConfig
    source: !Container docker-parser ❶
    path: docker.yaml ❹
    container: docker-smart ❺

Few comments:

  • ❶ -- container used for build, it's rebuilt automatically as a dependency for "somecontainer"
  • ❷ -- normal dependency rules apply, so you must add external files that are used to generate the container and vagga file in it
  • ❸ -- put generated vagga file inside a container
  • ❹ -- the "path" is relative to the source if the latter is set
  • ❺ -- name of the container used inside a "docker.yaml"

Warning

The functionality of !SubConfig is experimental and is a subject to change in future. In particular currently the /work mount point and current directory used to build container are those of initial vagga.yaml file. It may change in future.

The !SubConfig command may be used to include some commands from another file without building container. Just omit source command:

subdir:
  setup:
  - !SubConfig
    path: subdir/vagga.yaml
    container: containername

The YAML file used may be a partial container, i.e. it may contain just few commands, installing needed packages. The other things (including the name of the base distribution) can be set by original container:

# vagga.yaml
containers:
  ubuntu:
    setup:
    - !Ubuntu xenial
    - !SubConfig
      path: packages.yaml
      container: packages
  alpine:
    setup:
    - !Alpine v3.5
    - !SubConfig
      path: packages.yaml
      container: packages

# packages.yaml
containers:
  packages:
    setup:
    - !Install [redis, bash, make]