Should we add a 'base' build container so we have a technology agnostic foundation?

[grayside]

I've been thinking about how we present Outrigger as a technology-agnostic solution, but our build container isn't all that agnostic. It's a big pile of stuff which is aimed at the tools and tech stacks Phase2 tends to work with around PHP & Drupal.

What if we centralized the truly tech-agnostic stuff we want in a build container for any application into a "base", then extend that into others? I've put together a bit of a demonstration in the expando-matic sections below.

Base Dockerfile

FROM centos:7

# Install base packages.
RUN yum -y install epel-release yum-plugin-ovl deltarpm && \
  # Add the IUS repository. This is needed for git2.
  curl -L "https://centos7.iuscommunity.org/ius-release.rpm" > /usr/local/ius-release.rpm && \
  rpm -Uvh /usr/local/ius-release.rpm
  yum -y update && \
  yum -y install \
    bzip2 \
    curl \
    dnsutils \
    git2u-all \
    jq \
    less \
    make \
    openssl \
    patch \
    pv \
    rsync \
    sudo \
    ssh \
    sendmail \
    unzip \
    vim-minimal \
    # Necessary for drush and developer orientation.
    which \
  yum clean all

# Download & Install confd.
ENV CONFD_VERSION 0.11.0
RUN curl -L "https://github.com/kelseyhightower/confd/releases/download/v$CONFD_VERSION/confd-$CONFD_VERSION-linux-amd64" > /usr/bin/confd && \
    chmod +x /usr/bin/confd
ENV CONFD_OPTS '--backend=env --onetime'

# Ensure $HOME is set
ENV HOME /root

# Configure Git
# https://git-scm.com/docs/git-config#git-config-corepreloadIndex
RUN git config --global core.preloadindex true

# Run the s6-based init.
ENTRYPOINT ["/init"]

# Set up a standard volume for logs.
VOLUME ["/var/log/services"]

COPY root /

CMD [ "/versions.sh" ]

Extending php7.0 Dockerfile

FROM outrigger/build-base

RUN yum -y install centos-release-scl && \
  yum-config-manager --enable rhel-server-rhscl-7-rpms && \
  yum -y update && \
  yum -y install \
    https://rpms.remirepo.net/enterprise/remi-release-7.rpm \
    gcc-c++ \
    httpd-tools \
    mariadb \
    nmap-ncat \
    php70 \
    php70-php-devel \
    php70-php-gd \
    php70-php-xml \
    php70-php-pdo \
    php70-php-mysql \
    php70-php-mysqlnd \
    php70-php-mbstring \
    php70-php-fpm \
    php70-php-opcache \
    php70-php-pecl-memcache \
    php70-php-pecl-xdebug \
    php70-php-posix \
    php70-php-mcrypt \
    postgresql \
    ruby193 \
    ruby193-rubygems \
    ruby193-ruby-devel \
    # Necessary library for phantomjs per https://github.com/ariya/phantomjs/issues/10904
    fontconfig && \
    yum clean all

# Also the composer, ruby, node, and drush things.

Pros	Cons
Easy starting place for new build images	Another image to manage
Have leaner build image by application stack	Steer towards an explosive number of build images (Node app vs. PHP app have different build containers, instead of one build container with all for both)
Clarify our core definition of a build container	Confuse people whether a change belongs in base, all images for a given language, or a specific build image language version

[tekante]

I like the idea of having smaller build containers that are more specific and clear in concept but I wonder if practically it is worth it.

For example, if I hold up three different technologies as test cases, one based on Node, on based on Ruby (on Rails most likely) and one based on PHP. I think the Node and Ruby ones stand the best chance of not needing other technology but I know our PHP ones very frequently need Node at least and maybe Ruby (though libsass may have removed that). So in my mind we'd have the benefit of a set of smaller containers for those who weren't working on a PHP based project and those working on a PHP based one wind up with a build image with everything else they'd need to do a more technology specific project already in it.

It might still be valuable to split out a base container with all of the debugging utilities and stuff just to keep from needing to rebuild those layers every single time.

[grayside]

I'm not suggesting as part of this that we force build containers to be focused on a single technology stack. Rather I'm suggesting the base case of a build container is simply agnostic. We can then have ruby, node, and php/node images based on it.

If we do have a baseline image already built, it will reduce the uncompressed, download size of build images after the first one pulled by hundreds of megabytes. CentOS, gcc and so on tend to add a lot of stuff under the hood.

[febbraro]

I think that the right "level" for a base should be not just for "build" images, but for other things like one-shot / custom images for things like db restores or other maintenance related images. I consider "build" images to be focused around development, where as I could see cases where we need various production-mode utility images too and it would be nice to have a common toolset to build off of. I think your vision of a base image could be useful for both. Thoughts?

[grayside]

That seems about right. We call it the "build" container because we're making the point that it's where builds should run as a sort of mantra marketing for devs. In practice, this is our approach to command-line tools which don't make sense as traditional, hyper-focused containers.

The case for "making sense" is part one of user mindset and part breaking down some of the complexity of tools interacting with each other.

So given that, does the base image proposed above do too much? Too little?

[febbraro]

I think it looks about right, I might add the yaml equivalent of jq, and ensure that wget is there, but otherwise it looks like a good cut at it.

Now the hard part. What the hell do we NAME it?

[febbraro]

Might want to add the aws cli tool as well.

[grayside]

yq (the jq of YAML) is a wrapper around jq that parses the yaml and in-line converts it to json before processing with jq.

Both yq and aws-cli are python-based, which would mean we'd need to add the python stack to the image. I'm not sure we want to do that, as it will add several dozen megabytes. (maybe triple digits?)

yq is simple enough, would make a good golang learning project. :)

aws-cli isn't needed by most projects, maybe that should be kept standalone or that we should have an aws variant?

As far as names go, busybox is taken. I was thinking just called it build-base, but if we want to get more general purpose, we could outrigger/toolbox, outrigger/utility, outrigger/tools to keep it simple.

[grayside]

More names: Helm, Terminal, Command.

I assume by the request for a name you want this to have its own repo?

[tekante]

As a base for development extension it's pretty close for me. As a base for one shot utility instances I think it may be a little heavy. Depends on the stance the base image would take on being lean versus developer/user experience.

If we're going lean (has that ship already sailed with our choice of image to derive from . . .) for one shot instances I'd consider dropping: make, pv, git. I could even see dropping curl and the various zip utilities though I'd probably add them all back as the first layer on top of the base to get to the base for derived build images. If we don't really have a desire to shoot for lean-ness I'd leave them all in (and consider adding telnet and bind-utils which I find myself using for debugging all the time).

If we're going for more generalized names for the base and want to keep to the nautical theme perhaps keel.

[grayside]

If we're going lean (has that ship already sailed with our choice of image to derive from . . .)

Our philosophy of build containers is not really aimed at lean, single-purpose services, and I'm not arguing to change that here. The idea of this is more along the lines of a specialized remote server environment, with binary compatibility to production hosts (such that using Alpine could be done, but musl is just different enough that we're avoiding the complication).

My thought is to try to keep the image as tidy as possible while providing all the utilities commonly assumed in a build/console environment, especially one that may be activated in a production cloud to perform operational tasks.

From that standpoint, I'd say, yes, we should consider adding telnet, bind-utils, and perhaps strace.

As a base for one shot utility instances I think it may be a little heavy.

One-shot utility instances should probably stick with looking for the right base image in the Docker ecosystem. I usually pick the alpine equivalent of whatever official package is available when Dockerizing a tool. We can always build the lean tools, the problem is those times when the next step is needing to support executing other tools with Docker, which is something we can do in specialized situations but don't want to require people to manage/think on routinely.

If we're going for more generalized names for the base and want to keep to the nautical theme perhaps keel.

I like it. outrigger/keel seems good to me.

[grayside]

outrigger-keel is now published, with outrigger/keel:1.0 pinned to a tag and and outrigger/keel:dev as latest master.

I've been trying to refactor outrigger/build:php70 to lean on it, and getting into some yum issues.