week05-containers-and-debugging

CICF Week 5

The goals for week 5 lab are:

1. Make and run a simple Docker image.
2. Run a service packaged in an external Docker image.
3. Use the Python debugger (pdb).

Tutorial

First, install docker

$ sudo apt update
$ ./install-docker.sh

You will need to then enter your password for sudo. and press Y to agree to installing the packages. Then we need to add us to the docker group. (You can type $USER, the shell will expand the variable to be the current username).

sudo usermod -aG docker $USER

Now choose "Log Out..." from the menu in the upper right corner and then sign back in to get the group membership to be cached.

Log back in and docker should be set up. We can verify this by running

docker ps

This lists all the containers that are currently running. There should be no containers are running, but the important thing is that there are no errors.

Build an image

Our first task is to make a container image. The Dockerfile-hello contains the instructions on how to build the image.

cat Dockerfile-hello

We start with the official base python image, we copy in our script, and then we specificy the command to run when the container is started. (Don't forget the period at the end!)

docker build -t hello -f Dockerfile-hello .

This is making an image from that instruction file and we are naming the image hello.

docker images

And then to run the image we use:

docker run hello

We can see the layers, (i.e. sequence of commands to make the image) with the "history" command:

docker history hello

The most recent change is at the top and the oldest change is at the bottom.

We can see the all the containers (whether running or not) with the -a option on the ps command:

docker ps -a

Okay, lets see if the execution envrionment is really restricted:

docker run -it hello bash

This has less of everything:

env
ping
python3

Also by default containers have no network access.

Object Store

We are going to run a service using docker.

This is something called a object store, which we will visit when we talk about cloud computing. For now the important thing is that we will run it inside a container.

First, get the container image:

docker pull minio/minio

See that it has been downloaded:

docker images

We will now run it. This looks complicated since we need to pass some configuration on the command line. By default containers don't have network access, so we first need to tell docker to allow us to access this container on the ports 9000 and 9001.

We then need to configure where we want the service to store data. And we need to configure a root user and password.

mkdir -p data

docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -e "MINIO_ROOT_USER=cicf" -e "MINIO_ROOT_PASSWORD=cicf1234" minio/minio server /data --console-address ":9001"

The container should have started and be running in the background. Docker printed the id for this container. We will let {id} refer to the first few letters/digits of this id.

docker ps
docker logs {id}

Okay, lets set it up. In the webbrowser visit http://localhost:9001 Sign in with our user and password cicf and cicf1234 Create a new bucket named "cicf-data".

Create a new access key named "test-account". Download the secret info and move it into our current directory:

mv ~/Downloads/credentials.json .

This is in JSON format.

cat credentials.json

Aside: there is a great tool for working with JSON files.

sudo apt install jq
jq . credentials.json

Give the new key enough access privileges. Copy the minio-policy policy file:

cat minio-policy

Edit the key we just made by pasting in the file contents. And then save it.

Now upload some files. We want hello.py.

We will install the Python minio library. We created a virtual environment in the home directory for installing Python packages earlier in week 2. We will reuse that environment here:

source ~/venv/bin/activate

Now we can install minio Python package inside that virtual environment:

pip3 install minio

Copy the credentials into the store.py file. Run store.py. Refresh the bucket, nothing should have happened.

Now upload the fib-first file to the bucket and rerun store.py. Refresh the bucket, there should be a new object named result-first It should contain the 16th Fibonacci number, which is 987.

The name is not quite right. Lets put a breakpoint into the file Before line 25 insert the line breakpoint().

When we run the script, it will stop in the debugger. We can see variables with p prefix. There are three ways to advance execution: stepping, next, or continue. Step will take the smallest possible increase, in that it will go into functions and stop. Next will stop at the next line of the program. Continue will keep running until another breakpoint or until the program ends.

Resources

• Python pdb debugger
• Reprozip is a cool tool that will watch a program run (using the strace mechanism) and then save all the files opened and their contents into a zip file. It then has the ability to create a container from the zip file.
• Apptainer (docs) is a container system, like Docker, that is more common in High Performance Computing envrionments.
• List of Docker images for the IceCube Neutrino Observatory and their instructions for using the software.
• Minio docs (The object store we used in the tutorial).
• jq is a great command line tool for working with JSON files. And there is a playground for messing around.
• misadventures in process containment
• Fibonacci Sequence on the OEIS.
• Minio Python client reference documentation
• A tutorial on using GNSS reflection data to measure snow and water height that uses a docker image.