Skip to content

Work with Arktos vm runtime along with Arktos

Jump to bottom
Yunwen Bai edited this page Jun 18, 2020 · 5 revisions

This wiki describe the steps to use the local, standalone runtime deployment for a dev environment, which is simpler than restarting the runtime daemonset with during the runtime code dev and testing time.

1. Start Arktos onebox with the Arktos-up.sh as usual and delete the virtlet daemonset

This can be avoid if options added into the Arktos-up.sh to not install the daemonset.

root@ip-172-31-11-43:/work/src/github.com/Mirantis/virtlet# kubectl get ds -n kube-system
NAME      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
virtlet   1         1         1       1            1           <none>          5m4s
root@ip-172-31-11-43:/work/src/github.com/Mirantis/virtlet# kubectl delete ds virtlet -n kube-system
daemonset.extensions "virtlet" deleted

2. Build the vm runtime

The VM runtime build its docker images inside a docker container called virtlet-build. it dynamically load the dependencies into the /vendor/github.com/ folder in the container and build. A success build will generate the vmruntime:latest docker image, as shown below. You can always tag it and publish with the build/cmd.sh tool as well。

root@ip-172-31-11-43:/work/src/github.com/Mirantis/virtlet# ./build/cmd.sh build
...

After a success build, check the latest runtime image:

root@ip-172-31-11-43:/work/src/github.com/Mirantis/virtlet# docker images
REPOSITORY                               TAG                                   IMAGE ID            CREATED             SIZE
arktosstaging/vmruntime                  latest                                84b31e8f0b73        2 minutes ago       1.05GB

3. (Optionally) set the env if apparmor files are needed from your repo

export VIRTLET_DEPLOYMENT_FILES_DIR=/tmp2
export VIRTLET_DEPLOYMENT_FILES_SRC=https://raw.githubusercontent.com/yb01/arktos-vm-runtime/cgroupwork/deploy
export APPARMOR_ENABLED=true

4. Start the runtime docker containers

The local, standlone runtime deployment starts three docker containers with the tool below:

./deploy/SetupVmruntime.sh start

root@ip-172-31-11-43:/work/src/github.com/Mirantis/virtlet# docker ps
CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS               NAMES
a31f2f208af7        arktosstaging/vmruntime:latest   "/bin/bash -c '/libv…"   3 seconds ago       Up 3 seconds                            vmruntime_libvirt
578d14627def        arktosstaging/vmruntime:latest   "/bin/bash -c '/star…"   3 seconds ago       Up 3 seconds                            vmruntime_virtlet
de0f74087747        arktosstaging/vmruntime:latest   "/bin/bash -c '/vms.…"   3 seconds ago       Up 3 seconds                            vmruntime_vms
b07de4ec5782        virtlet_build:latest             "/bin/bash -c 'mount…"   49 minutes ago      Up 49 minutes                           virtlet-build

5. Runtime log files

All runtime log files ( virtlet.log, libvirt.log and vms.log ) are redirected to /var/log/virtlet folder. for example, this is the content for /var/log/virtlet/virtlet.log:

2020-06-18T07:08:33Z START VIRTLET LOG FILE
I0618 07:09:00.605779     530 interface.go:360] Looking for default routes with IPv4 addresses
I0618 07:09:00.605853     530 interface.go:365] Default route transits interface "eth0"
I0618 07:09:00.606129     530 interface.go:174] Interface eth0 is up
I0618 07:09:00.606197     530 interface.go:222] Interface "eth0" has 2 addresses :[172.31.11.43/20 fe80::8aa:83ff:fe8d:c386/64].
I0618 07:09:00.606231     530 interface.go:189] Checking addr  172.31.11.43/20.
I0618 07:09:00.606242     530 interface.go:196] IP found 172.31.11.43
I0618 07:09:00.606258     530 interface.go:228] Found valid IPv4 address 172.31.11.43 for interface "eth0".
I0618 07:09:00.606266     530 interface.go:371] Found active IP 172.31.11.43
I0618 07:09:00.606625     530 listener.go:66] UnixSocket Listener started
I0618 07:09:00.608900     530 manager.go:154] Starting server on socket /run/virtlet.sock
I0618 07:10:46.013336     530 grpc.go:110] ENTER: /runtime.v1alpha2.RuntimeService/Status():
{}
I0618 07:10:46.013579     530 grpc.go:119] LEAVE: /runtime.v1alpha2.RuntimeService/Status()
status:
  conditions:
  - status: true
    type: RuntimeReady
  - status: true
    type: NetworkReady
I0618 07:10:53.926902     530 grpc.go:110] ENTER: /runtime.v1alpha2.RuntimeService/Status():
{}
I0618 07:10:53.927061     530 grpc.go:119] LEAVE: /runtime.v1alpha2.RuntimeService/Status()

6. Create a test VM

root@ip-172-31-11-43:/var/log/virtlet# kubectl create -f /vm_default.yaml 
pod/vmdefault created
root@ip-172-31-11-43:/var/log/virtlet# kubectl get pod -w
NAME        HASHKEY              READY   STATUS    RESTARTS   AGE
vmdefault   259581071025608557   0/1     Pending   0          7s
vmdefault   259581071025608557   1/1     Running   0          7s

Trouble-shooting and debugging tools and tips

Log files and logging level

The log files described in step 5 are the main trouble-shooting means at this moment. you can specify a higher logging level with VIRTLET_LOGLEVEL environment variable. The default is 3 which is good for most of cases.

dmesg

dmesg is often used for identify apparmor related issues especially you are adding new features. there should not be any "Denied" message related with virtlet, libvirt or VMS processes

Known issues

Most of the issue we hit so far is with Apparmor, especially with the obsolete virtlet profiles. So running the vm runtime with APPARMOR_ENABLED flag is prefered so you won't need to deal with reset apparmor daemon.

Clone this wiki locally