[RC2] [Functest K8s] netperf TC failed

[sultetveny]

The opnfv/functest-kubernetes-benchmarking:v1.23 run_tests -t netperf fails on air-gapped environment

Tried to run the below k8s functional test, which failed because it can't download the image, as the environment is air gapped.

podman run -it --env-file ~/opnfv/env \
-v ~/opnfv/ca.pem:/home/opnfv/functest/ca.pem:Z \
-v ~/opnfv/config:/root/.kube/config:Z \
-v ~/opnfv/results:/home/opnfv/functest/results:Z \
-v ~/opnfv/repositories.yml:/home/opnfv/functest/repositories.yml:Z \
-v ~/opnfv/cluster-admin.pem:/home/opnfv/functest/cluster-admin.pem:Z \
-v ~/opnfv/cluster-admin-key.pem:/home/opnfv/functest/cluster-admin-key.pem:Z \
opnfv/functest-kubernetes-benchmarking:v1.23 /bin/bash

run_tests -t netperf
While other functional test cases are applicable to run on air-gapped environment this one is not, because no tag defined for the image and the imagepullpolicy set to always.

kubectl get rc -n netperf netperf-orch -o yaml

        image: sirot/netperf-latest
        imagePullPolicy: Always

One of the below option is true:
a) imagePullPolicy set to always
b) if you omit the imagePullPolicy field, and you don't specify the tag for the container image, imagePullPolicy is automatically set to Always
So either way the pull policy set to always.

However my air-gapped environment is configured to imagePullPolicy=IfNotPresent and the images downloaded locally, it's not used because the imagePullPolicy is overriden to always.

It shouldn't be a problem, in case I can set the mirror proxy instead. While I configured the mirror for docker it seems there is no effect, it still try to reach the docker.io and not the configured mirror repo.

Source: https://github.com/opnfv/functest-kubernetes:

Use on air gap environments (no access to Internet)
To test a Kubernetes without access to Internet, repository mirrors needs to be provided.

Currently, all tests supports this feature except cnf conformance.

There's two ways for providing the repository mirrors:

Give an environment variable (MIRROR_REPO) which gives a repository with all needed images.
Gives an environment variable per needed repo:
DOCKERHUB_REPO for DockerHub repository (docker.io)
GCR_REPO for Google Cloud repository (gcr.io)
K8S_GCR_REPO for Kubernetes repository (k8s.gcr.io)
QUAY_REPO for Quay repository (quay.io)

However I set the docker repository in the env file which appeared in the container's variable still not used.

# the environment file
cat env 
DOCKERHUB_REPO=registry1-docker-io.repo.cci.xxx.net

# within the running functest-kubernetes-benchmarking container
bash-5.1# printenv
DOCKERHUB_REPO=registry1-docker-io.repo.cci.xxx.net

# checking the pod
kubectl describe pod -n netperf netperf-orch-2wzvj | tail -n 20

Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Normal   Scheduled       36s   default-scheduler  Successfully assigned netperf/netperf-orch-2wzvj to cbis-sut1-worker-01
  Normal   AddedInterface  36s   multus             Add eth0 [192.168.231.161/32] from calico-network
  Normal   Pulling         36s   kubelet            Pulling image "sirot/netperf-latest"
  Warning  Failed          6s    kubelet            Failed to pull image "sirot/netperf-latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/sirot/netperf-latest:latest": failed to resolve reference "docker.io/sirot/netperf-latest:latest": failed to do request: Head "https://registry-1.docker.io/v2/sirot/netperf-latest/manifests/latest": dial tcp 34.205.13.154:443: i/o timeout
  Warning  Failed          6s    kubelet            Error: ErrImagePull
  Normal   BackOff         6s    kubelet            Back-off pulling image "sirot/netperf-latest"
  Warning  Failed          6s    kubelet            Error: ImagePullBackOff

I see 3 options:

this test is not applicable on air-gapped environment
using a version tag with unset imagePullPolicy or configure imagePullPolicy=IfNotPresent
enable to use the configured mirror repo for docker
What you expected to happen:

I expected the test case executed successfully.

How to reproduce it (as minimally and precisely as possible):

See above.

Anything else we need to know?:

Ticket originally opened under different github project, but recommended to open it under opnfv/anuket instead.
kubernetes/perf-tests#2241

Environment:

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:49:13Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:42:57Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration:
Openstack

OS (e.g: cat /etc/os-release):
NAME="Rocky Linux"
VERSION="8.6 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.6 (Green Obsidian)"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky Linux"
ROCKY_SUPPORT_PRODUCT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

Kernel (e.g. uname -a):
Linux cbis-sut1-control-01 4.18.0-372.9.1.el8.x86_64 https://github.com/kubernetes/perf-tests/pull/1 SMP Tue May 10 14:48:47 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux