Test opnfv/functest-kubernetes-security:v1.23 run_tests -t kube_bench_master unable to run because of taint

[sultetveny]

Overview

I've faced with the following issue running the kubernetes security test using kube_bench_master.

How did you run kube-bench?

podman run -it --env-file ~/opnfv/env \
-v ~/opnfv/ca.pem:/home/opnfv/functest/ca.pem:Z \
-v ~/opnfv/config:/root/.kube/config:Z \
-v ~/opnfv/results:/home/opnfv/functest/results:Z \
-v ~/opnfv/repositories.yml:/home/opnfv/functest/repositories.yml:Z \
-v ~/opnfv/cluster-admin.pem:/home/opnfv/functest/cluster-admin.pem:Z \
-v ~/opnfv/cluster-admin-key.pem:/home/opnfv/functest/cluster-admin-key.pem:Z \
opnfv/functest-kubernetes-security:v1.23 /bin/bash

# then within the container
run_tests -t kube_bench_master

What happened?

Test case failed. For more information please check attached file.
functest-kubernetes.debug.log

2023-02-24 14:10:38,039 - xtesting.ci.run_tests - INFO - Deployment description:

+-------------------------+----------------------------------------------------------+
|         ENV VAR         |                          VALUE                           |
+-------------------------+----------------------------------------------------------+
|         CI_LOOP         |                          daily                           |
|          DEBUG          |                           true                           |
|     DEPLOY_SCENARIO     |                 k8s-nosdn-nofeature-noha                 |
|      INSTALLER_TYPE     |                         unknown                          |
|        BUILD_TAG        |                                                          |
|        NODE_NAME        |                                                          |
|       TEST_DB_URL       |     http://testresults.opnfv.org/test/api/v1/results     |
|     TEST_DB_EXT_URL     |                                                          |
|     S3_ENDPOINT_URL     |                                                          |
|        S3_DST_URL       |                                                          |
|       HTTP_DST_URL      |                                                          |
+-------------------------+----------------------------------------------------------+

2023-02-24 14:10:38,049 - xtesting.ci.run_tests - INFO - Loading test case 'kube_bench_master'...
2023-02-24 14:10:38,418 - xtesting.ci.run_tests - INFO - Running test case 'kube_bench_master'...

2023-02-24 14:30:38,520 - xtesting.ci.run_tests - ERROR - 

Please fix the testcase kube_bench_master.
All exceptions should be caught by the testcase instead!

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/xtesting/ci/run_tests.py", line 171, in run_test
    test_case.run(**kwargs)
  File "/usr/lib/python3.9/site-packages/functest_kubernetes/security/security.py", line 212, in run
    self.details["report"] = ast.literal_eval(self.pod_log)
  File "/usr/lib/python3.9/ast.py", line 62, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/lib/python3.9/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 0
    
SyntaxError: unexpected EOF while parsing
2023-02-24 14:30:38,522 - xtesting.ci.run_tests - ERROR - The test case 'kube_bench_master' failed.
2023-02-24 14:30:38,522 - xtesting.ci.run_tests - INFO - Execution exit value: Result.EX_ERROR

The content of the env file successfully read as you see below the env variables set within the container properly. However their values somehow not respected (e.g., NON_BLOCKING_TAINTS).

If 2 worker nodes not enough, then the test may can use the control and edge nodes too.

bash-5.1# printenv
HOSTNAME=947d26a68598
GCR_REPO=gcr-io.repo.cci.nokia.net
PWD=/
container=podman
HOME=/root
KUBE_TEST_REPO_LIST=/home/opnfv/functest/repositories.yml
DEPLOY_SCENARIO=k8s-nosdn-nofeature-noha
TERM=xterm
SHLVL=1
NON_BLOCKING_TAINTS=is_control,is_edge
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG=true
K8S_GCR_REPO=k8s-gcr-io.repo.cci.nokia.net
_=/bin/printenv
bash-5.1# 

kubectl get events -A --watch

NAMESPACE          LAST SEEN   TYPE      REASON             OBJECT                        MESSAGE
kube-bench-f4dqn   56s         Warning   FailedScheduling   pod/kube-bench-master-t5l89   0/6 nodes are available: 1 node(s) had taint {is_edge: true}, that the pod didn't tolerate, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had taint {is_control: true}, that the pod didn't tolerate.
kube-bench-f4dqn   57s         Normal    SuccessfulCreate   job/kube-bench-master         Created pod: kube-bench-master-t5l89
kube-bench-f4dqn   0s          Warning   FailedScheduling   pod/kube-bench-master-t5l89   0/6 nodes are available: 1 node(s) had taint {is_edge: true}, that the pod didn't tolerate, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had taint {is_control: true}, that the pod didn't tolerate.
kube-bench-f4dqn   0s          Warning   FailedScheduling   pod/kube-bench-master-t5l89   0/6 nodes are available: 1 node(s) had taint {is_edge: true}, that the pod didn't tolerate, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had taint {is_control: true}, that the pod didn't tolerate.

kubectl get pods -n kube-bench-f4dqn -o wide

NAME                      READY   STATUS    RESTARTS   AGE     IP       NODE     NOMINATED NODE   READINESS GATES
kube-bench-master-t5l89   0/1     Pending   0          2m53s   <none>   <none>   <none>           <none>

What did you expect to happen:

I expected the test case executed successfully.

Environment

kubectl version

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:49:13Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:42:57Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

Running processes

Can't schedule the POD.

Configuration files

Anything else you would like to add:

The cluster contains:

• 3 control nodes (with taint)
• 2 worker nodes
• 1 edge node (with taint)