[RC2] [Functest K8s] kube_bench_node TC failed

[sultetveny]

I've faced with an issue running the kubernetes security test using kub_bench.

podman run -it --env-file ~/opnfv/env \
-v ~/opnfv/ca.pem:/home/opnfv/functest/ca.pem:Z \
-v ~/opnfv/config:/root/.kube/config:Z \
-v ~/opnfv/results:/home/opnfv/functest/results:Z \
-v ~/opnfv/repositories.yml:/home/opnfv/functest/repositories.yml:Z \
-v ~/opnfv/cluster-admin.pem:/home/opnfv/functest/cluster-admin.pem:Z \
-v ~/opnfv/cluster-admin-key.pem:/home/opnfv/functest/cluster-admin-key.pem:Z \
opnfv/functest-kubernetes-security:v1.23 /bin/bash

# then within the container
run_tests -t kube_bench_node

Test case failed. For more information please check attached files.
functest-kubernetes.debug (2).log

cat opnfv/results/functest-kubernetes.debug.log 

...

2023-02-22 08:08:30,957 - kubernetes.client.rest - DEBUG - response body: 
failed to get a set of executables needed for tests: unable to detect running programs for component "kubelet"

2023-02-22 08:08:30,958 - functest_kubernetes.security.security - INFO - 


failed to get a set of executables needed for tests: unable to detect running programs for component "kubelet"

2023-02-22 08:08:30,959 - xtesting.ci.run_tests - ERROR - 

Please fix the testcase kube_bench_node.
All exceptions should be caught by the testcase instead!

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/xtesting/ci/run_tests.py", line 171, in run_test
    test_case.run(**kwargs)
  File "/usr/lib/python3.9/site-packages/functest_kubernetes/security/security.py", line 212, in run
    self.details["report"] = ast.literal_eval(self.pod_log)
  File "/usr/lib/python3.9/ast.py", line 62, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/lib/python3.9/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 2
    failed to get a set of executables needed for tests: unable to detect running programs for component "kubelet"
           ^
SyntaxError: invalid syntax
2023-02-22 08:08:30,962 - xtesting.ci.run_tests - ERROR - The test case 'kube_bench_node' failed.
2023-02-22 08:08:30,962 - xtesting.ci.run_tests - INFO - Execution exit value: Result.EX_ERROR

kubectl logs -n kube-bench-phqn5 kube-bench-node-84zg7

failed to get a set of executables needed for tests: unable to detect running programs for component "kubelet"

kubectl get pod -n kube-bench-phqn5 -o wide

NAME                    READY   STATUS   RESTARTS   AGE   IP               NODE                  NOMINATED NODE   READINESS GATES
kube-bench-node-5k66d   0/1     Error    0          9h    192.168.72.66    cbis-sut1-worker-02   <none>           <none>
kube-bench-node-84zg7   0/1     Error    0          9h    192.168.72.104   cbis-sut1-worker-02   <none>           <none>
kube-bench-node-8t65k   0/1     Error    0          9h    192.168.72.123   cbis-sut1-worker-02   <none>           <none>
kube-bench-node-f2z9s   0/1     Error    0          9h    192.168.72.90    cbis-sut1-worker-02   <none>           <none>
kube-bench-node-mdhrw   0/1     Error    0          9h    192.168.72.92    cbis-sut1-worker-02   <none>           <none>
kube-bench-node-qgqnt   0/1     Error    0          9h    192.168.72.78    cbis-sut1-worker-02   <none>           <none>
kube-bench-node-wxl8h   0/1     Error    0          9h    192.168.72.89    cbis-sut1-worker-02   <none>           <none>

# worker-01
ps -eaf | grep kube

root       65980       1  1 Feb07 ?        05:45:56 /usr/local/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --config=/etc/kubernetes/kubelet-config.yml --register-node=true --hostname-override=cbis-sut1-worker-01 --node-labels=is_control=false,is_worker=true,is_edge=false,is_storage=false,bcmt_storage_node=false,rook_storage=false,rook_storage2=false,cpu_pooler_active=false,dynamic_local_storage_node=false,local_storage_node=true,ncs.nokia.com/group=group_02 --register-with-taints= --node-ip=172.20.110.3 --cloud-provider=external --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --v=1
root       66370   66265  0 Feb07 ?        00:05:33 /usr/local/bin/kube-proxy --config=/etc/kubernetes/kube-proxy-config.yml --oom-score-adj=-998
root       78307   78129  0 Feb07 ?        00:00:31 /csi-node-driver-registrar --csi-address=/csi/csi.sock --kubelet-registration-path=/var/lib/kubelet/plugins/cinder.csi.openstack.org/csi.sock --v=1
99         82177   82051  0 Feb07 ?        01:01:20 /usr/bin/agent -metrics-addr 0 -server-addr 127.0.0.1:65432 -v 1 -log-level info -kubelet-endpoint-point 10248 -kubelet-endpoint-ip 127.0.0.1 -kubelet-endpoint-scheme http
root     2990842 2990798  0 17:02 pts/1    00:00:00 grep --color=auto kube

# worker-02
ps -eaf | grep kube

root       66019       1  1 Feb07 ?        06:07:14 /usr/local/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --config=/etc/kubernetes/kubelet-config.yml --register-node=true --hostname-override=cbis-sut1-worker-02 --node-labels=is_control=false,is_worker=true,is_edge=false,is_storage=false,bcmt_storage_node=false,rook_storage=false,rook_storage2=false,cpu_pooler_active=false,dynamic_local_storage_node=false,local_storage_node=true,ncs.nokia.com/group=group_02 --register-with-taints= --node-ip=172.20.110.23 --cloud-provider=external --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --v=1
root       66408   66304  0 Feb07 ?        00:06:12 /usr/local/bin/kube-proxy --config=/etc/kubernetes/kube-proxy-config.yml --oom-score-adj=-998
root       78171   78013  0 Feb07 ?        00:00:30 /csi-node-driver-registrar --csi-address=/csi/csi.sock --kubelet-registration-path=/var/lib/kubelet/plugins/cinder.csi.openstack.org/csi.sock --v=1
99         81804   81680  0 Feb07 ?        01:02:26 /usr/bin/agent -metrics-addr 0 -server-addr 127.0.0.1:65432 -v 1 -log-level info -kubelet-endpoint-point 10248 -kubelet-endpoint-ip 127.0.0.1 -kubelet-endpoint-scheme http
root     2978125 2978090  0 17:02 pts/1    00:00:00 grep --color=auto kube

Need support to investigate the issue.
Ticket also opened here: aquasecurity/kube-bench#1384