test: support testing on external clusters

This is useful for bare metal testing.

docs/autotest.md

@@ -144,3 +144,70 @@ It is also possible to run just the sanity tests until one of them fails:
 $ REPO_ROOT=`pwd` ginkgo '-focus=sanity' -failFast ./test/e2e/
 ...
 ```
+
+## Testing on an existing cluster
+
+This can be done by emulating what `make start` does when setting up a
+QEMU-based cluster. Here is a (not necessarily complete) list:
+- Prepare a directory outside of `_work` with the following files.
+- Create `ssh.<0 to number of nodes -1>` scripts such that each script
+  logs into the corresponding node or executes commands. For example:
+``` console
+$ cat igk/ssh.0
+#!/bin/sh
+
+ssh igk-1 "$@"
+```
+- Create `ssh.<host name>` symlinks to the corresponding `ssh.<number>` file.
+- Create a `kubernetes.version` file with `<major>.<minor>` versions, for example 1.19.
+- Create `kube.config` and ensure that a local kubectl command can use it
+  to connect to the cluster. SSH portforwarding may be necessary for remote
+  clusters. The config must grant admin permissions.
+- Ensure that `ssh.<number> <command>` works for the following commands:
+  - `kubectl get nodes` (only needed for `ssh.0`)
+  - `sudo pvs`
+  - `sudo ndctl list -NR`
+- Label all worker nodes with PMEM with `storage=pmem` and
+  `feature.node.kubernetes.io/memory-nv.dax=true`.
+- Delete all namespaces.
+- On OpenShift 4.6 and 4.7: create the `scheduler-policy` config map with a fixed
+  host port (default: `TEST_SCHEDULER_EXTENDER_NODE_PORT=32000`) and reconfigure
+  `scheduler/cluster` as explained
+  in [OpenShift scheduler configuration](install.md#openshift-scheduler-configuration).
+  There is one crucial difference for the config map: in the `managedResources` array,
+  it must also include an entry for `second.pmem-csi.intel.com/scheduler` (use by
+  operator-lvm-production). The corresponding service will be created when deploying
+  PMEM-CSI as part of the tests.
+- Symlink `_work/<cluster>` to the directory.
+- Push a pmem-csi-driver image to a registry that the cluster can pull from.
+  The normal `make push-images` and `make test_e2e` work when overriding the test config
+  with a `test/test-config.d/xxx_external-cluster.sh` file that has the following content.
+  This is just an example, quay.io also works.
+```
+TEST_LOCAL_REGISTRY=docker.io/pohly
+TEST_PMEM_REGISTRY=docker.io/pohly
+TEST_BUILD_PMEM_REGISTRY=docker.io/pohly
+TEST_LOCAL_REGISTRY_SKIP_TLS=false
+```
+- Run `CLUSTER=<cluster> TEST_HAVE_OLM=false make test_e2e TEST_E2E_SKIP="raw.conversion"`. On OpenShift,
+  use `TEST_HAVE_OLM=true`. The only direct TCP connection is the one for the API server,
+  so with port forwarding a single developer machine can test multiple different remote
+  clusters.
+
+## Using ndctl on an OS which does not provide it
+
+If `ndctl` is not available for the OS but containers can be run, then
+the following workaround is possible. Create a wrapper script:
+``` ShellSession
+sudo tee /usr/local/bin/ndctl <<EOF
+#!/bin/sh
+
+podman run --privileged -u 0:0 --rm docker.io/intel/pmem-csi-driver:v0.9.1 ndctl "\$@"
+EOF
+sudo chmod a+rx /usr/local/bin/ndctl
+```
+
+Then ensure that `sudo ndctl` finds the wrapper there:
+``` ShellSession
+sudo sed -i -e 's;\(secure_path.*\);\1:/usr/local/bin;' /etc/sudoers
+```

test/start-kubernetes.sh

@@ -470,19 +470,21 @@ function init_pmem_regions() {
 }
 
 function check_status() { # intentionally a composite command, so "exit" will exit the main script
-    if ${INIT_CLUSTER} ; then
-        deployments=$(govm list -f '{{select (filterRegexp . "Name" "^'${DEPLOYMENT_ID}'-master$") "Name"}}')
-        if [ ! -z "$deployments" ]; then
-            echo "Kubernetes cluster ${CLUSTER} is already running, using it unchanged."
-            kubernetes_usage
-            exit 0
-        fi
-    else 
-        vm_count=$(govm list -f '{{select (filterRegexp . "Name" "^'$(node_filter ${NODES[@]})'$") "Name"}}' | wc -l)
-        if [ $vm_count == ${#NODES[@]} ]; then
-            echo "All needed nodes are already running, using them unchanged."
-            exit 0
+    if [ -e "${CLUSTER_DIRECTORY}/ssh.0" ]; then
+        # Directory and thus (presumably) the cluster exists.
+        if ${INIT_CLUSTER}; then
+            # Run some sanity checks.
+            local nodes
+            if ! nodes=$(${CLUSTER_DIRECTORY}/ssh.0 kubectl get nodes); then
+                die "'${CLUSTER_DIRECTORY}/ssh.0 kubectl get nodes' failed"
+            fi
+            num_nodes=$(( $(echo "$nodes" | wc -l) - 1 ))
+            num_ssh=$( ls -1 ${CLUSTER_DIRECTORY}/ssh.* | grep '/ssh.[0-9][0-9]*$' | wc -l )
+            if [ $num_ssh -ne $num_nodes ]; then
+                die "expected number of nodes $num_nodes to match number of ssh scripts $num_ssh"
+            fi
         fi
+        exit 0
     fi
 
     if ${TEST_CHECK_KVM} && [ ! -e /dev/kvm ]; then