TestSys is a framework that leverages Kubernetes (K8s) to create resources and run tests. To learn more about how it works, see the design document.
Any image can be built using make <desired image>.
To make all bottlerocket related images, make images can be used, then make tag-images can be used to give all images the same tag and make publish-images can be used to publish all images.
TestSys relies on Docker images to run tests and create resources. The Dockerfile contains the logic needed to create all bottlerocket related images.
TestSys is broken up into 3 main things: the controller, test agent, and resource agent.
agent-common contains logic that is related to both the test and resource agents.
test-agent provides the framework for creating tests.
A test is a program that runs a test and reports its outcome by updating the TestSys Test CRD object's status fields.
The test agent is packaged into a container image and the controller runs it in a pod as a Kubernetes Job.
To create a test agent, the Runner trait needs to be implemented.
Check out the example test agent to see how to create a test agent.
resource-agent provides the framework for creating resources.
A resource agent can create and destroy external resources (such as Kubernetes clusters or compute instances) that are needed for a test.
The resource agent program is packaged into a container image and is represented by a Resource CRD object.
A test can then depend on the existence of a resource and the controller will ensure the resource agent runs before the test is run.
The resource agent is also responsible for destroying a resource after a test runs.
To create a resource agent, the Create trait and Destroy trait need to be implemented.
Check out the example resource agent to see how to create a resource agent.
The bottlerocket directory of the repo is broken into 3 sections: agents, testsys, and types.
agents contains the implementations of all bottlerocket agents.
The types used by the agents can be found in types.
Cli is an example command line interface that uses the TestManager to interact with a TestSys cluster.
The controller orchestrates the interaction between TestSys tests and resources.
After a TestSys Test CRD or TestSys Resource CRD has been added to the cluster (cli run file), the controller begins the reconciliation process.
For tests, once necessary resources have been created (resources named in the resources field of the test spec), a K8s job is created and the test agent is run.
For resources, once necessary resources have been created (resources named in the depends_on field of the resource spec), a K8s job is created and the create function of the agent is run.
Once a resource is marked for deletion (cli delete), the destroy function of the resource is run, and the finalizers are removed so that the resource can be cleaned up by K8s.
For more info, see the design document.
The testsys-model library contains the infrastructure that TestSys is built upon. This includes the CRDs for tests and resources as well as everything needed to be installed to the TestSys cluster.
All interactions with the TestSys cluster should be done using the testsys-model library through TestManager, TestClient, or ResourceClient.
The model provides TestManager which can be used as an entry point for user interactions with TestSys. To see how to use the TestManager look at the example CLI.
Model also contains a TestClient for simple interactions with Tests and a ResourceClient for Resources.
Selftest is a library used to validate incoming pull request by testing the overall TestSys framework.
TestSys is currently tested on x86_64 machines, these steps may not work as expected on other operating systems.
Create a working directory and navigate to it. The example provided creates a directory in the users home, but a directory may be created anywhere.
cd ~
mkdir testsys && cd testsys
pwdWhile in the TestSys directory, clone the bottlerocket-test-system repo from GitHub.
git clone https://github.com/bottlerocket-os/bottlerocket-test-system.git
cd bottlerocket-test-system
export TESTSYS_DIR=$(pwd)The TestSys directory is now set up.
Instead of creating a long-standing EKS cluster for TestSys development, kind enables the quick creation and destruction of local clusters that exist within the developers Docker which helps improve the testing experience.
Follow the installation instructions to set up kind.
While all testing user interactions can be done with cli, it is helpful to look at test, resource, and pod objects in a K8s cluster with kubectl.
Follow the installation instructions to set up kubectl.
Docker is required for building TestSys agent images and the controller image. If Docker is not installed follow the installation instructions.
cd ${TESTSYS_DIR}First, the images need to be created and tagged so they can be loaded into a kind cluster.
make controller
make example-test-agent
docker tag controller controller:demo
docker tag example-test-agent example-test-agent:demoThis step uses the Makefile targets for building the controller image and the example test agent image. The images then need to be tagged so that the cluster uses loaded images instead of ones from external sources.
Install the cli from the repo.
cargo install --path cliThe yaml file tells the controller and the test agent how the test should be run.
echo '---
apiVersion: testsys.system/v1
kind: Test
metadata:
name: hello-bones
namespace: testsys
spec:
agent:
name: hello-agent
image: "example-test-agent:demo"
keepRunning: false
configuration:
mode: Fast
person: Bones the Cat
helloCount: 3
helloDurationMilliseconds: 500
resources: []
dependsOn: []' > example_test_agent.yamlCheck if the kind cluster already exists.
kind get clustersIf the cluster already exists, it should be deleted.
kind delete cluster --name testsys-demoCreate the new demo cluster.
kind create cluster --name testsys-demoNow the images created earlier need to be added to the cluster.
kind load docker-image controller:demo example-test-agent:demo --name testsys-demoBefore any tests can be run, the K8s cluster needs to understand what a test is and how to handle them. To do this TestSys needs to be installed to the cluster.
cli install --controller-uri controller:demoThe CLI installed earlier gives the ability to run tests on a cluster.
cli run file example_test_agent.yamlThis adds the test to the cluster. Once the controller sees the test, a test pod will be created that runs the image we provided.
The CLI also provides the ability to monitor the test.
If watch is available, watch cli status will continue to update the status until the test is complete.
Otherwise cli status can be called regularly to see updates.
In the event that a test fails, it may be helpful to read through the test logs to see what went wrong.
cli logs --test hello-bonesWhile debugging tests it may be helpful to monitor logs as they become available, the --follow flag is used to print logs that are sent after the initial cli call.
cli logs --test hello-bones --followWhen creating tests and resources it is important to maintain a clean state because tests must have a unique name in a cluster.
For instance, calling cli run file example_test_agent.yaml will cause an error because there is already a test named hello-bones.
To delete all tests and resources from the cluster use cli delete.
Another option is to remove everything from the TestSys namespace using cli uninstall.
This will delete the TestSys namespace and everything inside it, including the controller, secrets, tests and resources.
Note: If a kind cluster is being created and destroyed each time, there is no need to manually delete the tests.
kind delete cluster --name testsys-democd ${TESTSYS_DIR}First, the images need to be created and tagged so they can be loaded into a kind cluster.
make controller
make example-test-agent
make duplicator-resource-agent
docker tag controller controller:demo
docker tag example-test-agent example-test-agent:demo
docker tag duplicator-resource-agent duplicator-resource-agent:demoThis step uses the Makefile targets for building the controller image, the duplicator resource agent image, and the example test agent image. The images then need to be tagged so that the cluster uses loaded images instead of ones from external sources.
Install cli from the repo.
This step can be skipped if the cli is already installed.
cargo install --path cliThe yaml file describes 2 CRDs that will be created.
cli run file has the ability to create as many TestSys objects at the same time.
Each object should be separated by ---.
The first is the duplicator resource.
This resource takes whatever configuration is provided and returns it as a created resource.
In this example {info: 3} is the provided configuration.
The test agent is very similar to the previous example except we have added a depended resource.
The resources field of the test CRD signifies that the test should not be run until all resources have finished.
The other change is the hello count.
In the previous example, it was set to be 3.
In this example, the value is set based on the duplicator resources output.
Any field can be configured to take values from other resources by using ${<resource name>.<field name>}.
echo '---
apiVersion: testsys.system/v1
kind: Resource
metadata:
name: duplicator
namespace: testsys
spec:
agent:
name: dup-agent
image: "duplicator-resource-agent:demo"
keepRunning: false
configuration:
info: 3
dependsOn: []
---
apiVersion: testsys.system/v1
kind: Test
metadata:
name: hello-bones
namespace: testsys
spec:
agent:
name: hello-agent
image: "example-test-agent:demo"
keepRunning: false
configuration:
mode: Fast
person: Bones the Cat
helloCount: ${duplicator.info}
helloDurationMilliseconds: 500
resources: [duplicator]
dependsOn: []' > example.yamlCheck if the kind cluster already exists.
kind get clustersIf the cluster already exists, it should be deleted.
kind delete cluster --name testsys-demoCreate the new demo cluster.
kind create cluster --name testsys-demoNow the images created earlier need to be added to the cluster.
kind load docker-image controller:demo example-test-agent:demo duplicator-resource-agent:demo --name testsys-demoBefore any tests can be run, the K8s cluster needs to understand what a test is and how to handle them. To do this TestSys needs to be installed to the cluster.
cli install --controller-uri controller:demoThe CLI installed earlier gives the ability to run tests on a cluster.
cli run file example.yamlThis adds the test and duplicator resource to the cluster.
The CLI also provides the ability to monitor the test.
If watch is available, watch cli status will continue to update the status until the test is complete.
Otherwise cli status can be called regularly to see updates.
Since the test has duplicator as a resource, the duplicator resource should reach the completed state before the test starts running.
In the event that a resource fails, it may be helpful to read through the resource logs to see what went wrong. The resource state is required to see resource logs because a different pod is used for the resource creation and resource destruction.
cli logs --resource hello-bones --state CreationWhile debugging resources it may be helpful to monitor logs as they become available, the --follow flag is used to print logs that are sent after the initial cli call.
cli logs --resource hello-bones --state Creation --followWhen creating tests and resources it is important to maintain a clean state because tests must have a unique name in a cluster.
To delete all tests and resources from the cluster use cli delete.
Another option is to remove everything from the TestSys namespace using cli uninstall.
This will delete the TestSys namespace and everything inside it, including the controller, secrets, tests and resources.
Note: If a kind cluster is being created and destroyed each time, there is no need to manually delete the tests/resources.
kind delete cluster --name testsys-demo