2.4.0

Core Component Updates

• Fixes the permission issue faced by container user (of different litmus components) to access its directory in an Openshift cluster as the openshift cluster runs the container with an arbitrary uid in the root group.

• Enhance the stress chaos experiments to gracefully terminate the stress process in case it is not completed and remove on its own.

• Enhance the node drain experiment to skip the uncordon step post chaos if the target node is not available. It occurs if we enable the autoscaling for the node pool on the cloud provider.

• Updates the experiment contributing docs with the usage of litmus SDK and also adds the chaos-chart details in the repo readme.

• Adds an overview section in litmus-python with the detailed steps for preparing and using the experiment and workflows in python.

• Adds new e2e-pipeline for testing the vm-ware experiments from hub. It supports both nightly and manual builds.

• Enhance the e2e-dashboard to show the logs of the tests running in the e2e pipeline.

• Enhance GitHub Chaos Actions to support AWS EKS clusters, with this now we can integrate any external EKS cluster with chaos action to run the experiments.

Chaos Center Updates

• Upgrading argo workflow and event tracker API versions to support chaos center on Kubernetes 1.22.0+

• Addition of a new event tracker policy operator (known as Change) to track values of any field in the Kubernetes resource and removed the dependency of event tracker from last applied configuration annnotation

• Addition of new permissions to docker images of ChaosCenter server to OpenShift friendly

• Migration of APIs from graphql-server to auth-server and introduction of gRPC for the internal communication

• Addition of "--skipSSL" flag in litmusctl to disable SSL for the communication between control-plane and execution plan

• IPV6 support for control plane

• Addition of new E2E tests for observability.

Notes

• The CRDs for ChaosCenter will be versioned now . So, CRDs for ChaosCenter v2.x.x will be available in 2.x.x branch.
• The Project DB migration has been done in this release along with some enhancements (like addition of new flag --skipSSL), so any version of litmusctl below v0.7.0 won't be compatible with ChaosCenter 2.4.0 or above.

Installation

Litmus-2.4.0 (Stable) Cluster Scope manifest

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.4.0/mkdocs/docs/2.4.0/litmus-2.4.0.yaml

Or

Litmus-2.4.0 (Stable) namespace scope manifest.

export LITMUS_PORTAL_NAMESPACE="<namespace>"
kubectl create ns ${LITMUS_PORTAL_NAMESPACE}
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.4.0/litmus-portal/litmus-portal-crds.yml
curl https://raw.githubusercontent.com/litmuschaos/litmus/2.4.0/mkdocs/docs/2.4.0/litmus-namespaced-2.4.0.yaml --output litmus-portal-namespaced-k8s-template.yml
envsubst '${LITMUS_PORTAL_NAMESPACE}' < litmus-portal-namespaced-k8s-template.yml > ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml
kubectl apply -f ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml -n ${LITMUS_PORTAL_NAMESPACE}

Shoutouts!

Thanks to our existing and new contributors for this release- @Aman-Codes @DhananjayPurohit @tuananh @andrewhu-hcl @Vr00mm @nicjohnson145 @DarthBenro008 @mehabhalodiya @bbarin @amityt @gdsoumya @Saranya-jena @SarthakJain26 @ispeakc0de @Jonsy13 @rajdas98 @uditgaurav

2.3.0

New Features & Enhancements

• Adds support for ipv6 inside network experiments and other litmus components which enable us to run litmus on ipv6 (ipFamily) clusters

• During the initial few seconds of the pod run, sometimes pods failed to request the kubeapi server. Adds the retries inside experiment & runner pods while querying kubeapi server.

• Upgrade the Pumba older version(0.8.0) to the latest version. It fixes the vulnerability of the Pumba binaries.

• Adds the networking policies permissions(which are required for the pod-network-partition experiment) inside the litmus-admin rbac.

• Removes an extra job-name label from the helper pods and refactor the experiment code to improve the logs

• Fixes stress chaos experiment with Pumba lib to take custom stress image

• Reorder the RBAC permissions and add comments description for all the experiments

• Updates the Kyverno pod security policies. Also, enhance the FAQ and PSP in the experiment-docs

• Adds test-cases, readme, and rbac for pod-delete experiment in litmus-python. Modified the litmus-python SDK documentation for the latest changes

• Introduces chaosresult tag annotation to the dashboard for Postgres to add.

• Enhance e2e-dashboard to include card view for different pipelines, support dark-mode, and add all-workflows.

ChaosCenter -

• Fixed Subscriber version compatibility check for the smooth upgrade process.

• Upgraded Litmus-ui version for components optimization.

• Updated the token name to litmus-cc-token.

• Added advanced tune section for workflows for better visibility & allowing users to have more control over workflows.

• Optimized upload functionality for allowing upload of basic argo workflow ( Users will be able to run any generic workflow )

• Fixed some bugs in the frontend & graphql-server.

• Enhanced E2E pipeline by adding resource validation on k8s cluster after scheduling the workflows.

Note:

• With this release of ChaosCenter, Litmus core components are upgraded to v2.2.0 & default chaoshub is updated to 2.2.x

Installation

Litmus-2.3.0 (Stable) Cluster Scope manifest

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.3.0/mkdocs/docs/2.3.0/litmus-2.3.0.yaml

Or

Litmus-2.3.0 (Stable) namespace scope manifest.

export LITMUS_PORTAL_NAMESPACE="<namespace>"
kubectl create ns ${LITMUS_PORTAL_NAMESPACE}
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/litmus-portal/litmus-portal-crds.yml
curl https://raw.githubusercontent.com/litmuschaos/litmus/2.3.0/mkdocs/docs/2.3.0/litmus-namespaced-2.3.0.yaml --output litmus-portal-namespaced-k8s-template.yml
envsubst '${LITMUS_PORTAL_NAMESPACE}' < litmus-portal-namespaced-k8s-template.yml > ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml
kubectl apply -f ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml -n ${LITMUS_PORTAL_NAMESPACE}

2.2.0

New Features & Enhancements

• Adds more secure litmus hardened alpine base images to all litmus components.

• Fixes the image vulnerability, add security scanner and health checks for all the litmus components

• Adds the ability to provide terminationGracePeriodSeconds inside the helper pods. This also fixes the execution of node-level service kill experiments for a longer chaos duration.

• Refactor the experiments with enhanced logging, error handling, and go-report.

• Adds support to execute the vm-poweroff experiment in serial or parallel sequence

• Adds the okteto configuration to test the experiments generated by litmus python-sdk

• Introduces E2E tests for the litmus-python repository via GitHub actions

• Adds kyverno pod security policies for the litmus pods to enable the execution of all supported (out-of-the-box) experiments listed in the chaoshub.

• Adds the support of the hostNetwork inside the cmdProbe in workflows.

• Introduced a first-cut visualization of the e2e dashboard to monitor different pipelines and stability of an experiment

• Add a new pipeline to test GCP based chaos experiments that is vm-instance-stop and vm-disk-loss tests

ChaosCenter

• Adds frontend Bug fixes in Probes wizard, Analytics & Teaming

• Adds Nginx configuration in a separate configMap for more configuration

• Adds kelseyhightower/envconfig pkg for validating the environment variables

• Adds RBAC for the newly added mutations

• Adds Agent Upgrade feature, which will help in upgrading agents when the control plane is upgraded.

• Moves DB_ADMIN, DB_PASSWORD, and JWT_SECRET environment variables to k8s secret

• Adds option for adding tolerations to the agent plane components using litmusctl

• Enhances the ChaosCenter e2e pipeline for testing with LoadBalancer & Ingress based access setups.

Note:

• With this release of ChaosCenter, Litmus core components are upgraded to v2.1.1, which is having cmd probeSchema changes. In case users have their workflows scheduled with an older schema, they will have to update it according to the new schema.

• For using agent upgrade feature & tolerations addition to agents plane components, Users have to use v0.5.0 of litmusctl

Installation

Litmus-2.2.0 (Stable) Cluster Scope manifest

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.2.0/mkdocs/docs/2.2.0/litmus-2.2.0.yaml

Or

Litmus-2.2.0 (Stable) namespace scope manifest.

export LITMUS_PORTAL_NAMESPACE="<namespace>"
kubectl create ns ${LITMUS_PORTAL_NAMESPACE}
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/litmus-portal/litmus-portal-crds.yml
curl https://raw.githubusercontent.com/litmuschaos/litmus/2.2.0/mkdocs/docs/2.2.0/litmus-namespaced-2.2.0.yaml --output litmus-portal-namespaced-k8s-template.yml
envsubst '${LITMUS_PORTAL_NAMESPACE}' < litmus-portal-namespaced-k8s-template.yml > ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml
kubectl apply -f ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml -n ${LITMUS_PORTAL_NAMESPACE}

2.1.0

New Features & Enhancements

• Introduces azure-disk-loss experiment, which detaches the virtual disk from an Azure instance

• Introduces pod-network-partition experiment, which blocks Ingress and Egress traffic of the target application. It provides the option to specify the pod-selector, namespace-selector, and ports.

• Introduces the ability to provide default application status checks as a tunable for the pod-specific experiments from the ChaosEngine. It is helpful for the scenarios where we don’t want to validate the application status as a mandatory check during pre & post chaos.

• Enhance the GCP VM Instance Stop experiment for VMs that are a part of an Autoscaling Group, where the VMs are now awaited to fully stop before proceeding with the experiment.

• Enhancements for the cmdProbe to run the probe pod with the host network.

• Enhance probe pod functionality where the experiment serviceAccount is now passed to the probe pod. It is helpful for the scenarios where the probe pod needs specific RBAC permissions for the execution.

• litmuschaos_awaited_experiment metric contains an injection_time label. Adds injection_time label as a tunable via the INJECTION_TIME_FILTER ENV. Which can be used for the annotations inside the interleaving dashboards

• Enhance the chaos-scheduler to specify the minutes and hours separately. This provides the ability to schedule the chaos at the nth minutes per x hours. Which will be helpful for the staggered schedules to avoid collisions.

• Adds the ability to search experiments based on certain keywords in Chaos Hub. Added all the related keywords for individual experiments.

• Improves the error handling in the chaos-operator for chaosengine status nil check

• Removes the init container which modifies the permissions of the docker socket path, and instead provides the docker socket path inside the DOCKER_HOST ENV for the Pumba Lib.

• litmus metrics are now exported only for the active ChaosEngines. It stops overriding the value of existing metrics as soon as ChaosEngine comes to the completed state.

• Enhance the e2e pipeline by adding debug steps and using shared runners to run tests in parallel, also allow the ability to run a test with a custom service account to validate a specific set of permissions required for an experiment.

Major Bug Fixes

• Fixes chaos-result name generation where the experiment-name and instance-id are passed inside the helper pods, which are used to generate the chaos-result name.

Installation

kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v2.1.0.yaml

Verify your installation

• Verify if the chaos operator is running
  kubectl get pods -n litmus

• Verify if chaos CRDs are installed
  kubectl get crds | grep chaos

For more details refer to the documentation at Docs

2.0.0

Introduction

Version 2.0.0 brings newer capabilities to the LitmusChaos platform, enabling a more efficient practice of chaos engineering. The major version upgrade is being carried out to reflect significant improvements and new features in the platform - many of which were introduced & curated across several preceding 2.0 beta releases with community feedback (thanks to all the early adopters & beta testers for your continued support. Some of these changes, especially. newer experiments and observability improvements have been made available in 1.x too).

Litmus 1.x brought a cloud-native approach to chaos engineering to the definition and execution of chaos intent, along with a ready set of experiments maintained in the ChaosHub. Along the way, newer requirements were incorporated into the project, most notably around a centralized management approach for managing chaos across environments (K8s clusters and cloud instances) and the ability to define workflows to stitch together multiple experiments as part of a complex scenario.

The 2.0 GA release brings these features into the mainstream, having been validated for their usefulness & architecture. Subsequent improvements to these will be carried out in 2.x releases. Some salient features are described briefly in below sections:

Chaos Center

• A chaos control plane or portal which provides centralized management of chaos operations on multiple clusters across datacenters/cloud. The control plane carries out experiments through agents installed on the registered clusters.
• Comprises documented APIs that can be used to invoke chaos programmatically
• Provides visualization capabilities and analytics around chaos execution.
• Supports a project-teams-users structure to enable collaboration within teams for chaos operations.

Litmus Workflows

• Introduces chaos workflows - to (a) automate dependency setup (b) aid creation of complex chaos scenarios with multiple faults (c) support definition of load/validation jobs along with chaos injection
• Provides flexibility in creating/running workflows in different ways - via templates, from an integrated hub, and custom uploads.

Multi-Tenancy

• Supports setup (control plane & agents) and execution of chaos experiments in both: cluster-scoped and namespace-scoped modes to help operations in shared clusters with a self-service model

Observability & Steady State Hypothesis Validation

• Provides an increased set of Prometheus metrics with additional filters - which can be used for instrumenting application dashboards to observe chaos impact
• Provides diverse set of probes to automate validation of steady-state hypothesis - thereby improving the efficiency of running automated chaos experiments

GitOps for Chaos

• Integrates with Git-based SCM to provide a single-source-of-truth for chaos artifacts (workflows), such that changes are synchronized bi-directionally b/w the git source and the chaos center - thereby pulling the latest artifact for execution.
• Provides an event-tracker microservice to automatically launch “subscribed” chaos workflows upon app upgrades effected by GitOps tools like ArgoCD, Flux

Non-Kubernetes Chaos

Adds experiments to inject chaos on infrastructure (cloud) resources such as VMs/instances and disks (AWS, GCP, Azure, VMWare) - irrespective of whether they host a Kubernetes cluster or not.

Release Cadence & Versioning

The release cadence & naming conventions continue to adhere to the principles followed thus far in the Litmus project: the monthly minor version releases (2.x.0) will happen on the 15th, with patch releases/hotfixes going into 2.x.x, on a need/demand basis. The 1.x version will be stopped at 1.13.x (1.13.8 at this point) and further patches will be made only upon request/community need.

Backward Compatibility

Having said that, Litmus 2.x completely lends itself to the 1.x mode of execution the users are familiar with, i.e., you could still continue to deploy the latest chaos-operator deployment in admin/namespace mode, pull ChaosExperiment templates/CRs from the ChaosHub & trigger chaos by applying the ChaosEngine CR. The latest chaos-exporter & chaos-scheduler will continue to be operable as they are. However, the introduction of the Chaos-Center (also commonly referenced as Litmus Portal by the beta test community) simplifies the above process greatly while giving you additional nuts & bolts.

Migration from 1.x to 2.x

To make use of the Chaos Center and other capabilities of Litmus 2.0, please remove any existing ChaosEngines, uninstall the chaos operator deployment & follow the Litmus 2.x installation instructions.

If you would like to consume just the backend infrastructure components (chaos operator, crds et al), please follow the regular procedure in applying the latest operator manifest or start using the operator helm chart to allow for subsequent helm upgrades.

If you are a beta user on 2.0.0-beta9, follow the upgrade procedure to start using the Litmus 2.0 GA build.

Documentation

The documentation has undergone considerable changes - in terms of content and structure and it continues to undergo improvements as of the 2.0 release. We expect that a few more iterations are needed to sort out the Information Architecture.

The installation details for the 2.0 platform along with detailed introductions to concepts, architecture as well as a user guide are now available at https://docs.litmuschaos.io/

The latest chaos experiment details along with chaos custom resource schema specifications (tunables, examples, etc.,) and detailed FAQs & troubleshooting info can be found in https://litmuschaos.github.io/litmus/

For those continuing to use 1.x releases, please note that the docs are now moved to: https://v1-docs.litmuschaos.io/

Misc (monthly changelog between 15/07/2021 to 15/08/2021)

Notes on changes to control plane (chaos center) since 2.0.0-beta9

• Added new API routes to check the status of the authentication server and to update the user details
• Added an API to terminate chaos workflow
• Added namespace scope support for event tracker
• Bugs fixes/enhancement in the frontend
• Typo in the nodeSelector schema key
• Adheres to correct schema in the steady-state validation wizard for Litmus Probes
• Fixes the inability to login/authenticate after upgrade of chaos-center

Notes on changes to backend execution infrastructure (chaos operator, experiments) since 1.13.8

• Supports VM belonging to scale-sets (VMSS) as target resources in the Azure instance stop experiment
• Fixes the limitation/inability to perform abort operations in the namespaced mode of operation in a chaos operator.
• Fixes an issue (edge case in scaled scenarios) within the abort functionality for the “exec” based chaos experiments (pod-cpu-hog-exec & pod-memory-hog-exec) wherein chaos injection continues to occur even post issual of abort.
• Adds fix to fail faster when helper pods do not run successfully in an experiment (fail immediately upon identifying helper failure instead of waiting for the customary statusCheckTimeout of 180s, as the helper pods are usually brought up with restartNever policy)
• Fixes the inability of certain experiments (pod-cpu-hog-exec, pod-memory-hog-exec and pod-dns-error, pod-dns-spoof) to select targets serially for cases where 0 > PODS_AFFECTED_PERC <= 100.
• Adds a condition to error out/call out the engine schema when neither the .spec.appinfo.applabel nor TARGET_PODS env are specified.
• Adds missing ability to perform auxiliary application health check in the node-memory-hog experiment and missing support for specifying multiple target nodes via a comma-separated list in node-cpu-hog, node-memory-hog & node-io-stress experiment.
• Fixes a regression in recent 1.13.x experiments wherein the .spec.appinfo.appkind is mandated (in order to derive parent controller name for pods - as this is used to patch the chaosresult status with target info) even when .spec.annotationCheck set to false. With this fix, you will be able to see older behavior wherein appkind can be left empty for cases where annotationCheck is set to false in the ChaosEngine CR.

2.0.0-RC1

Fixed docs versioning (#3106)

Signed-off-by: Amit Kumar Das <[email protected]>

2.0.0-Beta9

Major Updates

• Refactored the authentication server to reduce the network latency and added support for enabling/disabling users
• Added a new pre-defined chaos workflow to check the resiliency of the Bank of Anthos application
• Enhanced the analytics module with API and user interface optimization
• Added a usage module in the litmusportal for admin users to check the agent and workflow usage across all accounts
• Replaced all the privileged permissions with minimal permissions in the cluster and namespace manifest
• Built a mirror docker image of frontend application with the based path of “/litmuschaos” that can be used in ingress paths
• Optimized the UI rendering by scoping out the header and sidebar scaffold to a global level and enhanced the user experience with the support of code-splitting at the component level.
• Added support for multi-architecture(ARM64 and AMD64) docker images for Argo workflow executor and Argo workflow controller
• Enhanced the E2E pipeline with parallel build to fasten the testing using comment driven approach

Minor Updates

• Added node Selector as a configurable environment variable for agent manifests
• Added support to tune chaos workflow environment variables from the user interface
• Upgraded the go version to 1.16, litmus UI to 1.4.0, and MyHub go pkg to a stable version of go-git

NOTE: With the revamp of authentication module and addition of a new feature Deactivate/Activate User, we have made some changes in Auth Schema and module. This has been done to reduce the network latency and optimize the user experience. With these new changes, who have been running Portal previously won't be able to login after upgrading. We are working on the streamlined Upgrade/Migration process, for now we suggest you to re-setup portal for upgrading to Beta9.

Installation

Applying k8s manifest

Litmus-2.0.0-Beta9 (Stable) cluster scope manifest

kubectl apply -f https://litmuschaos.github.io/litmus/2.0.0-Beta/litmus-2.0.0-Beta.yaml

Or

Litmus-2.0.0-Beta9 (Stable) namespace scope manifest.

export LITMUS_PORTAL_NAMESPACE="litmus"
kubectl create ns ${LITMUS_PORTAL_NAMESPACE}
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/litmus-portal/litmus-portal-crds.yml
curl https://raw.githubusercontent.com/litmuschaos/litmus/master/docs/2.0.0-Beta/litmus-namespaced-2.0.0-Beta.yaml --output litmus-portal-namespaced-k8s-template.yml
envsubst < litmus-portal-namespaced-k8s-template.yml > ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml
kubectl apply -f ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml -n ${LITMUS_PORTAL_NAMESPACE}

2.0.0-Beta8

Major Updates

• Added chaos scheduler dependency in the control plane server to support cron scheduling.
• Added support for ARM64 architecture for Litmus control plane and agent plane components
• A warning will be displayed for long-running workflow after 20 mins and added support for deleting and syncing long-running chaos workflows.
• Added support to use predefined workflow from the connected MyHubs with the project.
• Optimised the workflow’s graphql queries with support of pagination, sorting, and filtering.
• Introduced chaos engine as a standalone workflow which makes it independent of Argo Workflow.
• Added podGC strategy along with revert chaos to remove workflow related artifacts after the completion of workflow
  Note: Logs won’t be accessible if the revert chaos step is enabled.
• Added new graph components (stack-bar graph, line-area graph, and radial charts) in the litmus-ui NPM package
• Restructured the directory of the subscriber and bug fixes of kubeobject in the namespace mode.
• Redesigned analytics dashboards with the new UI components and enhanced some of the existing features like workflow comparison, query manipulation, and dashboard operation. Also, Optimised the graphql APIs to fasten the real-time response.

Minor Updates

• Removed Argo server deployment and its dependencies from the agent plane list
• Enhanced the user interface of workflow editor in the litmus portal.

2.0.0-Beta7

Added CRD for Event-tracker (#2812)

Signed-off-by: Jonsy13 <[email protected]>

2.0.0-Beta6

Major Updates

• Added MongoDB go-interface and refactored the database operations and structure to accommodate the test cases easily.
• Support for adding custom container image registry to chaos workflow manifest.
• Enhanced the performance of the analytics APIs with memory caching and added APIs to fetching labels and values for a Prometheus series.
• Added support for mutating the sequence of the workflow steps by drag and drop which reflect the live changes in the DAG.
• Enhanced the workflow graph to show other node phases such as Omitted, Skipped, and Error for a good user experience.
• Enhanced the verify and commit page to allow users to have a final review and edit their workflow details before scheduling the same.
• Bug fixed for some user management operations and refactored teaming APIs to increase the performance.
• Enhanced the litmusportal user interface to fastens the onboarding process.

Minor Updates

• Adding support for liveness check of the dependent applications in the agent plane before going active.
• AirGapped support for the pre-defined workflows by moving the fetching logic to the backend.
• Added instance-id label in the chaos workflow manifest to avoid multiple scheduling in the multi-Argo server cluster.
• Added validations for workflow name, GitHub URL, and different probe inputs.