Skip to content

Commit faa6e48

Browse files
suhyenimmoggaanamkyu1999
authored
feat: add otel-demo tutorial (#4858)
* feat: add otel-demo tutorial Co-authored-by: Suhyen Im <[email protected]> Co-authored-by: Jaeyeon Park <[email protected]> Signed-off-by: Suhyen Im <[email protected]> Signed-off-by: Jaeyeon Park <[email protected]> * feat: add otel-demo screenshots Co-authored-by: Suhyen Im <[email protected]> Co-authored-by: Jaeyeon Park <[email protected]> Signed-off-by: Suhyen Im <[email protected]> Signed-off-by: Jaeyeon Park <[email protected]> * chore: update tutorial and architecture image Co-authored-by: Suhyen Im <[email protected]> Co-authored-by: Jaeyeon Park <[email protected]> Signed-off-by: Suhyen Im <[email protected]> Signed-off-by: Jaeyeon Park <[email protected]> --------- Signed-off-by: Suhyen Im <[email protected]> Signed-off-by: Jaeyeon Park <[email protected]> Co-authored-by: Jaeyeon Park <[email protected]> Co-authored-by: Namkyu Park <[email protected]>
1 parent dcc7f60 commit faa6e48

14 files changed

+1486
-0
lines changed

‎monitoring/README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,10 @@ This directory contains chaos interleaved grafana dashboards along with the util
1212

1313
> Contains utilities required to setup monitoring infrastructure on a kubernetes cluster.
1414
15+
- [Tutorials](./tutorials)
16+
17+
> Contains tutorials for users on monitoring target applications under chaos using various tools.
18+
1519
## Setup the LitmusChaos Infrastructure
1620

1721
- Install the litmus chaos operator and CRDs

‎monitoring/tutorials/README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Tutorials
2+
3+
This directory contains tutorials for users on monitoring target applications under chaos using various tools.
4+
5+
- [Otel-demo](./otel-demo)
6+
7+
> Contains a tutorial on injecting chaos into target applications using LitmusChaos and observing the chaos with OpenTelemetry.

‎monitoring/tutorials/otel-demo/README.md

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# Otel-demo tutorial
2+
3+
This tutorial provides a step-by-step guide for injecting chaos into target applications using LitmusChaos and observing the chaos with OpenTelemetry.
4+
5+
<img src="./screenshots/otel_demo_tutorial_architecture.png" alt="otel_demo_tutorial_architecture" width="800" />
6+
7+
### 0. Prerequisites
8+
- Kubernetes 1.24+
9+
- 8 GB of free RAM
10+
- Helm 3.9+
11+
12+
### 1. Install Litmus
13+
1. Create the `litmus` namespace.
14+
```bash
15+
kubectl create ns litmus
16+
```
17+
2. Add the Litmus Helm repository.
18+
```bash
19+
helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
20+
```
21+
3. Install Litmus using Helm.
22+
```bash
23+
helm install chaos litmuschaos/litmus \
24+
--namespace=litmus \
25+
--set portal.frontend.service.type=NodePort \
26+
--set mongodb.image.registry=ghcr.io/zcube \
27+
--set mongodb.image.repository=bitnami-compat/mongodb \
28+
--set mongodb.image.tag=6.0.5
29+
```
30+
4. Verify the installation.
31+
```bash
32+
kubectl get all -n litmus
33+
```
34+
5. Forward the Litmus frontend service port.
35+
```bash
36+
kubectl port-forward svc/chaos-litmus-frontend-service 9091:9091 -n litmus
37+
```
38+
Access the Litmus frontend at [http://localhost:9091](http://localhost:9091) and log in with `admin` / `litmus`.
39+
40+
### 2. Set Up Litmus Environment
41+
1. Create a new environment.
42+
- Environment Name: `local`
43+
- Environment Type: `Production`
44+
2. Configure a new chaos infrastructure.
45+
- Name: `local`
46+
- Chaos Components Installation: `Cluster-wide access`
47+
- Installation Location (Namespace): `litmus`
48+
- Service Account Name: `litmus`
49+
3. Deploy the new chaos infrastructure.
50+
```bash
51+
cd ~/Downloads
52+
kubectl apply -f local-litmus-chaos-enable.yml
53+
```
54+
Wait until the status shows `CONNECTED`.
55+
56+
### 3. Install Otel-demo microservices & Observability tools
57+
1. Create the `otel-demo` namespace.
58+
```bash
59+
kubectl create ns otel-demo
60+
```
61+
2. Add the OpenTelemetry Helm repository.
62+
```bash
63+
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
64+
```
65+
3. Install Otel-demo microservices and Observability tools using Helm.
66+
```bash
67+
cd litmus/monitoring/tutorials/otel-demo
68+
helm install my-otel-demo open-telemetry/opentelemetry-demo --namespace otel-demo --values custom_otel_demo_values.yml
69+
```
70+
It contains Otel-demo microservices, OpenTelemetry(with chaos metrics), Prometheus, Jaeger and Grafana.
71+
4. Verify the installation.
72+
```bash
73+
kubectl get all -n otel-demo
74+
```
75+
5. Forward the Otel-demo frontend proxy port.
76+
```bash
77+
kubectl port-forward svc/my-otel-demo-frontendproxy 8080:8080 -n otel-demo
78+
```
79+
6. Access the following services.
80+
- Web store: [http://localhost:8080/](http://localhost:8080/)
81+
- Grafana: [http://localhost:8080/grafana/](http://localhost:8080/grafana/)
82+
- Load Generator UI: [http://localhost:8080/loadgen/](http://localhost:8080/loadgen/)
83+
- Jaeger UI: [http://localhost:8080/jaeger/ui/](http://localhost:8080/jaeger/ui/)
84+
85+
### 4. Add Grafana Panel
86+
Import the `chaos-experiments-dashboard.json` file into Grafana to visualize the results of chaos experiments.
87+
88+
### 5. Observe chaos
89+
Explore the following experiments to observe chaos on the Otel-demo microservices.
90+
91+
- [Pod Network Latency](./cart-service)
92+
> Performs a pod network latency experiment on the cart service.
93+
94+
- [Pod Delete](./recommendation-service)
95+
> Performs a pod delete experiment on the recommendation service.

‎monitoring/tutorials/otel-demo/cart-service/README.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# cart service pod network latency
2+
## Description
3+
- This experiment injects network latency to the cart service pod.
4+
- The Probe checks Prometheus metrics Latency of cart service requests.
5+
## Steps
6+
### 1. Probe Settings
7+
- probe type: `Prometheus Probe`
8+
- name: `cart-service-pod-network-latency-probe`
9+
- timeout: 3s
10+
- interval: 3s
11+
- prometheus endpoint: `http://my-otel-demo-prometheus-server.otel-demo:9090`
12+
- prometheus query: `histogram_quantile(0.99, sum(rate(duration_milliseconds_bucket{service_name=\"cartservice\"}[5m])) by (le))/1000`
13+
- Data Comparison:
14+
- Type: Float
15+
- Criteria: `<`
16+
- Value: `3.0`
17+
### 2. Make Experiment
18+
1. New Experimnet
19+
2. Complete Overview
20+
3. Start off by Upload YML(cart-service-pod-network-latency.yml)
21+
### 3. Run Experiment
22+
1. Click on the `Run` button
23+
2. Check Experiment Status and Logs
24+
3. Check the Resilience Score
25+
4. Check the Chaos Exporter metrics using Grafana and confirm if the experiment failed. ![cart_service_pod_network_latency_experiment_result_dashboard.png](../screenshots/cart_service_pod_network_latency_experiment_result_dashboard.png)
26+
5. Check cart service Spanmetrics Metrics using Grafana ![cartservice_spanmetrics.png](../screenshots/cartservice_spanmetrics.png)

0 commit comments

Comments
 (0)