[EPIC]: Testground's Infrastructure as Code

[Bidon15]

Background

ATM, testground infrastructure is written in bash scripts. This solves the initial task of using eks to execute test runs.
Nevertheless, this is not enough to reliably use testground for:

• 1. Infra module upgrades
• 2. Metrics collection / aggregation
• 3. Logs collection / aggregation
• 4. Metrics / Logs storage
• 5. CI/CD for big network tests

Hence, we need to redesign the infrastructure to a layers, which will let us be more flexible on setups

• Layer 1 - Basic Infrastructure in EKS(AWS)

---

Goal: Setup basics for the layers above

• 1. Control Plane + node groups + security groups + OIDC
• 2. EBS, EFS, Autoscaling
• S3 -> to check
• 3. ArgoCD

TerraForm is how we are going to implement all 3 points

Every item in points above are treated as TerraForm modules

• testground/infra/layer1: node groups  #148
• testground/infra/cli: add my IP to a security group #149

Layer 2 - Testground Applications(Mandatory)

Goal: make it easy aka helm chart

1. Master helm chart for testground, which contains:
   1. helm charts for each of components that we have rn in functions.sh
   2. ArgoCD will take care of deployments

Required Components (caveats → we need to finalise which ones are DS/STS/RS):

• 1. Redis

  • this

• 2. Multus

• 3. Weave

• 4. Flannel

  • We user the Amazon CNI

• 5. Testground-Sidecar

• 6. Testground-Daemon

• 7. Sync-Service

• testground/infra/layer2: migrating mandatory components to TF #150

• testground/daemon: improvements for cluster:k8s #151

• Layer 3 - Add-ons(dashboards, metrics, observability logs)

---

Goal: full visibility for the dev teams to analyse test runs

Components -> We need to finalise which ones below are DS/STS/RS:

• 1. Prometheus
• 2. InfluxDB
• 3. Promtail
• 4. Grafana
• 5. Metrics Scraper (we need do develop it in-house) or this testground/app: forward metrics from app-nodes to influxdb  #179
• 6. Log Metrics Analysers

[Bidon15]

cc: @sysrex @jrmanes

[smuu]

Why do we need to develop the Metrics Scraper in-house? Can't we use Prometheus for scraping?

[Bidon15]

Cosmos support: #179

[Bidon15]

Grooming 10/03/2023: Moved to Q2
Prob half a day to solve the PoC with 1 avail zone

[Bidon15]

Grooming 12/05/2023:

Putting it to the board. It will be closed by celestiaorg/devops#123

[Bidon15]

Closing as done celestiaorg/devops#123