diff --git a/prombench/docs/eks.md b/prombench/docs/eks.md index f529f57ca..28afaa922 100644 --- a/prombench/docs/eks.md +++ b/prombench/docs/eks.md @@ -107,8 +107,38 @@ Run Prombench tests in [Elastic Kubernetes Service (EKS)](https://aws.amazon.com ```bash make node_create ``` +3. **Setting Up Benchmarking Data** + When setting up a benchmarking environment, it’s often useful to have pre-generated data available. This data can help speed up testing and make benchmarks more realistic by simulating actual workloads. -3. **Deploy the Kubernetes Objects**: +In this setup, you have two choices: + +Here’s how each option works: +- **Option 1: Download data from object storage** + + To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name `object-config.yml` with the necessary credentials as per your object storage. This secret enables access to the stored data. +> Note: Make sure this secret applied before `3b_prometheus-test_deployment.yaml` + +- **Option 2: Skip downloading data** + +If you don’t Want to Download data create an empty secret like this - + +```yaml +# Empty Secret to Skip Downloading Data +apiVersion: v1 +kind: Secret +metadata: + name: bucket-secret + namespace: prombench-{{ .PR_NUMBER }} +type: Opaque +stringData: + object-config.yml: +``` + +Regardless of the option chosen, data stored in Prometheus will only be retained based on the configured retention settings (```--storage.tsdb.retention.size```). + +> **⚠️ Warning:** The benchmark will change its basis when the retention size limit is reached and older downloaded blocks are deleted. Ensure that you have sufficient retention settings configured to avoid data loss that could affect benchmarking results. + +4. **Deploy the Kubernetes Objects**: ```bash make resource_apply diff --git a/prombench/docs/gke.md b/prombench/docs/gke.md index dbcb4b292..8c6c0e68d 100644 --- a/prombench/docs/gke.md +++ b/prombench/docs/gke.md @@ -96,8 +96,38 @@ Run Prombench tests in [Google Kubernetes Engine (GKE)](https://cloud.google.com ```bash make node_create ``` +3. **Setting Up Benchmarking Data** + When setting up a benchmarking environment, it’s often useful to have pre-generated data available.This data can help speed up testing and make benchmarks more realistic by simulating actual workloads. -3. **Deploy the Kubernetes Objects**: +In this setup, you have two choices: + +Here’s how each option works: +- **Option 1: Download data from object storage** + + To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name `object-config.yml` with the necessary credentials as per your object storage. This secret enables access to the stored data. +> Note: Make sure this secret applied before `3b_prometheus-test_deployment.yaml` + +- **Option 2: Skip downloading data** + +If you don’t Want to Download data create an empty secret like this - + +```yaml +# Empty Secret to Skip Downloading Data +apiVersion: v1 +kind: Secret +metadata: + name: bucket-secret + namespace: prombench-{{ .PR_NUMBER }} +type: Opaque +stringData: + object-config.yml: +``` + +Regardless of the option chosen, data stored in Prometheus will only be retained based on the configured retention settings (```--storage.tsdb.retention.size```). + +> **⚠️ Warning:** The benchmark will change its basis when the retention size limit is reached and older downloaded blocks are deleted. Ensure that you have sufficient retention settings configured to avoid data loss that could affect benchmarking results. + +4. **Deploy the Kubernetes Objects**: ```bash make resource_apply diff --git a/prombench/docs/kind.md b/prombench/docs/kind.md index 9b054c1b4..cc25376d9 100644 --- a/prombench/docs/kind.md +++ b/prombench/docs/kind.md @@ -114,7 +114,38 @@ If used with the GitHub integration: export PR_NUMBER= ``` -2. Deploy the Kubernetes objects: +3. **Setting Up Benchmarking Data** + When setting up a benchmarking environment, it’s often useful to have pre-generated data available.This data can help speed up testing and make benchmarks more realistic by simulating actual workloads. + +In this setup, you have two choices: + +Here’s how each option works: +- **Option 1: Download data from object storage** + + To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name `object-config.yml` with the necessary credentials as per your object storage. This secret enables access to the stored data. +> Note: Make sure this secret applied before `3b_prometheus-test_deployment.yaml` + +- **Option 2: Skip downloading data** + +If you don’t Want to Download data create an empty secret like this - + +```yaml +# Empty Secret to Skip Downloading Data +apiVersion: v1 +kind: Secret +metadata: + name: bucket-secret + namespace: prombench-{{ .PR_NUMBER }} +type: Opaque +stringData: + object-config.yml: +``` + +Regardless of the option chosen, data stored in Prometheus will only be retained based on the configured retention settings (```--storage.tsdb.retention.size```). + +> **⚠️ Warning:** The benchmark will change its basis when the retention size limit is reached and older downloaded blocks are deleted. Ensure that you have sufficient retention settings configured to avoid data loss that could affect benchmarking results. + +3. Deploy the Kubernetes objects: > **_Note:_** If you encounter a `too many files open` error caused by promtail, increase the default value of `/proc/sys/fs/inotify/max_user_instances` from 128 to 512: > ```bash > sudo sysctl fs.inotify.max_user_instances=512 diff --git a/prombench/manifests/prombench/benchmark/3b_prometheus-test_deployment.yaml b/prombench/manifests/prombench/benchmark/3b_prometheus-test_deployment.yaml index a64a001e3..7355b6f8f 100644 --- a/prombench/manifests/prombench/benchmark/3b_prometheus-test_deployment.yaml +++ b/prombench/manifests/prombench/benchmark/3b_prometheus-test_deployment.yaml @@ -45,9 +45,23 @@ spec: value: "{{ .GITHUB_ORG }}" - name: GITHUB_REPO value: "{{ .GITHUB_REPO }}" + - name: STORAGE + value: "/storage-paths" volumeMounts: - name: prometheus-executable mountPath: /prometheus-builder + - name: key + mountPath: /storage-paths + - name: data-downloader + image: docker.io/prominfra/block-sync:master + imagePullPolicy: Always + volumeMounts: + - name: instance-ssd + mountPath: /data + - name: bucket-config + mountPath: /config + - name: key + mountPath: /key containers: - name: prometheus image: quay.io/prometheus/busybox:latest @@ -67,7 +81,8 @@ spec: "--web.external-url=http://{{ .DOMAIN_NAME }}/{{ .PR_NUMBER }}/prometheus-pr", "--storage.tsdb.path=/prometheus", "--config.file=/etc/prometheus/prometheus.yml", - "--log.level=debug" + "--log.level=debug", + "--storage.tsdb.retention.size=5GB" ] volumeMounts: - name: config-volume @@ -88,6 +103,11 @@ spec: path: /mnt/disks/ssd0 #gke ssds - name: prometheus-executable emptyDir: {} + - name: bucket-config # Define the Secret volume + secret: + secretName: bucket-secret + - name: key + emptyDir: {} terminationGracePeriodSeconds: 300 nodeSelector: node-name: prometheus-{{ .PR_NUMBER }} @@ -144,6 +164,33 @@ spec: - prometheus securityContext: runAsUser: 0 + initContainers: + - name: download-key + image: docker.io/prominfra/prometheus-builder:master + imagePullPolicy: Always + command: [ "/go/src/github.com/key.sh" ] + env: + - name: PR_NUMBER + value: "{{ .PR_NUMBER }}" + - name: GITHUB_ORG + value: "{{ .GITHUB_ORG }}" + - name: GITHUB_REPO + value: "{{ .GITHUB_REPO }}" + - name: STORAGE + value: "/storage-paths" + volumeMounts: + - name: key + mountPath: /storage-paths + - name: data-downloader + image: docker.io/prominfra/block-sync:master + imagePullPolicy: Always + volumeMounts: + - name: instance-ssd + mountPath: /data + - name: bucket-config + mountPath: /config + - name: key + mountPath: /key containers: - name: prometheus image: quay.io/prometheus/prometheus:{{ .RELEASE }} @@ -153,7 +200,8 @@ spec: "--web.external-url=http://{{ .DOMAIN_NAME }}/{{ .PR_NUMBER }}/prometheus-release", "--storage.tsdb.path=/prometheus", "--config.file=/etc/prometheus/prometheus.yml", - "--log.level=debug" + "--log.level=debug", + "--storage.tsdb.retention.size=5GB" ] volumeMounts: - name: config-volume @@ -172,6 +220,11 @@ spec: # /mnt is where GKE keeps it's SSD # don't change this if you want Prometheus to take advantage of these local SSDs path: /mnt/disks/ssd0 + - name: bucket-config # Define the Secret volume + secret: + secretName: bucket-secret + - name: key + emptyDir: {} terminationGracePeriodSeconds: 300 nodeSelector: node-name: prometheus-{{ .PR_NUMBER }} diff --git a/tools/prometheus-builder/Dockerfile b/tools/prometheus-builder/Dockerfile index 96bb9576e..d353c5674 100644 --- a/tools/prometheus-builder/Dockerfile +++ b/tools/prometheus-builder/Dockerfile @@ -4,6 +4,8 @@ RUN mkdir -p /go/src/github.com COPY ./build.sh /go/src/github.com/build.sh -RUN chmod +x /go/src/github.com/build.sh +COPY ./key.sh /go/src/github.com/key.sh + +RUN chmod +x /go/src/github.com/* ENTRYPOINT ["/go/src/github.com/build.sh"] diff --git a/tools/prometheus-builder/build.sh b/tools/prometheus-builder/build.sh index a07328e62..d04233593 100755 --- a/tools/prometheus-builder/build.sh +++ b/tools/prometheus-builder/build.sh @@ -2,7 +2,7 @@ DIR="/go/src/github.com/prometheus/prometheus" -if [[ -z $PR_NUMBER || -z $VOLUME_DIR || -z $GITHUB_ORG || -z $GITHUB_REPO ]]; then +if [[ -z $PR_NUMBER || -z $VOLUME_DIR || -z $STORAGE || -z $GITHUB_ORG || -z $GITHUB_REPO ]]; then echo "ERROR:: environment variables not set correctly" exit 1; fi @@ -24,6 +24,16 @@ fi git checkout pr-branch +# Here, STORAGE is specified as an environment variable of the prometheus-builder init container, +# where it will copy the bucket-config.yml file from the Prometheus directory to the volume section of the +# emptyDir. This file will later be used by the data-downloader init container. +if [ -f "$DIR/bucket-config.yml" ]; then + echo "INFO:: bucket-config.yml file is Present on $DIR/bucket-config.yml directory so download the block from ObjecStorage." + cp "$DIR/bucket-config.yml" "$STORAGE/bucket-config.yml" +else + echo "INFO:: bucket-config.yml File does not exist on $DIR/bucket-config.yml directory so data is not downloaded from ObjectStorage." +fi + echo ">> Creating prometheus binaries" if ! make build PROMU_BINARIES="prometheus"; then echo "ERROR:: Building of binaries failed" diff --git a/tools/prometheus-builder/key.sh b/tools/prometheus-builder/key.sh new file mode 100755 index 000000000..bd405b119 --- /dev/null +++ b/tools/prometheus-builder/key.sh @@ -0,0 +1,35 @@ +#!/bin/bash + +DIR="/go/src/github.com/prometheus/prometheus" + +if [[ -z $PR_NUMBER || -z $STORAGE || -z $GITHUB_ORG || -z $GITHUB_REPO ]]; then + echo "ERROR:: environment variables not set correctly" + exit 1; +fi + +# Clone the repository with a shallow clone +echo ">> Cloning repository $GITHUB_ORG/$GITHUB_REPO (shallow clone)" +if ! git clone --depth 1 https://github.com/$GITHUB_ORG/$GITHUB_REPO.git $DIR; then + echo "ERROR:: Cloning of repo $GITHUB_ORG/$GITHUB_REPO failed" + exit 1; +fi + +cd $DIR || exit 1 + +echo ">> Fetching Pull Request $GITHUB_ORG/$GITHUB_REPO/pull/$PR_NUMBER" +if ! git fetch origin pull/$PR_NUMBER/head:pr-branch; then + echo "ERROR:: Fetching of PR $PR_NUMBER failed" + exit 1; +fi + +git checkout pr-branch + +# Here, STORAGE is specified as an environment variable of the download-key init container, +# where it will copy the key.yml file from the Prometheus directory to the volume section of the +# emptyDir. This file will later be used by the data-downloader init container. +if [ -f "$DIR/bucket-config.yml" ]; then + echo "INFO:: bucket-config.yml file is Present on $DIR/bucket-config.yml directory so download the block from ObjecStorage." + cp "$DIR/bucket-config.yml" "$STORAGE/bucket-config.yml" +else + echo "INFO:: bucket-config.yml File does not exist on $DIR/bucket-config.yml directory so data is not downloaded from ObjectStorage." +fi