Integrating block-sync in an init-container

.circleci/config.yml

@@ -110,6 +110,14 @@ jobs:
           organization: "$DOCKER_ORG"
           login_variable: DOCKER_LOGIN
           password_variable: DOCKER_PASSWORD
+      - prometheus/publish_images:
+          container_image_name: block-sync
+          dockerfile_path: "tools/block-sync/Dockerfile"
+          dockerbuild_context: "tools/block-sync/"
+          registry: docker.io
+          organization: "$DOCKER_ORG"
+          login_variable: DOCKER_LOGIN
+          password_variable: DOCKER_PASSWORD   
 workflows:
   version: 2
   test-infra:

prombench/docs/eks.md

@@ -107,8 +107,37 @@ Run Prombench tests in [Elastic Kubernetes Service (EKS)](https://aws.amazon.com
     ```bash
     make node_create
     ```
+3. **Setting Up Benchmarking Data**    
+ When setting up a benchmarking environment, it’s often useful to have pre-generated data available.This data can help speed up testing and make benchmarks more realistic by simulating actual workloads.
 
-3. **Deploy the Kubernetes Objects**:
+In this setup, you have two choices:
+
+Here’s how each option works:
+- **Option 1: Download data from object storage**
+
+   To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name ```object-config.yml```  with the necessary credentials as per your object storage. This secret enables access to the stored data.
+> Note: Make sure this secret applied before ```3b_prometheus-test_deployment.yaml```
+
+- **Option 2: Skip downloading data**
+
+If you don’t Want to Download data create an empty secret like this -
+
+```yaml
+# Empty Secret to Skip Downloading Data
+apiVersion: v1
+kind: Secret
+metadata:
+  name: bucket-secret
+  namespace: prombench-{{ .PR_NUMBER }} 
+type: Opaque
+stringData:
+  object-config.yml: 
+
+Regardless of the option chosen, data stored in Prometheus will only be retained based on the configured retention settings (`--storage.tsdb.retention.size`). 
+
+> **⚠️ Warning:** The benchmark will change its basis when the retention size limit is reached and older downloaded blocks are deleted. Ensure that you have sufficient retention settings configured to avoid data loss that could affect benchmarking results. 
+
+4. **Deploy the Kubernetes Objects**:
 
     ```bash
     make resource_apply

prombench/docs/gke.md

@@ -96,8 +96,38 @@ Run Prombench tests in [Google Kubernetes Engine (GKE)](https://cloud.google.com
     ```bash
     make node_create
     ```
+3. **Setting Up Benchmarking Data**  
 
-3. **Deploy the Kubernetes Objects**:
+When setting up a benchmarking environment, it’s often useful to have pre-generated data available.This data can help speed up testing and make benchmarks more realistic by simulating actual workloads.
+
+In this setup, you have two choices:
+
+Here’s how each option works:
+- **Option 1: Download data from object storage**
+
+   To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name ```object-config.yml```  with the necessary credentials as per your object storage. This secret enables access to the stored data.
+> Note: Make sure this secret applied before ```3b_prometheus-test_deployment.yaml```
+
+- **Option 2: Skip downloading data**
+
+If you don’t Want to Download data create an empty secret like this -
+
+```yaml
+# Empty Secret to Skip Downloading Data
+apiVersion: v1
+kind: Secret
+metadata:
+  name: bucket-secret
+  namespace: prombench-{{ .PR_NUMBER }} 
+type: Opaque
+stringData:
+  object-config.yml: 
+
+Regardless of the option chosen, data stored in Prometheus will only be retained based on the configured retention settings (`--storage.tsdb.retention.size`). 
+
+> **⚠️ Warning:** The benchmark will change its basis when the retention size limit is reached and older downloaded blocks are deleted. Ensure that you have sufficient retention settings configured to avoid data loss that could affect benchmarking results. 
+
+4. **Deploy the Kubernetes Objects**:
 
     ```bash
     make resource_apply

prombench/docs/kind.md

@@ -114,7 +114,38 @@ If used with the GitHub integration:
    export PR_NUMBER=<PR to benchmark against the selected $RELEASE>
    ```
 
-2. Deploy the Kubernetes objects:
+2. ### Setting Up Benchmarking Data
+
+When setting up a benchmarking environment, it’s often useful to have pre-generated data available.This data can help speed up testing and make benchmarks more realistic by simulating actual workloads.
+
+In this setup, you have two choices:
+
+Here’s how each option works:
+- **Option 1: Download data from object storage**
+
+   To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name ```object-config.yml```  with the necessary credentials as per your object storage. This secret enables access to the stored data.
+> Note: Make sure this secret applied before ```3b_prometheus-test_deployment.yaml```
+
+- **Option 2: Skip downloading data**
+
+If you don’t Want to Download data create an empty secret like this -
+
+```yaml
+# Empty Secret to Skip Downloading Data
+apiVersion: v1
+kind: Secret
+metadata:
+  name: bucket-secret
+  namespace: prombench-{{ .PR_NUMBER }} 
+type: Opaque
+stringData:
+  object-config.yml: 
+
+Regardless of the option chosen, data stored in Prometheus will only be retained based on the configured retention settings (`--storage.tsdb.retention.size`). 
+
+> **⚠️ Warning:** The benchmark will change its basis when the retention size limit is reached and older downloaded blocks are deleted. Ensure that you have sufficient retention settings configured to avoid data loss that could affect benchmarking results. 
+
+3. Deploy the Kubernetes objects:
    > **_Note:_** If you encounter a `too many files open` error caused by promtail, increase the default value of `/proc/sys/fs/inotify/max_user_instances` from 128 to 512:
    > ```bash
    > sudo sysctl fs.inotify.max_user_instances=512

prombench/manifests/prombench/benchmark/3b_prometheus-test_deployment.yaml

@@ -45,9 +45,23 @@ spec:
           value: "{{ .GITHUB_ORG }}"
         - name: GITHUB_REPO
           value: "{{ .GITHUB_REPO }}"
+        - name: STORAGE
+          value: "/storage-paths"  
         volumeMounts:
         - name: prometheus-executable
           mountPath: /prometheus-builder
+        - name: key
+          mountPath: /storage-paths     
+      - name: data-downloader
+        image: docker.io/prominfra/block-sync:master
+        imagePullPolicy: Always                
+        volumeMounts:
+        - name: instance-ssd
+          mountPath: /data
+        - name: bucket-config
+          mountPath: /config
+        - name: key
+          mountPath: /key
       containers:
       - name: prometheus
         image: quay.io/prometheus/busybox:latest
@@ -67,7 +81,8 @@ spec:
           "--web.external-url=http://{{ .DOMAIN_NAME }}/{{ .PR_NUMBER }}/prometheus-pr",
           "--storage.tsdb.path=/prometheus",
           "--config.file=/etc/prometheus/prometheus.yml",
-          "--log.level=debug"
+          "--log.level=debug",
+          "--storage.tsdb.retention.size=5GB"
         ]
         volumeMounts:
         - name: config-volume
@@ -88,6 +103,11 @@ spec:
           path: /mnt/disks/ssd0 #gke ssds
       - name: prometheus-executable
         emptyDir: {}
+      - name: bucket-config   # Define the Secret volume
+        secret:
+          secretName: bucket-secret
+      - name: key
+        emptyDir: {}                  
       terminationGracePeriodSeconds: 300
       nodeSelector:
         node-name: prometheus-{{ .PR_NUMBER }}
@@ -144,6 +164,33 @@ spec:
                 - prometheus
       securityContext:
         runAsUser: 0
+      initContainers:
+      - name: download-key
+        image: docker.io/prominfra/prometheus-builder:master
+        imagePullPolicy: Always
+        command: [ "/go/src/github.com/key.sh" ]
+        env:
+        - name: PR_NUMBER
+          value: "{{ .PR_NUMBER }}"
+        - name: GITHUB_ORG
+          value: "{{ .GITHUB_ORG }}"
+        - name: GITHUB_REPO
+          value: "{{ .GITHUB_REPO }}"          
+        - name: STORAGE
+          value: "/storage-paths"
+        volumeMounts:
+        - name: key
+          mountPath: /storage-paths
+      - name: data-downloader
+        image: docker.io/prominfra/block-sync:master
+        imagePullPolicy: Always                  
+        volumeMounts:
+        - name: instance-ssd
+          mountPath: /data
+        - name: bucket-config
+          mountPath: /config
+        - name: key
+          mountPath: /key            
       containers:
       - name: prometheus
         image: quay.io/prometheus/prometheus:{{ .RELEASE }}
@@ -153,7 +200,8 @@ spec:
           "--web.external-url=http://{{ .DOMAIN_NAME }}/{{ .PR_NUMBER }}/prometheus-release",
           "--storage.tsdb.path=/prometheus",
           "--config.file=/etc/prometheus/prometheus.yml",
-          "--log.level=debug"
+          "--log.level=debug",
+          "--storage.tsdb.retention.size=5GB"
         ]
         volumeMounts:
         - name: config-volume
@@ -172,6 +220,11 @@ spec:
           # /mnt is where GKE keeps it's SSD
           # don't change this if you want Prometheus to take advantage of these local SSDs
           path: /mnt/disks/ssd0
+      - name: bucket-config   # Define the Secret volume
+        secret:
+          secretName: bucket-secret
+      - name: key
+        emptyDir: {}                              
       terminationGracePeriodSeconds: 300
       nodeSelector:
         node-name: prometheus-{{ .PR_NUMBER }}

tools/block-sync/upload_download.go

@@ -38,6 +38,10 @@ func newStore(tsdbPath, objectConfig, objectKey string, logger *slog.Logger) (*S
 	if err != nil {
 		return nil, fmt.Errorf("failed to read config file: %w", err)
 	}
+	if len(configBytes) == 0 {
+		fmt.Println("Config file is empty, exiting container.")
+		os.Exit(0)
+	}
 
 	bucket, err := client.NewBucket(log.NewNopLogger(), configBytes, "block-sync")
 	if err != nil {

tools/prometheus-builder/Dockerfile

@@ -4,6 +4,8 @@ RUN mkdir -p /go/src/github.com
 
 COPY ./build.sh /go/src/github.com/build.sh
 
-RUN chmod +x /go/src/github.com/build.sh
+COPY ./key.sh /go/src/github.com/key.sh
+
+RUN chmod +x /go/src/github.com/*
 
 ENTRYPOINT ["/go/src/github.com/build.sh"]

tools/prometheus-builder/build.sh

@@ -2,7 +2,7 @@
 
 DIR="/go/src/github.com/prometheus/prometheus"
 
-if [[ -z $PR_NUMBER || -z $VOLUME_DIR || -z $GITHUB_ORG || -z $GITHUB_REPO ]]; then
+if [[ -z $PR_NUMBER || -z $VOLUME_DIR || -z $STORAGE || -z $GITHUB_ORG || -z $GITHUB_REPO ]]; then
     echo "ERROR:: environment variables not set correctly"
     exit 1;
 fi
@@ -24,6 +24,16 @@ fi
 
 git checkout pr-branch
 
+# Here, STORAGE is specified as an  environment variable of the prometheus-builder init container, 
+# where it will copy the key.yml file from the Prometheus directory to the volume section of the
+# emptyDir. This file will later be used by the data-downloader init container.
+if [ -f "$DIR/key.yml" ]; then
+    echo "INFO:: key.yml file is Present on $DIR/key.yml directory."
+    cp  "$DIR/key.yml" "$STORAGE/key.yml"
+else
+    echo "INFO:: key.yml File does not exist on $DIR/key.yml directory."
+fi
+
 echo ">> Creating prometheus binaries"
 if ! make build PROMU_BINARIES="prometheus"; then
     echo "ERROR:: Building of binaries failed"

tools/prometheus-builder/key.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+DIR="/go/src/github.com/prometheus/prometheus"
+
+if [[ -z $PR_NUMBER || -z $STORAGE  || -z $GITHUB_ORG || -z $GITHUB_REPO ]]; then
+    echo "ERROR:: environment variables not set correctly"
+    exit 1;
+fi
+
+# Clone the repository with a shallow clone
+echo ">> Cloning repository $GITHUB_ORG/$GITHUB_REPO (shallow clone)"
+if ! git clone --depth 1 https://github.com/$GITHUB_ORG/$GITHUB_REPO.git $DIR; then
+    echo "ERROR:: Cloning of repo $GITHUB_ORG/$GITHUB_REPO failed"
+    exit 1;
+fi
+
+cd $DIR || exit 1
+
+echo ">> Fetching Pull Request $GITHUB_ORG/$GITHUB_REPO/pull/$PR_NUMBER"
+if ! git fetch origin pull/$PR_NUMBER/head:pr-branch; then
+    echo "ERROR:: Fetching of PR $PR_NUMBER failed"
+    exit 1;
+fi
+
+git checkout pr-branch
+
+# Here, STORAGE is specified as an  environment variable of the download-key init container, 
+# where it will copy the key.yml file from the Prometheus directory to the volume section of the
+# emptyDir. This file will later be used by the data-downloader init container.
+if [ -f "$DIR/key.yml" ]; then
+    echo "INFO:: key.yml file is Present on $DIR/key.yml directory."
+    cp  "$DIR/key.yml" "$STORAGE/key.yml"
+else
+    echo "INFO:: key.yml File does not exist on $DIR/key.yml directory."
+fi