DevSecOps : Adopt/Deploy Load-Testing Framework

[emvaldes]

Objective:

Implement a robust load testing framework to simulate production-level traffic, measure system performance under various loads, and identify bottlenecks. The framework will support testing batch processing, APIs, and overall system scalability.

---

Deliverables

1. Fully functional load testing framework integrated into the CI/CD pipeline.
2. Reusable and parameterized test scripts for API and batch processing.
3. Monitoring integration for tracking load test results in real time.
4. Documentation of best practices for load testing.

---

Integration With Existing Sections

• This section integrates closely with:
  1. Reproducing the Production Environment: Ensures the staging environment supports realistic load testing.
  2. Batch Size and Concurrency Testing: Provides scripts and automation for testing these scenarios.
  3. Chaos Testing: Enhances the framework to support failure simulations during load tests.

[emvaldes]

Select the Appropriate Load Testing Framework

Goal: Identify and adopt a load testing framework that aligns with the system requirements and DevSecOps practices.

---

Tasks:

1. Evaluate Available Tools

   • Sub-Tasks:
     1. Research open-source tools like K6, Locust, Apache JMeter, and Taurus.
     2. Evaluate cloud-native options like Azure Load Testing and third-party tools like BlazeMeter.
     3. Compare features like scalability, scripting capabilities, integrations, and monitoring support.

2. Define Selection Criteria

   • Sub-Tasks:
     1. Identify key requirements for the framework:
        • Support for API and batch testing.
        • Scalability to simulate large loads.
        • Integration with GitHub Actions and CI/CD pipelines.
     2. Document pros and cons of shortlisted tools.

3. Select the Framework

   • Sub-Tasks:
     1. Obtain stakeholder approval for the selected framework.
     2. Document the decision and reasoning.

[emvaldes]

Set Up the Load Testing Framework

Goal: Install and configure the chosen framework for the test environment.

---

Tasks:

1. Framework Installation

   • Sub-Tasks:
     1. Install the framework in a staging environment (e.g., install K6 or JMeter on a dedicated VM/container).
     2. Verify installation with a simple test script.

2. Create Testing Infrastructure

   • Sub-Tasks:
     1. Deploy infrastructure (e.g., VMs, AKS clusters) for running load tests at scale.
     2. Ensure monitoring tools (Azure Monitor, Application Insights) are integrated with the test environment.

3. Configure Authentication

   • Sub-Tasks:
     1. Set up access tokens or API keys for authenticating test scripts.
     2. Store secrets securely using Azure Key Vault or GitHub Secrets.

[emvaldes]

Develop Load Testing Scripts

Goal: Write reusable and parameterized scripts for simulating production-like workloads.

---

Tasks:

1. Write Basic Scripts

   • Sub-Tasks:
     1. Create API load testing scripts (e.g., GET, POST requests).
     2. Write batch processing scripts to test varying batch sizes and concurrency levels.

2. Incorporate Realistic Scenarios

   • Sub-Tasks:
     1. Simulate normal, peak, and failure scenarios.
     2. Randomize request patterns and payloads to mimic real-world behavior.

3. Parameterize Test Scripts

   • Sub-Tasks:
     1. Allow batch sizes, concurrency levels, and payloads to be configured dynamically.
     2. Use environment variables or configuration files for parameterization.

4. Integrate Monitoring Hooks

   • Sub-Tasks:
     1. Add logging and custom metrics collection (e.g., batch latency, error rates).
     2. Send metrics to Azure Monitor or Log Analytics.

[emvaldes]

Automate Load Testing in CI/CD

Goal: Integrate the load testing framework into CI/CD pipelines for continuous validation.

---

Tasks:

1. Set Up GitHub Actions Workflow

   • Sub-Tasks:
     1. Create a GitHub Actions workflow to trigger load tests on every deployment.
     2. Use parameters to configure load levels (e.g., VUs, batch sizes).

2. Define Success Criteria

   • Sub-Tasks:
     1. Set thresholds for test results (e.g., max latency, min throughput).
     2. Define conditions for passing or failing the pipeline.

3. Report Results

   • Sub-Tasks:
     1. Upload test results as artifacts in GitHub Actions.
     2. Visualize results in dashboards or reports (e.g., Power BI, Azure Monitor).

[emvaldes]

Validate and Optimize the Framework

Goal: Test the framework's effectiveness and refine it for long-term use.

---

Tasks:

1. Run Initial Tests

   • Sub-Tasks:
     1. Execute basic load tests to validate the framework setup.
     2. Monitor system behavior and test results for correctness.

2. Optimize Framework Configuration

   • Sub-Tasks:
     1. Tune framework settings (e.g., request rate, logging levels).
     2. Resolve issues like script failures, resource exhaustion.

3. Document Best Practices

   • Sub-Tasks:
     1. Write guidelines for using the framework (e.g., scripting, running tests).
     2. Share learnings with the team.

[emvaldes]

Detailed GitHub Actions Workflow for Load Testing

Below is a GitHub Actions workflow to automate load testing with K6. It can be customized for other tools like Apache JMeter or Locust.

Workflow Example: Load Testing with K6

name: Load Testing Workflow

on:
  push:
    branches:
      - main

jobs:
  load-test:
    runs-on: ubuntu-latest

    steps:
      # Step 1: Checkout Code
      - name: Checkout code
        uses: actions/checkout@v3

      # Step 2: Install K6
      - name: Install K6
        run: sudo apt-get update && sudo apt-get install -y k6

      # Step 3: Run K6 Load Test
      - name: Run K6 Load Test
        run: |
          k6 run --vus 100 --duration 5m tests/load_test.js

      # Step 4: Upload Test Results
      - name: Upload Test Results
        uses: actions/upload-artifact@v3
        with:
          name: k6-results
          path: ./k6-results.json

      # Step 5: Generate Load Test Report (Optional)
      - name: Generate HTML Report
        run: k6 run --out json=k6-results.json tests/load_test.js && k6 report --out=html=k6-report.html

      # Step 6: Upload HTML Report
      - name: Upload HTML Report
        uses: actions/upload-artifact@v3
        with:
          name: k6-html-report
          path: ./k6-report.html

Key Points:

1. Replace tests/load_test.js with your actual K6 test script.
2. Adjust --vus (virtual users) and --duration for your specific load scenarios.
3. Artifacts like k6-results.json and k6-report.html can be downloaded for further analysis.

[emvaldes]

Sample Load Testing Scripts

A. API Load Testing Script (K6)

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
  stages: [
    { duration: '1m', target: 10 },  // Ramp up to 10 users
    { duration: '3m', target: 50 },  // Maintain 50 users
    { duration: '1m', target: 0 },   // Ramp down to 0 users
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'], // 95% of requests must complete below 500ms
    http_req_failed: ['rate<0.01'],   // Error rate should be less than 1%
  },
};

export default function () {
  http.get('https://your-api-endpoint.com/data');
  sleep(1);
}

[emvaldes]

B. Batch Processing Load Testing Script (K6)

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
  stages: [
    { duration: '2m', target: 20 },  // Ramp up to 20 users
    { duration: '5m', target: 100 }, // Maintain 100 users
    { duration: '2m', target: 0 },   // Ramp down to 0 users
  ],
};

export default function () {
  const payload = JSON.stringify({
    batchId: `batch-${__VU}-${__ITER}`, // Unique batch ID
    data: Array.from({ length: 2500 }, (_, i) => i + 1), // Simulate 2.5k batch size
  });

  const params = {
    headers: {
      'Content-Type': 'application/json',
    },
  };

  http.post('https://your-api-endpoint.com/process-batch', payload, params);
  sleep(1);
}

[emvaldes]

C. Concurrency Load Testing Script (K6)

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
  vus: 50,  // Simulate 50 virtual users concurrently
  duration: '10m', // Run test for 10 minutes
};

export default function () {
  const payload = JSON.stringify({
    batchId: `batch-${__VU}-${__ITER}`, // Unique batch ID
    data: Array.from({ length: 1000 }, (_, i) => i + 1), // Simulate 1k batch size
  });

  const params = {
    headers: {
      'Content-Type': 'application/json',
    },
  };

  http.post('https://your-api-endpoint.com/process-batch', payload, params);
  sleep(Math.random() * 2); // Random wait between requests
}

[emvaldes]

Monitoring Integration with Azure Monitor

A. Monitor Load Testing Metrics

Use Azure Monitor and Application Insights to collect performance metrics during load tests.

Metrics to Monitor:

1. Application Metrics:
   • API response time (latency).
   • Error rates for API calls or batch processing.
   • Throughput (requests per second or batches per second).
2. Infrastructure Metrics:
   • CPU and memory utilization on VMs or containers.
   • Disk IOPS (Input/Output Operations Per Second).
   • Network bandwidth usage.

[emvaldes]

B. KQL Queries for Azure Monitor

1. API Latency and Success Rate

   requests
   | summarize AvgLatency = avg(duration), SuccessRate = countif(success == true) * 100.0 / count() by bin(timestamp, 1m)
   | order by timestamp asc

2. Throughput (Requests Per Second)

   requests
   | summarize Throughput = count() by bin(timestamp, 1s)
   | order by timestamp asc

3. Resource Utilization

   Perf
   | where CounterName in ("% Processor Time", "Available MBytes")
   | summarize AvgCPU = avg(CounterValue) by bin(TimeGenerated, 1m)

[emvaldes]

Visualizing Results with Azure Workbooks

Dashboard Ideas

1. Latency Trends:
   • Use a line chart to show latency over time (e.g., average and 95th percentile).
2. Error Rates:
   • Use a bar chart to display errors grouped by endpoint or batch size.
3. Throughput:
   • Use a line chart to show requests or batches processed per second.

---

Creating Azure Workbook

1. Go to Azure Monitor > Workbooks.
2. Create a new workbook and add the queries for API performance, throughput, and resource utilization.
3. Use visualizations (line, bar, pie charts) to make results actionable.

[emvaldes]

Templates for Azure Workbook Dashboards

Below are detailed dashboard ideas and queries you can use in Azure Workbooks to visualize load testing results.

A. Latency and Throughput Trends

1. Latency Trends:

   • Use a line chart to plot average and 95th percentile latency over time.
   • Query:

     requests
     | summarize AvgLatency = avg(duration), P95Latency = percentile(duration, 95) by bin(timestamp, 1m)

   • Visualization Example:
     • X-Axis: Time (1-minute intervals).
     • Y-Axis: Latency (ms).

2. Throughput Trends:

   • Use a line chart to show the number of requests processed per second.
   • Query:

     requests
     | summarize Throughput = count() by bin(timestamp, 1s)

   • Visualization Example:
     • X-Axis: Time (1-second intervals).
     • Y-Axis: Requests per Second.

[emvaldes]

B. Error Rate Analysis

1. Error Count by Endpoint:

   • Use a bar chart to display errors grouped by endpoint.
   • Query:

     requests
     | where success == false
     | summarize ErrorCount = count() by name
     | order by ErrorCount desc

2. Error Trends Over Time:

   • Use a line chart to plot error rates over time.
   • Query:

     requests
     | summarize ErrorRate = countif(success == false) * 100.0 / count() by bin(timestamp, 1m)

[emvaldes]

C. Resource Utilization Dashboard

1. CPU and Memory Usage Trends:

   • Use a line chart to track resource usage.
   • Query:

     Perf
     | where ObjectName in ("Processor", "Memory")
     | summarize AvgCPU = avg(CounterValue), AvgMemory = avg(CounterValue) by bin(TimeGenerated, 1m), CounterName

2. Heatmap of Resource Utilization:

   • Use a heatmap to visualize resource usage (CPU, memory, IOPS) under different load scenarios.

[emvaldes]

D. Workbook Setup

1. Navigate to Azure Monitor > Workbooks.
2. Create a new workbook and add visualizations using the queries above.
3. Organize sections into tabs:
   • Latency and Throughput.
   • Errors and Failures.
   • Resource Utilization.

[emvaldes]

Complex Load Testing Scripts

A. Simulating Variable Traffic Patterns

This script simulates a mix of GET and POST requests with randomized traffic patterns.

import http from 'k6/http';
import { sleep, check } from 'k6';

export let options = {
  scenarios: {
    steady_traffic: {
      executor: 'constant-vus',
      vus: 50,
      duration: '10m',
    },
    spike_traffic: {
      executor: 'ramping-vus',
      startVUs: 10,
      stages: [
        { duration: '1m', target: 100 }, // Ramp up to 100 users
        { duration: '5m', target: 100 }, // Hold at 100 users
        { duration: '2m', target: 0 },   // Ramp down
      ],
    },
  },
  thresholds: {
    http_req_duration: ['p(95)<400'], // 95% of requests must complete below 400ms
  },
};

export default function () {
  const urls = ['https://your-api-endpoint.com/data', 'https://your-api-endpoint.com/process'];
  const payload = JSON.stringify({ key: 'value' });

  const res = Math.random() > 0.5
    ? http.get(urls[0])
    : http.post(urls[1], payload, { headers: { 'Content-Type': 'application/json' } });

  check(res, {
    'status was 200': (r) => r.status === 200,
  });

  sleep(Math.random() * 2);
}

[emvaldes]

B. Stress Testing with Large Batch Sizes

This script pushes the system to handle increasingly large batch sizes.

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
  stages: [
    { duration: '2m', target: 50 },  // Ramp up to 50 users
    { duration: '5m', target: 200 }, // Maintain 200 users
    { duration: '1m', target: 0 },   // Ramp down
  ],
};

export default function () {
  const payload = JSON.stringify({
    batchId: `batch-${__VU}-${__ITER}`,
    data: Array.from({ length: __ITER * 100 }, (_, i) => i + 1),
  });

  const params = {
    headers: { 'Content-Type': 'application/json' },
  };

  const res = http.post('https://your-api-endpoint.com/process-batch', payload, params);
  sleep(1);
}

[emvaldes]

Automating Test Result Reporting

A. GitHub Actions Integration

Add a reporting step in your GitHub Actions workflow to upload results.

# Workflow snippet for reporting
- name: Generate HTML Report
  run: |
    k6 run --out json=k6-results.json tests/load_test.js
    k6 report --out=html=k6-report.html
- name: Upload Report
  uses: actions/upload-artifact@v3
  with:
    name: k6-html-report
    path: ./k6-report.html

[emvaldes]

B. Post-Test Azure Monitor Integration

Send results from test scripts to Azure Monitor for centralized visualization.

• Add a hook in your test script to send metrics:

  export default function () {
    // Post batch processing example
    let res = http.post('https://your-api-endpoint.com/process-batch', payload);
  
    // Log custom metrics to Azure Monitor (pseudo code)
    logToAzureMonitor({
      latency: res.timings.duration,
      statusCode: res.status,
      batchSize: payload.data.length,
    });
  
    sleep(1);
  }

[emvaldes]

C. Visualization Using Power BI

1. Export test results (e.g., JSON or CSV) from load testing tools.
2. Import the data into Power BI to create custom dashboards for stakeholders.