Skip to content

Latest commit

 

History

History
295 lines (218 loc) · 9.58 KB

GKE.md

File metadata and controls

295 lines (218 loc) · 9.58 KB
Raw

Distributed load testing with GKE

NOTE The below instructions to for multi instance load operations deploy as a kubernetes deployment. This means that kubernetes will continuously restart your load or run instances until you manually stop them. It is important that you do not leave them running indefinitely.

Setup Environment

export PROJECT_ID=spanner-test
export SPANNER_INSTANCE=test-instance
export SPANNER_DATABASE=test-database
export GKE_CLUSTER_NAME=test-cluster
export GCP_REGION=us-west2

Create a GCP Project

gcloud projects create $PROJECT_ID

Enable services

gcloud services enable spanner.googleapis.com --project $PROJECT_ID
gcloud services enable cloudbuild.googleapis.com --project $PROJECT_ID
gcloud services enable container.googleapis.com --project $PROJECT_ID
gcloud services enable artifactregistry.googleapis.com --project $PROJECT_ID

Create a service account

gcloud iam service-accounts create gcsb-test-sa \
    --description="GCSB Test Account" \
    --display-name="gcsb" \
    --project $PROJECT_ID

Grant access to spanner

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:gcsb-test-sa@${PROJECT_ID}.iam.gserviceaccount.com" \
    --role="roles/spanner.databaseUser"

Create key for service account

gcloud iam service-accounts keys create key.json --iam-account=gcsb-test-sa@${PROJECT_ID}.iam.gserviceaccount.com

Setup Spanner Database

Create a Spanner Instance

gcloud alpha spanner instances create $SPANNER_INSTANCE --config=regional-us-east1 --processing-units=1000 --project $PROJECT_ID

Create a database

gcloud spanner databases create $SPANNER_DATABASE --instance=$SPANNER_INSTANCE --project $PROJECT_ID

Setup GKE

Create GKE Cluster

Please see gcloud compute machine-types list for a list of machine types

gcloud container clusters create $GKE_CLUSTER_NAME \
  --project $PROJECT_ID \
  --region $GCP_REGION \
  --machine-type=n1-standard-8 \
  --num-nodes 3

Import Service Account Key

kubectl create secret generic gcsb-sa-key --from-file=key.json=./key.json

Build Docker Container

gcloud builds submit --tag gcr.io/$PROJECT_ID/gcsb .

Run the tool

CLI Only

To run inside Kubernetes, you will need to mount the ServiceAccount key secret into the container. Additionally, you will need to set the environment variable GOOGLE_APPLICATION_CREDENTIALS pointing to this key file. Below, there are instructions for single container launches versus parallel launches.

Single instance load operation

If you want to simply run a single instance of gcsb to perform a load operation, you can launch a POD via the kubectl run command.

kubectl run gcsb-load \
  --image=gcr.io/$PROJECT_ID/gcsb \
  --replicas=1 \
  --restart=Never \
  --overrides='{
     "apiVersion": "v1",
     "spec": {
        "containers": [{
          "name": "gcsb-load",
          "image": "gcr.io/'$PROJECT_ID'/gcsb",
          "command": [ "/gcsb", "load" ],
          "args": [ "--project='$PROJECT_ID'", "--instance='$SPANNER_INSTANCE'", "--database='$SPANNER_DATABASE'", "--table=SingleSingers", "--operations=1000000"],
          "volumeMounts": [{"mountPath": "/var/secrets/google", "name": "google-cloud-key"}],
          "env": [{ "name": "GOOGLE_APPLICATION_CREDENTIALS", "value": "/var/secrets/google/key.json" }]
        }],
        "volumes": [ { "name": "google-cloud-key", "secret": { "secretName": "gcsb-sa-key" } } ]
     }
  }'

Multi instance load operation

To create a load operation named 'gcsb-load', you must edit the gke_load.yaml file, supplying your spanner information.

For example, everywhere you se the comment EDIT: you must specify your information.

          - --project=YOUR_PROJECT_ID   # EDIT: Your GCP Project ID
          - --instance=YOUR_INSTANCE_ID # EDIT: Your Spanner Instance ID
          - --database=YOUR_DATABASE    # EDIT: Your Spanner Database Name
          - --table=YOUR_TABLE          # EDIT: Your Table Name
          - --operations=1000000        # EDIT: Number of Operations
          - --threads=10                # EDIT: Number of Threads

Once you have completed the necessary file edits, run the following.

kubectl apply -f docs/gke_load.yaml

to stop the test

kubectl delete deploy gcsb-load

Single instance run operation

If you want to simply run a single instance of gcsb to perform a run operation, you can launch a POD via the kubectl run command.

kubectl run gcsb-load \
  --image=gcr.io/$PROJECT_ID/gcsb \
  --replicas=1 \
  --restart=Never \
  --overrides='{
     "apiVersion": "v1",
     "spec": {
        "containers": [{
          "name": "gcsb-load",
          "image": "gcr.io/'$PROJECT_ID'/gcsb",
          "command": [ "/gcsb", "run" ],
          "args": [ "--project='$PROJECT_ID'", "--instance='$SPANNER_INSTANCE'", "--database='$SPANNER_DATABASE'", "--table=SingleSingers", "--operations=1000000"],
          "volumeMounts": [{"mountPath": "/var/secrets/google", "name": "google-cloud-key"}],
          "env": [{ "name": "GOOGLE_APPLICATION_CREDENTIALS", "value": "/var/secrets/google/key.json" }]
        }],
        "volumes": [ { "name": "google-cloud-key", "secret": { "secretName": "gcsb-sa-key" } } ]
     }
  }'

Multi instance run operation

To create a load operation named 'gcsb-run', you must edit the gke_run.yaml file, supplying your spanner information.

For example, everywhere you se the comment EDIT: you must specify your information.

          - --project=YOUR_PROJECT_ID   # EDIT: Your GCP Project ID
          - --instance=YOUR_INSTANCE_ID # EDIT: Your Spanner Instance ID
          - --database=YOUR_DATABASE    # EDIT: Your Spanner Database Name
          - --table=YOUR_TABLE          # EDIT: Your Table Name
          - --operations=1000000        # EDIT: Number of Operations
          - --threads=10                # EDIT: Number of Threads
          - --reads=50                  # EDIT: Read Weight (Example: 50 = 50% reads)
          - --writes=50                 # EDIT: Write Weight (Example: 50 = 50% writes)
          - --sample-size=5             # EDIT: Percentage of table to sample for generating reads (Example: 5 = 5% of the rows in the table)

Once you have completed the necessary file edits, run the following.

kubectl apply -f docs/gke_run.yaml

to stop the test

kubectl delete deploy gcsb-run

Custom Configuration

If you prefer to mount your custom gcsb file into the container, you should follow these instructions

Create Configmap

The below example assumes your conig file is named gcsb.yaml

kubectl create configmap gcsb-config --from-file=gcsb.yaml

Multi instance load operation

To create a load operation named 'gcsb-load', you must edit the gke_load_custom.yaml file, supplying your spanner information.

For example, everywhere you se the comment EDIT: you must specify your information.

          - --table=YOUR_TABLE              # EDIT: Your Table Name

Once you have completed the necessary file edits, run the following.

kubectl apply -f docs/gke_load_custom.yaml

to stop the test

kubectl delete deploy gcsb-load

Multi instance run operation

To create a load operation named 'gcsb-run', you must edit the gke_run_custom.yaml file, supplying your spanner information.

For example, everywhere you se the comment EDIT: you must specify your information.

          - --table=YOUR_TABLE              # EDIT: Your Table Name

Once you have completed the necessary file edits, run the following.

kubectl apply -f docs/gke_run_custom.yaml

to stop the test

kubectl delete deploy gcsb-run

Troubleshooting

Kubectl errors

SchemaError(io.k8s.api.autoscaling.v2beta2.MetricTarget): invalid object doesn't have additional properties

There are issues with some kubectl installations from homebrew. Please relink your kubectl installation by following these instructions.