Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
165 changes: 165 additions & 0 deletions content/en/experiments/connecting_redshift.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
---
title: Connect Redshift for Warehouse Native Experiment Analysis
description: Connect a Redshift service account to enable warehouse native experiment analysis.
private: true
further_reading:
- link: "/experiments/defining_metrics"
tag: "Documentation"
text: "Defining metrics in Datadog Experiments"
- link: "https://www.datadoghq.com/blog/experimental-data-datadog/"
tag: "Blog"
text: "How to bridge speed and quality in experiments through unified data"
---

## Overview

This guide walks through connecting AWS Redshift to Datadog to enable warehouse-native experiment analysis in four steps: connecting AWS to Datadog, creating necessary resources in AWS, granting permissions to the Datadog service user, and configuring experiment-specific settings in Datadog.

Check warning on line 16 in content/en/experiments/connecting_redshift.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.aws

Use the proper trademark 'Amazon Redshift' instead of 'AWS Redshift'.

Check notice on line 16 in content/en/experiments/connecting_redshift.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.sentencelength

Suggestion: Try to keep your sentence length to 25 words or fewer.

## Step 1: Connect AWS to Datadog

Check warning on line 18 in content/en/experiments/connecting_redshift.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.headings

'Step 1: Connect AWS to Datadog' should use sentence-style capitalization.

Datadog Experiments connects to Redshift through the Datadog AWS integration. If you already have an [AWS integration][1] configured for the account containing your Redshift cluster, you can skip to the next step.

If you haven't set up the AWS integration yet:

1. Navigate to the [AWS integration tile][2] in Datadog.
2. Click **Add AWS Account** and follow the setup flow using the CloudFormation template. This creates an IAM role that allows Datadog to make API calls to your AWS account.
3. For more details, see the [AWS integration documentation][1].

If you're only using the AWS integration for warehouse native experiment analysis, you can opt out of collecting other resources.

## Step 2: Create resources in AWS

Check warning on line 30 in content/en/experiments/connecting_redshift.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.headings

'Step 2: Create resources in AWS' should use sentence-style capitalization.

Before you connect Datadog Experiments to your data warehouse, create a service user for Datadog.

You will also need to create a schema for Datadog Experiments to write intermediary tables to, as well as grant the service User read access to tables you'd like Datadog Experiments to query SQL definitions from.

Check warning on line 34 in content/en/experiments/connecting_redshift.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.tense

Avoid temporal words like 'will'.

Check notice on line 34 in content/en/experiments/connecting_redshift.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.sentencelength

Suggestion: Try to keep your sentence length to 25 words or fewer.

### Create a Service User for Datadog Experiments

Check warning on line 36 in content/en/experiments/connecting_redshift.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.headings

'Create a Service User for Datadog Experiments' should use sentence-style capitalization.

1. Connect to your Redshift cluster with a user that has superuser or admin privileges.
2. Create a user with the following command:

```sql
CREATE USER datadog_experiments_user PASSWORD '<strong_password>';
```

3. Grant this user the appropriate privileges for any tables needed for building experiment metrics. Replace `<schema>` and `<table>` with the appropriate values.

```sql
GRANT USAGE ON SCHEMA <schema> TO datadog_experiments_user;

-- individual tables
GRANT SELECT ON TABLE <schema>.<table> TO datadog_experiments_user;

-- all tables
GRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO datadog_experiments_user;
```

### Create Schema for Datadog Experiments to Write Results to

Check warning on line 57 in content/en/experiments/connecting_redshift.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.headings

'Create Schema for Datadog Experiments to Write Results to' should use sentence-style capitalization.

Create a schema for Datadog Experiments to write intermediate results and temporary tables.

```sql
CREATE SCHEMA IF NOT EXISTS datadog_experiments_output;
GRANT ALL ON SCHEMA datadog_experiments_output TO datadog_experiments_user;
```

### Create an S3 Bucket

Check warning on line 66 in content/en/experiments/connecting_redshift.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.headings

'Create an S3 Bucket' should use sentence-style capitalization.

Create an S3 bucket for importing exposure events into your warehouse. The bucket must start with `"datadog-experimentation-"`, e.g. `datadog-experimentation-[AWS account ID]`. Default settings can be used.

Check warning on line 68 in content/en/experiments/connecting_redshift.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.abbreviations_latin

Use 'for example' instead of abbreviations like 'e.g.'.

## Step 3: Grant additional IAM permissions

Because Datadog Experiments writes data into your warehouse, your AWS IAM role used by the Datadog integration needs additional permissions beyond those required for the standard [Amazon Redshift Datadog integration][1].

Check notice on line 72 in content/en/experiments/connecting_redshift.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.sentencelength

Suggestion: Try to keep your sentence length to 25 words or fewer.

Add the following permissions to the IAM role used by your Datadog AWS integration:

```JSON
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GetClusterCreds",
"Effect": "Allow",
"Action": [
"redshift:GetClusterCredentials"
],
"Resource": [
"[Redshift cluster ARN]",
"[Redshift user ARN]",
"[Redshift database ARN]"
]
},
{
"Sid": "QueryRedshift",
"Effect": "Allow",
"Action": [
"redshift-data:ExecuteStatement",
"redshift-data:GetStatementResult",
"redshift-data:DescribeStatement",
"redshift-data:ListStatements",
"redshift-data:CancelStatement"
],
"Resource": "*"
},
{
"Sid": "ListTheBucket",
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": "[S3 bucket ARN]"
},
{
"Sid": "ObjectRW",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": "[S3 bucket ARN]/*"
}
]
}
```

Replace the following with the appropriate values:

| Field | Example |
|-------|---------|
| `[Redshift cluster ARN]` | `arn:aws:redshift:us-east-1:[account-id]:namespace:[namespace-id]` |
| `[Redshift user ARN]` | `arn:aws:redshift:us-east-1:312078929830:dbuser:[cluster-name]/[user]` |
| `[Redshift database ARN]`| `arn:aws:redshift:us-east-1:[account-id]:dbname:[cluster-name]` |
| `[S3 bucket ARN]` | `arn:aws:s3:::[bucket-name]` |

## Step 4: Configure experiment settings

Once your AWS service user is connected to Datadog, follow these steps to finish the integration.

Check warning on line 137 in content/en/experiments/connecting_redshift.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.words_case_sensitive

Use 'After' instead of 'Once'.

1. Navigate to the [Warehouse Connections page][3].
2. Click **Redshift**.
3. Under **Select AWS Account**, choose the AWS account you configured above.
4. Under **Cluster Connection**, enter:
- **AWS region** — the region your Redshift cluster is in (e.g., `us-east-1`)

Check warning on line 143 in content/en/experiments/connecting_redshift.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.dashes

Don't put a space before or after a dash.
- **Cluster identifier** — the name of your Redshift cluster
- **Cluster endpoint** — the full endpoint URL for your cluster
- **Port** — the port your cluster is listening on (default: `5439`)
5. Under Database and Storage, enter:
- **Database** — the database containing your source tables
- **Database user** — the service user you created earlier (e.g., `datadog_experiments_user`)
- **Schema** — the schema you created for Datadog Experiments to write to (e.g., `datadog_experiments_output`)
- **Temp S3 bucket** — the S3 bucket you created earlier (e.g., `datadog-experimentation-[AWS account ID]`)
6. Click **Save**

{{< img src="/product_analytics/experiment/guide/redshift_experiment_setup.png" alt="The Edit Data Warehouse modal with Redshift selected, showing inputs for AWS account, AWS region, cluster identifier, cluster endpoint ,database, database user, schema, and temp S3 bucket" style="width:90%;" >}}

After you save your warehouse connection, create experiment metrics using your Redshift data. See [Create Experiment Metrics][4].

## Further reading

{{< partial name="whats-next/whats-next.html" >}}

[1]: https://docs.datadoghq.com/integrations/amazon-web-services/
[2]: https://app.datadoghq.com/integrations/aws
[3]: https://app.datadoghq.com/product-analytics/experiments/settings/warehouse-connections
[4]: /experiments/defining_metrics
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading