diff --git a/content/operate/rc/databases/rdi/define.md b/content/operate/rc/databases/rdi/define.md index 0c80fc8976..0d18696136 100644 --- a/content/operate/rc/databases/rdi/define.md +++ b/content/operate/rc/databases/rdi/define.md @@ -20,19 +20,15 @@ After you have [prepared your source database]({{}} -1. Enter a **Pipeline name**. This pipeline name will be the prefix to all keys generated by this pipeline in the target database. +1. Enter a **Pipeline name**. {{The pipeline name and deployment CIDR fields.}} -1. Enter the **Deployment CIDR** for your pipeline, or use the one generated for you. This CIDR should not conflict with your apps or other databases. +1. A **Deployment CIDR** is automatically generated for you. If, for any reason, a CIDR is not generated, enter a valid CIDR that does not conflict with your applications or other databases. 1. In the **Source database connectivity** section, enter the **PrivateLink service name** of the [PrivateLink connected to your source database]({{< relref "/operate/rc/databases/rdi/setup#set-up-connectivity" >}}). {{The Source database connectivity section, with database connection details and connectivity options.}} 1. Enter your database details. This depends on your database type, and includes: - **Port**: The database's port - **Database**: Your database's name, or the root database *(PostgreSQL, Oracle only)*, or a comma-separated list of one or more databases you want to connect to *(SQL Server only)* - - **Database Server ID**: Unique ID for the replication client. *(mySQL and mariaDB only)* - Use the following SQL command to get this value: - ```sql - SHOW VARIABLES LIKE 'server_id'; - ``` + - **Database Server ID**: Unique ID for the replication client. Enter a number that is not used by any existing replication clients. *(mySQL and mariaDB only)* - **PDB**: Name of the Oracle pluggable database *(Oracle only)* 1. Enter the ARN of your [database credentials secret]({{< relref "/operate/rc/databases/rdi/setup#share-source-database-credentials" >}}) in the **Source database secrets ARN** field. 1. Select **Start pipeline setup**. @@ -45,7 +41,7 @@ At this point, Redis Cloud will provision the pipeline infrastructure that will {{The Pipeline setup in progress screen.}} -Pipelines are provisioned in the background. You aren't allowed to make changes to your data pipeline or to your database during provisioning. This process will take a long time, so you can close the window and come back later. +Pipelines are provisioned in the background. You aren't allowed to make changes to your data pipeline or to your database during provisioning. This process will take about an hour, so you can close the window and come back later. When your pipeline is provisioned, select **Complete setup**. You will then [define your data pipeline](#define-data-pipeline). @@ -57,12 +53,8 @@ After your pipeline is provisioned, you will be able to define your pipeline. Yo ### Configure a new pipeline -1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab. If your pipeline is already provisioned, select **Complete setup** to go to the **Pipeline definition** section. +1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab. If your pipeline is already provisioned, select **Complete setup** to go to the **Select data** section. {{The complete setup button.}} -1. For the **Configure a new pipeline** option, select the Redis data type to write keys to the target. You can choose **Hash** or **JSON** if the target database supports JSON. - {{The Pipeline definition screen. Configure a new pipeline is selected.}} - Select **Continue**. - {{The continue button.}} 1. Select the Schema and Tables you want to migrate to the target database from the **Source data selection** list. {{The select source data section. }} @@ -87,6 +79,11 @@ After your pipeline is provisioned, you will be able to define your pipeline. Yo After you've selected the schemas and tables you want to sync, select **Continue**. {{The continue button.}} + +1. In the **Pipeline definition** section, select the Redis data type to write keys to the target. You can choose **Hash** or **JSON** if the target database supports JSON. + {{The pipeline definition screen.}} + Select **Continue**. + {{The continue button.}} 1. Review the tables you selected in the **Summary**. If everything looks correct, select **Start ingest** to start ingesting data from your source database. diff --git a/content/operate/rc/databases/rdi/setup.md b/content/operate/rc/databases/rdi/setup.md index 6ce49ab7a6..f62675ee0b 100644 --- a/content/operate/rc/databases/rdi/setup.md +++ b/content/operate/rc/databases/rdi/setup.md @@ -5,7 +5,7 @@ categories: - docs - operate - rc -description: Prepare your source database and database credentials for Data integration. +description: Prepare your source database, network setup, and database credentials for Data integration. hideListLinks: true weight: 1 --- @@ -38,18 +38,31 @@ See [Prepare source databases]({{}}) - [Oracle]({{}}) - [SQL Server]({{}}) -- [AWS Aurora and PostgreSQL]({{}}) +- [AWS Aurora PostgreSQL]({{}}) +- [AWS RDS PostgreSQL]({{}}) See the [RDI architecture overview]({{< relref "/integrate/redis-data-integration/architecture#overview" >}}) for more information about CDC. ## Set up connectivity -To ensure that you can connect your Redis Cloud database to the source database hosted on an AWS EC2 instance, you need to set up an endpoint service through AWS PrivateLink. To do this: +To ensure that you can connect your Redis Cloud database to the source database, you need to set up an endpoint service through AWS PrivateLink. -1. [Create a network load balancer](#create-network-load-balancer) that will route incoming HTTP requests to your database. -1. [Create an endpoint service](#create-endpoint-service) through AWS PrivateLink. +Choose the steps for your database setup: +- [Database hosted on an AWS EC2 instance](#database-hosted-on-an-aws-ec2-instance) +- [Database hosted on AWS RDS or AWS Aurora](#database-hosted-on-aws-rds-or-aws-aurora) -### Create network load balancer +### Database hosted on an AWS EC2 instance + +The following diagram shows the network setup for a database hosted on an AWS EC2 instance. + +{{The network setup for a database hosted on an AWS EC2 instance.}} + +To do this: + +1. [Create a network load balancer](#create-network-load-balancer-ec2) that will route incoming HTTP requests to your database. +1. [Create an endpoint service](#create-endpoint-service-ec2) through AWS PrivateLink. + +#### Create network load balancer {#create-network-load-balancer-ec2} In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Compute** > **EC2**. [Create a network load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html#configure-load-balancer) with the following settings: @@ -67,7 +80,7 @@ In the [AWS Management Console](https://console.aws.amazon.com/), use the **Serv 1. In **Register targets**, select the EC2 instance that runs your source database, enter the port, and select **Include as pending below**. Then, select **Create target group** to create your target group. Return **Listeners and routing** in the Network Load Balancer setup. 1. Set the following **Listener** properties: - **Protocol**: Select **TCP**. - - **Port**: Enter **80**. + - **Port**: Enter your source database's port. - **Default action**: Select the target group you created in the previous step. 1. Review the network load balancer settings, and then select **Create load balancer** to continue. 1. After the network load balancer is active, select **Security**, and then select the security group ID to open the Security group settings. @@ -76,11 +89,79 @@ In the [AWS Management Console](https://console.aws.amazon.com/), use the **Serv - **Source**: Select **Anywhere - IPv4**. Select **Save rules** to save your changes. -### Create endpoint service +#### Create endpoint service {#create-endpoint-service-ec2} In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Networking & Content Delivery** > **VPC**. There, select **PrivateLink and Lattice** > **Endpoint services**. [Create an endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html) with the following settings: -1. In **Available load balancers**, select the [network load balancer](#create-network-load-balancer) you created. +1. In **Available load balancers**, select the [network load balancer](#create-network-load-balancer-ec2) you created. +1. In **Additional settings**, choose the following settings: + - **Require acceptance for endpoint**: Select **Acceptance required**. + - **Supported IP address types**: Select **IPv4**. +1. Select **Create** to create the endpoint service. + +After you create the endpoint service, you need to add Redis Cloud as an Allowed Principal on your [endpoint service VPC permissions](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#add-remove-permissions). + +1. In the Redis Cloud Console, copy the Amazon Resource Name (ARN) provided in the **Setup connectivity** section. +1. Return to the endpoint service list on the [Amazon VPC console](https://console.aws.amazon.com/vpc/). Select the endpoint service you just created. +1. Navigate to **Allow principals** tab. +1. Add the Redis Cloud ARN you copied and choose **Allow principals**. +1. Save the service name for later. + +For more details on AWS PrivateLink, see [Share your services through AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html). + +### Database hosted on AWS RDS or AWS Aurora + +The following diagram shows the network setup for a database hosted on AWS RDS or AWS Aurora. + +{{The network setup for a database hosted on AWS RDS or AWS Aurora.}} + +To do this: + +1. [Create an RDS Proxy](#create-rds-proxy) that will route requests to your database. +1. [Create a network load balancer](#create-network-load-balancer-rds) that will route incoming HTTP requests to the RDS proxy. +1. [Create an endpoint service](#create-endpoint-service-rds) through AWS PrivateLink. + +#### Create RDS proxy {#create-rds-proxy} + +In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Database** > **Aurora and RDS**. [Create an RDS proxy](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy-creating.html) that can access your database. + +#### Create network load balancer {#create-network-load-balancer-rds} + +In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Compute** > **EC2**. [Create a network load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html#configure-load-balancer) with the following settings: + +1. In **Basic configuration**: + - **Scheme**: Select **Internal**. + - **Load balancer IP address type**: Select **IPv4**. +1. In **Network mapping**, select the VPC and availability zone associated with your source database. +1. In **Security groups**, select the security group associated with your source database. +1. In **Listeners and routing**: + 1. Select **Create target group** to [create a target group](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-target-group.html) with the following settings: + 1. In **Specify group details**: + - **Target type**: Select **IP Addresses**. + - **Protocol : Port**: Select **TCP**, and then enter the port number where your database is exposed. + - The **IP address type** and **VPC** should be selected already and match the VPC you selected earlier. + 1. In **Register targets**, enter the static IP address of your RDS proxy, enter the port, and select **Include as pending below**. Then, select **Create target group** to create your target group. Return **Listeners and routing** in the Network Load Balancer setup. + To get the static IP address of your RDS Proxy, run the following command on an EC2 instance in the same VPC as the Proxy: + ```sh + $ nslookup + ``` + Replace `` with the endpoint of your RDS proxy. + 1. Set the following **Listener** properties: + - **Protocol**: Select **TCP**. + - **Port**: Enter your source database's port. + - **Default action**: Select the target group you created in the previous step. +1. Review the network load balancer settings, and then select **Create load balancer** to continue. +1. After the network load balancer is active, select **Security**, and then select the security group ID to open the Security group settings. +1. Select **Edit inbound rules**, then **Add rule** to add a rule with the following settings: + - **Type**: Select **HTTP**. + - **Source**: Select **Anywhere - IPv4**. + Select **Save rules** to save your changes. + +#### Create endpoint service {#create-endpoint-service-rds} + +In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Networking & Content Delivery** > **VPC**. There, select **PrivateLink and Lattice** > **Endpoint services**. [Create an endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html) with the following settings: + +1. In **Available load balancers**, select the [network load balancer](#create-network-load-balancer-rds) you created. 1. In **Additional settings**, choose the following settings: - **Require acceptance for endpoint**: Select **Acceptance required**. - **Supported IP address types**: Select **IPv4**. @@ -106,7 +187,7 @@ To do this, you need to: ### Create encryption key -In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Key Management Service**. [Create an encryption key](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html) with the following settings: +In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Key Management Service**. [Create an encryption key](https://docs.aws.amazon.com/kms/latest/developerguide/create-symmetric-cmk.html) with the following settings: 1. In **Step 1 - Configure key**: - **Key type**: Select **Symmetric**. @@ -118,6 +199,8 @@ In the [AWS Management Console](https://console.aws.amazon.com/), use the **Serv 1. In **Step 3 - Define key administrative permissions**, under **Key deletion**, select **Allow key administrators to delete this key**. 1. In **Step 4 - Define key usage permissions**, under **Other AWS accounts**, select **Add another AWS account**. Enter the AWS account ID for the Redis Cloud cluster that you saved earlier. +Review the key policy and key settings, and then select **Finish** to create the key. + ### Create database credentials secret In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Secrets Manager**. [Create a secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) of type **Other type of secret** with the following settings: @@ -159,7 +242,6 @@ If your source database has TLS or mTLS enabled, we recommend that you enter the After you store this secret, you can view and copy the [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_iam-permissions.html#iam-resources) of your secret on the secret details page. - ## Next steps After you have set up your source database and prepared connectivity and credentials, select **Define source database** to [define your source connection and data pipeline]({{}}). diff --git a/content/operate/rc/databases/rdi/view-edit.md b/content/operate/rc/databases/rdi/view-edit.md index 3a4d1c8ee2..a63f4b2da8 100644 --- a/content/operate/rc/databases/rdi/view-edit.md +++ b/content/operate/rc/databases/rdi/view-edit.md @@ -1,19 +1,21 @@ --- -Title: View data pipeline +Title: View and edit data pipeline alwaysopen: false categories: - docs - operate - rc -description: Observe and stop your data pipeline. +description: Observe and change your data pipeline. hideListLinks: true weight: 3 --- -Use the **Data pipeline** tab in your database to view your data pipeline. +Use the **Data pipeline** tab in your database to view and edit your data pipeline. The **Data pipeline** tab gives an overview of your data pipeline and lets you view your data stream metrics. + +<--! TODO: REPLACE SCREENSHOT --> {{The select source database type list.}} The **Status** table shows statistics for the whole data pipeline: @@ -41,7 +43,6 @@ The **Data stream metrics** table shows the following metrics for each data stre | **Filtered** | Number of records from the source table that were filtered from being inserted into the target database. | | **Rejected** | Number of records from the source table that could not be parsed or inserted into the target database. | - +At this point, the data pipeline will re-ingest data from the source database to your target Redis database. ## Stop and restart data pipeline diff --git a/static/images/rc/rdi/pipeline-status-error.png b/static/images/rc/rdi/pipeline-status-error.png index 2429dd725c..d0c10cee88 100644 Binary files a/static/images/rc/rdi/pipeline-status-error.png and b/static/images/rc/rdi/pipeline-status-error.png differ diff --git a/static/images/rc/rdi/pipeline-status-initial-sync.png b/static/images/rc/rdi/pipeline-status-initial-sync.png index 6892cb3d6a..8cf4beaf51 100644 Binary files a/static/images/rc/rdi/pipeline-status-initial-sync.png and b/static/images/rc/rdi/pipeline-status-initial-sync.png differ diff --git a/static/images/rc/rdi/pipeline-status-stopped.png b/static/images/rc/rdi/pipeline-status-stopped.png index 19368975f7..cace177ea1 100644 Binary files a/static/images/rc/rdi/pipeline-status-stopped.png and b/static/images/rc/rdi/pipeline-status-stopped.png differ diff --git a/static/images/rc/rdi/pipeline-status-streaming.png b/static/images/rc/rdi/pipeline-status-streaming.png index 18fa67c96c..912c8c2b24 100644 Binary files a/static/images/rc/rdi/pipeline-status-streaming.png and b/static/images/rc/rdi/pipeline-status-streaming.png differ diff --git a/static/images/rc/rdi/rdi-add-schema.png b/static/images/rc/rdi/rdi-add-schema.png index d7053d35b6..d33a0be927 100644 Binary files a/static/images/rc/rdi/rdi-add-schema.png and b/static/images/rc/rdi/rdi-add-schema.png differ diff --git a/static/images/rc/rdi/rdi-apply-changes.png b/static/images/rc/rdi/rdi-apply-changes.png new file mode 100644 index 0000000000..01e1d23285 Binary files /dev/null and b/static/images/rc/rdi/rdi-apply-changes.png differ diff --git a/static/images/rc/rdi/rdi-complete-setup.png b/static/images/rc/rdi/rdi-complete-setup.png index 39450aca05..e5467ef40f 100644 Binary files a/static/images/rc/rdi/rdi-complete-setup.png and b/static/images/rc/rdi/rdi-complete-setup.png differ diff --git a/static/images/rc/rdi/rdi-configure-new-pipeline.png b/static/images/rc/rdi/rdi-configure-new-pipeline.png index b6a8fbdb77..4ceeb8fa25 100644 Binary files a/static/images/rc/rdi/rdi-configure-new-pipeline.png and b/static/images/rc/rdi/rdi-configure-new-pipeline.png differ diff --git a/static/images/rc/rdi/rdi-continue-button.png b/static/images/rc/rdi/rdi-continue-button.png index b06ebfcc9b..704086a163 100644 Binary files a/static/images/rc/rdi/rdi-continue-button.png and b/static/images/rc/rdi/rdi-continue-button.png differ diff --git a/static/images/rc/rdi/rdi-create-data-pipeline.png b/static/images/rc/rdi/rdi-create-data-pipeline.png index c6403da462..0bbee9396f 100644 Binary files a/static/images/rc/rdi/rdi-create-data-pipeline.png and b/static/images/rc/rdi/rdi-create-data-pipeline.png differ diff --git a/static/images/rc/rdi/rdi-define-connectivity.png b/static/images/rc/rdi/rdi-define-connectivity.png index 1b8ca02f7d..e0f297a5c4 100644 Binary files a/static/images/rc/rdi/rdi-define-connectivity.png and b/static/images/rc/rdi/rdi-define-connectivity.png differ diff --git a/static/images/rc/rdi/rdi-define-pipeline-cidr.png b/static/images/rc/rdi/rdi-define-pipeline-cidr.png index 66cb61caff..d40ca25af4 100644 Binary files a/static/images/rc/rdi/rdi-define-pipeline-cidr.png and b/static/images/rc/rdi/rdi-define-pipeline-cidr.png differ diff --git a/static/images/rc/rdi/rdi-define-source-database.png b/static/images/rc/rdi/rdi-define-source-database.png index 3922b68b64..ab1b9e4450 100644 Binary files a/static/images/rc/rdi/rdi-define-source-database.png and b/static/images/rc/rdi/rdi-define-source-database.png differ diff --git a/static/images/rc/rdi/rdi-delete-schema.png b/static/images/rc/rdi/rdi-delete-schema.png index aff121e8a7..003cfe3b5c 100644 Binary files a/static/images/rc/rdi/rdi-delete-schema.png and b/static/images/rc/rdi/rdi-delete-schema.png differ diff --git a/static/images/rc/rdi/rdi-edit-button.png b/static/images/rc/rdi/rdi-edit-button.png index c8b9af90c3..28cbec8c87 100644 Binary files a/static/images/rc/rdi/rdi-edit-button.png and b/static/images/rc/rdi/rdi-edit-button.png differ diff --git a/static/images/rc/rdi/rdi-missing-unique-constraint.png b/static/images/rc/rdi/rdi-missing-unique-constraint.png index 23871ce556..b1eb80b732 100644 Binary files a/static/images/rc/rdi/rdi-missing-unique-constraint.png and b/static/images/rc/rdi/rdi-missing-unique-constraint.png differ diff --git a/static/images/rc/rdi/rdi-pipeline-setup-in-progress.png b/static/images/rc/rdi/rdi-pipeline-setup-in-progress.png index 78c74858c9..d63d5b1263 100644 Binary files a/static/images/rc/rdi/rdi-pipeline-setup-in-progress.png and b/static/images/rc/rdi/rdi-pipeline-setup-in-progress.png differ diff --git a/static/images/rc/rdi/rdi-select-colummns.png b/static/images/rc/rdi/rdi-select-colummns.png new file mode 100644 index 0000000000..23d0420555 Binary files /dev/null and b/static/images/rc/rdi/rdi-select-colummns.png differ diff --git a/static/images/rc/rdi/rdi-select-constraints.png b/static/images/rc/rdi/rdi-select-constraints.png index 37a79afc1a..2ed7d04eb0 100644 Binary files a/static/images/rc/rdi/rdi-select-constraints.png and b/static/images/rc/rdi/rdi-select-constraints.png differ diff --git a/static/images/rc/rdi/rdi-select-source-data.png.png b/static/images/rc/rdi/rdi-select-source-data.png.png new file mode 100644 index 0000000000..d19e5046de Binary files /dev/null and b/static/images/rc/rdi/rdi-select-source-data.png.png differ diff --git a/static/images/rc/rdi/rdi-select-source-db.png b/static/images/rc/rdi/rdi-select-source-db.png index a44f529bcb..db039556c0 100644 Binary files a/static/images/rc/rdi/rdi-select-source-db.png and b/static/images/rc/rdi/rdi-select-source-db.png differ diff --git a/static/images/rc/rdi/rdi-setup-diagram-aurora.png b/static/images/rc/rdi/rdi-setup-diagram-aurora.png new file mode 100644 index 0000000000..212e2f3a68 Binary files /dev/null and b/static/images/rc/rdi/rdi-setup-diagram-aurora.png differ diff --git a/static/images/rc/rdi/rdi-setup-diagram-ec2.png b/static/images/rc/rdi/rdi-setup-diagram-ec2.png new file mode 100644 index 0000000000..c5cee65fe9 Binary files /dev/null and b/static/images/rc/rdi/rdi-setup-diagram-ec2.png differ diff --git a/static/images/rc/rdi/rdi-source-dataset-size.png b/static/images/rc/rdi/rdi-source-dataset-size.png index f2a9c7f2be..35b78e2ecf 100644 Binary files a/static/images/rc/rdi/rdi-source-dataset-size.png and b/static/images/rc/rdi/rdi-source-dataset-size.png differ diff --git a/static/images/rc/rdi/rdi-start-ingest.png b/static/images/rc/rdi/rdi-start-ingest.png index 9c09b04206..b23878a723 100644 Binary files a/static/images/rc/rdi/rdi-start-ingest.png and b/static/images/rc/rdi/rdi-start-ingest.png differ diff --git a/static/images/rc/rdi/rdi-update-preferences.png b/static/images/rc/rdi/rdi-update-preferences.png new file mode 100644 index 0000000000..7774ca2ba4 Binary files /dev/null and b/static/images/rc/rdi/rdi-update-preferences.png differ