diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 6d7e7812e..397424b00 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -26,10 +26,14 @@ * xref:networking:index.adoc[Networking] ** xref:networking:cloud-security-network.adoc[] ** xref:networking:cidr-ranges.adoc[] +** xref:networking:serverless/index.adoc[Serverless] +*** xref:networking:serverless/aws/index.adoc[AWS] +**** xref:networking:serverless/aws/privatelink-ui.adoc[Configure PrivateLink in the Cloud Console] +**** xref:networking:serverless/aws/privatelink-api.adoc[Configure PrivateLink with the Cloud API] ** xref:networking:byoc/index.adoc[BYOC] *** xref:networking:byoc/aws/index.adoc[AWS] **** xref:networking:byoc/aws/vpc-peering-aws.adoc[Add a Peering Connection] -**** xref:networking:configure-privatelink-in-cloud-ui.adoc[Configure PrivateLink in the Cloud UI] +**** xref:networking:configure-privatelink-in-cloud-ui.adoc[Configure PrivateLink in the Cloud Console] **** xref:networking:aws-privatelink.adoc[Configure PrivateLink with the Cloud API] **** xref:networking:byoc/aws/transit-gateway.adoc[Add a Transit Gateway] *** xref:networking:byoc/azure/index.adoc[Azure] @@ -37,20 +41,20 @@ **** xref:networking:azure-private-link.adoc[] *** xref:networking:byoc/gcp/index.adoc[GCP] **** xref:networking:byoc/gcp/vpc-peering-gcp.adoc[Add a Peering Connection] -**** xref:networking:configure-private-service-connect-in-cloud-ui.adoc[Configure Private Service Connect in the Cloud UI] +**** xref:networking:configure-private-service-connect-in-cloud-ui.adoc[Configure Private Service Connect in the Cloud Console] **** xref:networking:gcp-private-service-connect.adoc[Configure Private Service Connect with the Cloud API] **** xref:networking:byoc/gcp/enable-global-access.adoc[Enable Global Access] ** xref:networking:dedicated/index.adoc[Dedicated] *** xref:networking:dedicated/aws/index.adoc[AWS] **** xref:networking:dedicated/aws/vpc-peering.adoc[Add a Peering Connection] -**** xref:networking:configure-privatelink-in-cloud-ui.adoc[Configure PrivateLink in the Cloud UI] +**** xref:networking:configure-privatelink-in-cloud-ui.adoc[Configure PrivateLink in the Cloud Console] **** xref:networking:aws-privatelink.adoc[] *** xref:networking:dedicated/azure/index.adoc[Azure] -**** xref:networking:azure-private-link-in-ui.adoc[] +**** xref:networking:azure-private-link-in-ui.adoc[Configure Private Link in the Cloud Console] **** xref:networking:azure-private-link.adoc[] *** xref:networking:dedicated/gcp/index.adoc[GCP] **** xref:networking:dedicated/gcp/vpc-peering-gcp.adoc[Add a Peering Connection] -**** xref:networking:dedicated/gcp/configure-psc-in-ui.adoc[Configure Private Service Connect in the Cloud UI] +**** xref:networking:dedicated/gcp/configure-psc-in-ui.adoc[Configure Private Service Connect in the Cloud Console] **** xref:networking:dedicated/gcp/configure-psc-in-api.adoc[Configure Private Service Connect with the Cloud API] * xref:security:index.adoc[Security] diff --git a/modules/get-started/pages/cloud-overview.adoc b/modules/get-started/pages/cloud-overview.adoc index 9e03a35b8..57b85e371 100644 --- a/modules/get-started/pages/cloud-overview.adoc +++ b/modules/get-started/pages/cloud-overview.adoc @@ -59,7 +59,7 @@ Redpanda Cloud offers three fully-managed cloud deployment options, each designe | 20 (default), 32 (max) | *Private networking* -| ✗ +| ✓ | ✓ | ✓ @@ -118,7 +118,6 @@ Serverless is the fastest and easiest way to start data streaming. With Serverle [NOTE] ==== -* Serverless on AWS is currently in a glossterm:LA[,limited availability (LA)] release. * Serverless on GCP is currently in a glossterm:beta[] release. ==== @@ -177,7 +176,6 @@ Serverless clusters are a good fit for the following use cases: Consider BYOC or Dedicated if you need more control over the deployment or if you have workloads with consistently-high throughput. BYOC and Dedicated clusters offer the following features: -* Private networking * Multiple availability zones (AZs). A multi-AZ cluster provides higher resiliency in the event of a failure in one of the zones. * Role-based access control (RBAC) in the data plane * Kafka Connect @@ -382,7 +380,6 @@ Features in limited availability are production-ready and are covered by Redpand The following features are currently in limited availability in Redpanda Cloud: -* Serverless * Dedicated for Azure == Features in beta diff --git a/modules/get-started/pages/cluster-types/serverless.adoc b/modules/get-started/pages/cluster-types/serverless.adoc index c720133e4..5a3251776 100644 --- a/modules/get-started/pages/cluster-types/serverless.adoc +++ b/modules/get-started/pages/cluster-types/serverless.adoc @@ -7,7 +7,6 @@ Serverless is the fastest and easiest way to start data streaming. With Serverle [NOTE] ==== -* Serverless on AWS is currently in a glossterm:LA[,limited availability (LA)] release. * Serverless on GCP is currently in a glossterm:beta[] release. ==== @@ -50,9 +49,15 @@ To create a Serverless cluster: . Select a cloud provider and xref:reference:tiers/serverless-regions.adoc[region]. For best performance, select the region closest to your applications. Redpanda expects your applications to be deployed in the same cloud provider and region as your Serverless cluster. + -Serverless clusters are not guaranteed to be pinned to a particular availability zone within the selected region. +Serverless clusters are available in the regions listed in xref:reference:tiers/serverless-regions.adoc[Serverless regions]. Redpanda expects your applications to be deployed in the same region. For best performance, select the region closest to your applications. Serverless is not guaranteed to be pinned to a particular availability zone within that region. ++ +Clusters on AWS can enable private access between their VPC and Redpanda, so data does not traverse the public internet. Private connectivity is implemented using AWS PrivateLink for secure ingress traffic. A Serverless cluster can have both public and private access enabled. Enabling private access incurs additional charges. If you select private access, you can either create a new PrivateLink, or, if you have PrivateLinks for other Serverless clusters in this same resource group, you can use an existing PrivateLink. ++ +You can enable or disable private access at any time from the cluster's *Settings* page. After private access is disabled, attempts to reach the private endpoints will fail. However, the PrivateLink endpoint in your AWS account and the PrivateLink resource in Redpanda Cloud both remain provisioned and continue to incur charges until you explicitly delete them. + +. Click **Create cluster**. -. Add team members and grant them access with glossterm:ACL[,access control lists (ACLs)] on the *Security* page. +. To start working with your cluster, go to the *Topics* page to create a topic. Under the *Actions* dropdown, you can produce messages to it. Add team members and grant them access with ACLs on the *Security* page. == Interact with your cluster @@ -89,7 +94,6 @@ Explore the rest of the UI: Not all features included in BYOC clusters are available in Serverless. For example, the following features are not supported: * HTTP Proxy API -* Private networking (VPC peering or AWS PrivateLink) * Multiple availability zones (AZs) * RBAC in the data plane and mTLS authentication for Kafka API clients * Kafka Connect diff --git a/modules/get-started/pages/whats-new-cloud.adoc b/modules/get-started/pages/whats-new-cloud.adoc index 512f3cf8c..5a1b1f426 100644 --- a/modules/get-started/pages/whats-new-cloud.adoc +++ b/modules/get-started/pages/whats-new-cloud.adoc @@ -8,10 +8,15 @@ This page lists new features added to Redpanda Cloud. == January 2026 +=== Serverless on AWS: GA + +xref:get-started:cluster-types/serverless.adoc[Serverless] on AWS is now generally available (GA). This GA release includes private networking with AWS PrivateLink and the ability to view and export metrics from Serverless clusters to third-party monitoring systems. + === Redpanda Connect and Roles in Terraform provider The xref:manage:terraform-provider.adoc[Redpanda Terraform provider] now supports managing roles and Redpanda Connect pipelines. Use the provider to create and manage role-based access control and data pipelines in Redpanda Cloud. + == December 2025 === Remote MCP: GA diff --git a/modules/networking/pages/azure-private-link-in-ui.adoc b/modules/networking/pages/azure-private-link-in-ui.adoc index 912e075a9..c9fbdfdb2 100644 --- a/modules/networking/pages/azure-private-link-in-ui.adoc +++ b/modules/networking/pages/azure-private-link-in-ui.adoc @@ -1,7 +1,7 @@ -= Configure Azure Private Link in the Cloud UI -:description: Set up Azure Private Link in the Redpanda Cloud UI. += Configure Azure Private Link in the Cloud Console +:description: Set up Azure Private Link in the Redpanda Cloud Console. -NOTE: This guide is for configuring new clusters with Azure Private Link using the Redpanda Cloud UI. To configure and manage Private Link on an existing cluster, you must use the xref:networking:azure-private-link.adoc[Cloud API]. +NOTE: This guide is for configuring new clusters with Azure Private Link using the Redpanda Cloud Console. To configure and manage Private Link on an existing cluster, you must use the xref:networking:azure-private-link.adoc[Cloud API]. The Redpanda Azure Private Link service provides secure access to Redpanda Cloud from your own VNet. Traffic over Private Link does not go through the public internet because these connections are treated as their own private Azure service. While your VNet has access to the Redpanda virtual network, Redpanda cannot access your VNet. @@ -19,7 +19,7 @@ TIP: In Kafka clients, set `connections.max.idle.ms` to a value less than 350 se == Enable endpoint service for new clusters -. In the Redpanda Cloud UI, create a new cluster. +. In the Redpanda Cloud Console, create a new cluster. . On the *Networking* page: .. For *Connection type*, select *Private*. .. For *Azure Private Link*, select *Enabled*. diff --git a/modules/networking/pages/azure-private-link.adoc b/modules/networking/pages/azure-private-link.adoc index 8fc7e6780..d0038c179 100644 --- a/modules/networking/pages/azure-private-link.adoc +++ b/modules/networking/pages/azure-private-link.adoc @@ -1,7 +1,7 @@ = Configure Azure Private Link with the Cloud API :description: Set up Azure Private Link with the Cloud API. -NOTE: For UI-based configuration of Azure Private Link on new clusters, see xref:networking:azure-private-link-in-ui.adoc[Configure Azure Private Link in the Cloud UI]. +NOTE: For UI-based configuration of Azure Private Link on new clusters, see xref:networking:azure-private-link-in-ui.adoc[Configure Azure Private Link in the Cloud Console]. The Redpanda Azure Private Link service provides secure access to Redpanda Cloud from your own virtual network. Traffic over Azure Private Link does not go through the public internet, but instead through Microsoft's backbone network. While clients can initiate connections against the Redpanda Cloud cluster endpoints, Redpanda Cloud services cannot access your virtual networks directly. @@ -52,7 +52,7 @@ If you have not yet created a cluster in Redpanda Cloud, <> or <> using `rpk` or cURL. +. In the Redpanda Cloud Console, go to https://cloud.redpanda.com/users?tab=users[**Users**^] and create a new user to authenticate the Private Link endpoint connections with the service. You will need the username and password to <> or <> using `rpk` or cURL. . Call the link:/api/doc/cloud-controlplane/operation/operation-clusterservice_getcluster[`GET /v1/clusters/\{id}`] endpoint to check the service status and retrieve the service ID, DNS name, and Redpanda Console URL to use. + @@ -288,7 +288,7 @@ az network private-dns record-set a add-record \ == Connect to Redpanda services through Private Link endpoints -After you enable Private Link for your cluster, your connection URLs are available in the *How to Connect* section of the cluster overview in the Redpanda Cloud UI. +After you enable Private Link for your cluster, your connection URLs are available in the *How to Connect* section of the cluster overview in the Redpanda Cloud Console. include::networking:partial$private-links-access-rp-services-through-vpc.adoc[] diff --git a/modules/networking/pages/configure-privatelink-in-cloud-ui.adoc b/modules/networking/pages/configure-privatelink-in-cloud-ui.adoc index 88fa75806..873b5fc52 100644 --- a/modules/networking/pages/configure-privatelink-in-cloud-ui.adoc +++ b/modules/networking/pages/configure-privatelink-in-cloud-ui.adoc @@ -1,8 +1,8 @@ -= Configure AWS PrivateLink in the Cloud UI -:description: Set up AWS PrivateLink in the Redpanda Cloud UI. += Configure AWS PrivateLink in the Cloud Console +:description: Set up AWS PrivateLink in the Redpanda Cloud Console. :page-aliases: deploy:deployment-option/cloud/configure-privatelink-in-cloud-ui.adoc -NOTE: This guide is for configuring AWS PrivateLink using the Redpanda Cloud UI. To configure and manage PrivateLink on an existing public cluster, you must use the xref:networking:aws-privatelink.adoc[Redpanda Cloud API]. +NOTE: This guide is for configuring AWS PrivateLink using the Redpanda Cloud Console. To configure and manage PrivateLink on an existing public cluster, you must use the xref:networking:aws-privatelink.adoc[Redpanda Cloud API]. The Redpanda AWS PrivateLink endpoint service provides secure access to Redpanda Cloud from your own VPC. Traffic over PrivateLink does not go through the public internet because these connections are treated as their own private AWS service. While your VPC has access to the Redpanda VPC, Redpanda cannot access your VPC. @@ -29,7 +29,7 @@ include::networking:partial$dns_resolution.adoc[] == Enable endpoint service for existing clusters -. In the Redpanda Cloud UI, select your https://cloud.redpanda.com/clusters[cluster^], and go to the *Cluster settings* page. +. In the Redpanda Cloud Console, select your https://cloud.redpanda.com/clusters[cluster^], and go to the *Cluster settings* page. . For AWS PrivateLink, click *Enable*. . On the Enable PrivateLink page, for Allowed principal ARNs, click *Add*, and enter the Amazon Resource Names (ARNs) for each AWS principal allowed to access the endpoint service. For example, for all principals in a specific account, use `arn:aws:iam:::root`. See the AWS documentation on https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#add-remove-permission[configuring an endpoint service^] for details. . Click *Add* after entering each ARN, and when finished, click *Enable*. @@ -39,7 +39,7 @@ NOTE: For help with issues when enabling PrivateLink, contact https://support.re == Access Redpanda services through VPC endpoint -After you have enabled PrivateLink for your cluster, your connection URLs are available in the *How to Connect* section of the cluster overview in the Redpanda Cloud UI. +After you have enabled PrivateLink for your cluster, your connection URLs are available in the *How to Connect* section of the cluster overview in the Redpanda Cloud Console. include::networking:partial$private-links-access-rp-services-through-vpc.adoc[] diff --git a/modules/networking/pages/dedicated/aws/vpc-peering.adoc b/modules/networking/pages/dedicated/aws/vpc-peering.adoc index 8882f1d01..aae4eb4e6 100644 --- a/modules/networking/pages/dedicated/aws/vpc-peering.adoc +++ b/modules/networking/pages/dedicated/aws/vpc-peering.adoc @@ -22,7 +22,7 @@ To create a peering connection between your VPC and Redpanda's VPC: . In the Redpanda Cloud UI, go to the *Overview* page for your cluster. . In the Details section, click the name of the Redpanda network. -. On the Networks page, click *VPC peering walkthrough*. +. On the Networking page, click *VPC peering walkthrough*. . For *Connection name*, enter a name. For example, the name might refer to the VPC ID of the VPC you created in AWS. . For *AWS account number*, enter the account number associated with the VPC you want to connect to. . For *AWS VPC ID*, enter the VPC ID by copying it from the AWS VPC Console. diff --git a/modules/networking/pages/dedicated/gcp/configure-psc-in-api.adoc b/modules/networking/pages/dedicated/gcp/configure-psc-in-api.adoc index 1b0a773ee..c65430803 100644 --- a/modules/networking/pages/dedicated/gcp/configure-psc-in-api.adoc +++ b/modules/networking/pages/dedicated/gcp/configure-psc-in-api.adoc @@ -6,7 +6,7 @@ include::networking:partial$psc-api.adoc[] == Create a new cluster with Private Service Connect -. In the https://cloud.redpanda.com/[Redpanda Cloud UI], go to **Resource groups** and select the resource group in which you want to create a cluster. +. In the https://cloud.redpanda.com/[Redpanda Cloud Console], go to **Resource groups** and select the resource group in which you want to create a cluster. + Copy and store the resource group ID (UUID) from the URL in the browser. + @@ -104,7 +104,7 @@ Enabling Private Service Connect on your VPC interrupts all communication on exi To avoid disruption, consider using a staged approach. See: xref:networking:dedicated/gcp/vpc-peering-gcp.adoc#switch-from-vpc-peering-to-private-service-connect[Switch from VPC peering to Private Service Connect]. ==== -. In the Redpanda Cloud UI, go to the cluster overview and copy the cluster ID from the **Details** section. +. In the Redpanda Cloud Console, go to the cluster overview and copy the cluster ID from the **Details** section. + [,bash] ---- diff --git a/modules/networking/pages/dedicated/gcp/configure-psc-in-ui.adoc b/modules/networking/pages/dedicated/gcp/configure-psc-in-ui.adoc index 85af2f836..b2b8125d3 100644 --- a/modules/networking/pages/dedicated/gcp/configure-psc-in-ui.adoc +++ b/modules/networking/pages/dedicated/gcp/configure-psc-in-ui.adoc @@ -1,11 +1,11 @@ -= Configure GCP Private Service Connect in the Cloud UI -:description: Set up GCP Private Service Connect in the Redpanda Cloud UI. += Configure GCP Private Service Connect in the Cloud Console +:description: Set up GCP Private Service Connect in the Redpanda Cloud Console. :env-dedicated: true [NOTE] ==== -* This guide is for configuring GCP Private Service Connect using the Redpanda Cloud UI. To configure and manage Private Service Connect on an existing cluster with *public* networking, you must use the xref:networking:gcp-private-service-connect.adoc[Cloud API for BYOC] or the xref:networking:dedicated/gcp/configure-psc-in-api.adoc[Cloud API for Dedicated]. +* This guide is for configuring GCP Private Service Connect using the Redpanda Cloud Console. To configure and manage Private Service Connect on an existing cluster with *public* networking, you must use the xref:networking:gcp-private-service-connect.adoc[Cloud API for BYOC] or the xref:networking:dedicated/gcp/configure-psc-in-api.adoc[Cloud API for Dedicated]. * The latest version of Redpanda GCP Private Service Connect (available March, 2025) supports AZ affinity. This allows requests from Private Service Connect endpoints to stay within the same availability zone, avoiding additional networking costs. * DEPRECATION: The original Redpanda GCP Private Service Connect is deprecated and will be removed in a future release. For more information, see xref:manage:maintenance.adoc#deprecated-features[Deprecated features]. ==== @@ -30,7 +30,7 @@ Consider using Private Service Connect if you have multiple VPC networks and cou == Enable Private Service Connect for existing clusters -. In the Redpanda Cloud UI, open your https://cloud.redpanda.com/clusters[cluster^], and click **Cluster settings**. +. In the Redpanda Cloud Console, open your https://cloud.redpanda.com/clusters[cluster^], and click **Cluster settings**. . Under Private Service Connect, click **Enable**. ifdef::env-byoc[] . For xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[BYOVPC clusters], you need a NAT subnet with `purpose` set to `PRIVATE_SERVICE_CONNECT`. You also need to create VPC network firewall rules to allow Private Service Connect traffic. You can use the `gcloud` CLI: diff --git a/modules/networking/pages/dedicated/gcp/vpc-peering-gcp.adoc b/modules/networking/pages/dedicated/gcp/vpc-peering-gcp.adoc index cba79c27c..e2365ffe1 100644 --- a/modules/networking/pages/dedicated/gcp/vpc-peering-gcp.adoc +++ b/modules/networking/pages/dedicated/gcp/vpc-peering-gcp.adoc @@ -21,7 +21,7 @@ A peering becomes active after both Redpanda and GCP create a peering that targe . In the Redpanda Cloud UI, go to the *Overview* page for your cluster. . In the Details section, click the name of the Redpanda network. -. On the Networks page for your cluster, click *VPC peering walkthrough*. +. On the Networking page for your cluster, click *VPC peering walkthrough*. . For *Connection name*, enter a name for the connection. + For example, the name might refer to the VPC ID of the VPC you created in GCP. diff --git a/modules/networking/pages/serverless/aws/index.adoc b/modules/networking/pages/serverless/aws/index.adoc new file mode 100644 index 000000000..d6b6210db --- /dev/null +++ b/modules/networking/pages/serverless/aws/index.adoc @@ -0,0 +1,3 @@ += AWS +:description: Learn how to configure private networking for Serverless clusters on AWS. +:page-layout: index \ No newline at end of file diff --git a/modules/networking/pages/serverless/aws/privatelink-api.adoc b/modules/networking/pages/serverless/aws/privatelink-api.adoc new file mode 100644 index 000000000..4a4e281b3 --- /dev/null +++ b/modules/networking/pages/serverless/aws/privatelink-api.adoc @@ -0,0 +1,276 @@ += Configure AWS PrivateLink with the Cloud API +:description: Set up AWS PrivateLink with the Cloud API for Serverless clusters. + +The Redpanda AWS PrivateLink endpoint service provides secure access to Redpanda Cloud from your own VPC. Traffic over PrivateLink does not go through the public internet because a PrivateLink connection is treated as its own private AWS service. While your VPC has access to the Redpanda VPC, Redpanda cannot access your VPC. + +Consider using the PrivateLink endpoint service if you have multiple VPCs and could benefit from a more simplified approach to network management. + +[NOTE] +==== +* Each client VPC can have one endpoint connected to the PrivateLink service. +* PrivateLink allows overlapping xref:networking:cidr-ranges.adoc[CIDR ranges] in VPC networks. +* PrivateLink does not add extra connection limits. However, VPC peering is limited to 125 connections. See https://aws.amazon.com/privatelink/faqs/[How scalable is AWS PrivateLink?^] +* You control which AWS principals are allowed to connect to the endpoint service. +==== + +After <>, you can <>, or you can <>. + +== Requirements + +* Install `rpk`. +* Your Redpanda Serverless cluster and <> must be in the same region. +* This guide uses the link:/api/doc/cloud-controlplane/topic/topic-cloud-api-overview[Redpanda Cloud API] to enable the Redpanda endpoint service for your Serverless clusters. Follow the steps below to <>. +* Use the https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html[AWS CLI^] to create a new client VPC or modify an existing one to use the PrivateLink endpoint. + +TIP: In Kafka clients, set `connections.max.idle.ms` to a value less than 350 seconds. + +NOTE: Enabling PrivateLink changes private DNS behavior for your cluster. Before configuring connections, review <>. + +== Get a Cloud API access token + +include::networking:partial$private-links-api-access-token.adoc[] + +== Create new cluster with PrivateLink endpoint service enabled + +. In the https://cloud.redpanda.com/[Redpanda Cloud Console^], go to **Resource groups** and select the resource group in which you want to create a cluster. ++ +Copy and store the resource group ID (UUID) from the URL in the browser. ++ +[,bash] +---- +export RESOURCE_GROUP_ID= +---- + +. Create a new Serverless cluster with the endpoint service enabled by calling link:/api/doc/cloud-controlplane/operation/operation-serverlessclusterservice_createserverlesscluster[`POST /v1/serverless-clusters`]. ++ +In the example below, make sure to set your own values for the following fields: ++ +-- +- `name` +- `serverless_region`: for example, `"pro-us-east-1"` +- `private_link_id`: The ID of an existing PrivateLink resource in the same resource group +- `networking_config.private` and `networking_config.public`: Valid values are `STATE_ENABLED` or `STATE_DISABLED`. At least one must be enabled. If neither is specified, `public` defaults to `STATE_ENABLED`. +-- ++ +[,bash] +---- +SERVERLESS_REGION= +SERVERLESS_PRIVATE_LINK_ID= + +CLUSTER_POST_BODY=`cat << EOF +{ + "serverless_cluster": { + "name": "", + "resource_group_id": "$RESOURCE_GROUP_ID", + "serverless_region": "$SERVERLESS_REGION", + "private_link_id": "$SERVERLESS_PRIVATE_LINK_ID", + "networking_config": { + "private": "STATE_ENABLED", + "public": "STATE_ENABLED" + } + } +} +EOF` + +CLUSTER_ID=`curl -vv -X POST \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $AUTH_TOKEN" \ + -d "$CLUSTER_POST_BODY" $PUBLIC_API_ENDPOINT/v1/serverless/clusters | jq -r .operation.metadata.cluster_id` + +echo $CLUSTER_ID +---- + +== Enable PrivateLink endpoint service for existing clusters + + +. In the Redpanda Cloud Console, go to the cluster overview and copy the cluster ID from the **Details** section. ++ +[,bash] +---- +CLUSTER_ID= +---- + +. Make a link:/api/doc/cloud-controlplane/operation/operation-serverlessclusterservice_updateserverlesscluster[`PATCH /v1/serverless-clusters/{cluster.id}`] request to update the cluster with the Redpanda PrivateLink Endpoint Service enabled. ++ +In the example below, make sure to set your own value for the following fields: ++ +-- +- `private_link_id`: The ID of an existing PrivateLink resource in the same resource group +- `networking_config.private`: Set to `STATE_ENABLED` to enable private access +-- ++ +[,bash] +---- +CLUSTER_PATCH_BODY=`cat << EOF +{ + "networking_config": { + "private": "STATE_ENABLED" + }, + "private_link_id": "$SERVERLESS_PRIVATE_LINK_ID" +} +EOF` + +curl -vv -X PATCH \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $AUTH_TOKEN" \ + -d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1/serverless/clusters/$CLUSTER_ID +---- + +== DNS resolution with PrivateLink + +PrivateLink changes how DNS resolution works for your cluster. When you query cluster hostnames outside the VPC that contains your PrivateLink endpoint, DNS may return private IP addresses that aren't reachable from your location. + +To resolve cluster hostnames from other VPCs or on-premise networks, set up DNS forwarding using Route 53 Resolver: + +. In the VPC that contains your PrivateLink endpoint, create a Route 53 Resolver inbound endpoint. ++ +Ensure that the inbound endpoint's security group allows inbound UDP/TCP port 53 from each VPC or on-prem network that will forward queries. + +. In each other VPC that must resolve the cluster domain, create a Resolver outbound endpoint and a forwarding rule for `` that targets the inbound endpoint IPs from the previous step. Associate the rule to those VPCs. ++ +The cluster domain is the suffix after the seed hostname. For example, if your bootstrap server URL is: cki01qgth38kk81ard3g.any.us-east-1.aw.priv.prd.cloud.redpanda.com:9092, then `cluster_domain` is: `cki01qgth38kk81ard3g.any.us-east-1.aw.priv.prd.cloud.redpanda.com`. +. For on-premises DNS, create a conditional forwarder for `` that forwards to the inbound endpoint IPs from the earlier step (over VPN/Direct Connect). ++ +[IMPORTANT] +==== +Do not configure forwarding rules to target the VPC's Amazon-provided DNS resolver (VPC base CIDR + 2). Rules must target the IP addresses of Route 53 Resolver endpoints. +==== + +== Configure PrivateLink connection to Redpanda Cloud + +When you have a PrivateLink-enabled cluster, you can create an endpoint to connect your VPC and your cluster. + +=== Get cluster domain + +Get the domain (`cluster_domain`) of the cluster from the cluster details in the Redpanda Cloud Console. + +For example, if the bootstrap server URL is: `cki01qgth38kk81ard3g.any.us-east-1.aw.priv.prd.cloud.redpanda.com:9092`, then `cluster_domain` is: `cki01qgth38kk81ard3g.any.us-east-1.aw.priv.prd.cloud.redpanda.com`. + +[,bash] +---- +CLUSTER_DOMAIN= +---- + +NOTE: Use `` as the domain you target with your DNS conditional forward (optionally also `*.` if your DNS platform requires a wildcard). + +=== Get name of PrivateLink endpoint service + +The service name is required to <>. Run the following command to get the service name: + +[,bash] +---- +PL_SERVICE_NAME=`curl -X GET \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $AUTH_TOKEN" \ + $PUBLIC_API_ENDPOINT/v1/serverless/private-links/$SERVERLESS_PRIVATE_LINK_ID | jq -r .serverless_private_link.status.aws.vpc_endpoint_service_name` +---- + +=== Create client VPC + +If you are not using an existing VPC, you must create a new one. + +The VPC region must be the same region where the Redpanda cluster is deployed. To create the VPC, run: + +[,bash] +---- +# See https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html for +# information on profiles and credential files +PROFILE= + +aws ec2 create-vpc --region $REGION --profile $PROFILE --cidr-block 10.0.0.0/20 + +# Store the client VPC ID from the command output +CLIENT_VPC_ID= +---- + +You can also use an existing VPC. You need the VPC ID to <>. + +=== Modify VPC DNS attributes + +To modify the VPC attributes, run: + +[,bash] +---- +aws ec2 modify-vpc-attribute --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \ + --enable-dns-hostnames "{\"Value\":true}" + +aws ec2 modify-vpc-attribute --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \ + --enable-dns-support "{\"Value\":true}" +---- + +These commands enable DNS hostnames and resolution for instances in the VPC. + +=== Create security group + +You need the security group ID `security_group_id` from the command output to <>. To create a security group, run: + +[,bash] +---- +aws ec2 create-security-group --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \ + --description "Redpanda endpoint service client security group" \ + --group-name "${CLUSTER_ID}-sg" +SECURITY_GROUP_ID= +---- + +=== Add security group rules + +The following example adds security group rules to allow access to Redpanda services. + +[,bash] +---- +# Allow Kafka API bootstrap (seed) +aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \ + --group-id $SECURITY_GROUP_ID --protocol tcp --port 9092 --cidr 0.0.0.0/0 + +# Allow Schema Registry +aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \ + --group-id $SECURITY_GROUP_ID --protocol tcp --port 8081 --cidr 0.0.0.0/0 + + +# Allow Redpanda Cloud Data Plane API / Prometheus (if needed) +aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \ + --group-id $SECURITY_GROUP_ID --protocol tcp --port 443 --cidr 0.0.0.0/0 +---- + +=== Create VPC subnet + +You need the subnet ID `subnet_id` from the command output to <>. Run the following command, specifying the subnet availability zone (for example, `usw2-az1`): + +[,bash] +---- +aws ec2 create-subnet --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \ + --availability-zone \ + --cidr-block 10.0.1.0/24 +SUBNET_ID= +---- + +=== Create VPC endpoint + +[,bash] +---- +aws ec2 create-vpc-endpoint \ + --region $REGION --profile $PROFILE \ + --vpc-id $CLIENT_VPC_ID \ + --vpc-endpoint-type "Interface" \ + --ip-address-type "ipv4" \ + --service-name $PL_SERVICE_NAME \ + --subnet-ids $SUBNET_ID \ + --security-group-ids $SECURITY_GROUP_ID \ + --private-dns-enabled +---- + +== Access Redpanda services through VPC endpoint + +After you have enabled PrivateLink for your cluster, your connection URLs are available in the *How to Connect* section of the cluster overview in the Redpanda Cloud Console. + +include::networking:partial$private-links-access-rp-service-serverless.adoc[] + +== Test the connection + +You can test the PrivateLink connection from any VM or container in the consumer VPC. If configuring a client isn't possible right away, you can do these checks using `rpk` or cURL: + +include::networking:partial$private-links-test-connection-serverless.adoc[] + +include::shared:partial$suggested-reading.adoc[] + +* link:/api/doc/cloud-controlplane/topic/topic-cloud-api-overview[Cloud API Overview] diff --git a/modules/networking/pages/serverless/aws/privatelink-ui.adoc b/modules/networking/pages/serverless/aws/privatelink-ui.adoc new file mode 100644 index 000000000..7b6dfe31a --- /dev/null +++ b/modules/networking/pages/serverless/aws/privatelink-ui.adoc @@ -0,0 +1,72 @@ += Configure AWS PrivateLink in the Cloud Console +:description: Set up AWS PrivateLink in the Redpanda Cloud Console for Serverless clusters. + +The Redpanda AWS PrivateLink endpoint service provides secure access to Redpanda Cloud from your own VPC. Traffic over PrivateLink does not go through the public internet because these connections are treated as their own private AWS service. While your VPC has access to the Redpanda VPC, Redpanda cannot access your VPC. + +Consider using the endpoint service if you have multiple VPCs and could benefit from a more simplified approach to network management. + +Existing serverless clusters can have their network modified by either the Console or the API. + +[NOTE] +==== +* Each client VPC can have one endpoint connected to the PrivateLink service. +* PrivateLink allows overlapping xref:networking:cidr-ranges.adoc[CIDR ranges] in VPC networks. +* PrivateLink does not add extra connection limits. However, VPC peering is limited to 125 connections. See https://aws.amazon.com/privatelink/faqs/[How scalable is AWS PrivateLink?^] +* You control which AWS principals are allowed to connect to the endpoint service. +==== + +== Requirements + +* Your Redpanda Serverless cluster and VPC must be in the same region. +* Use the https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html[AWS CLI] to create a new client VPC or modify an existing one to use the PrivateLink endpoint. + +TIP: In Kafka clients, set `connections.max.idle.ms` to a value less than 350 seconds. + +== DNS resolution with PrivateLink + +PrivateLink changes how DNS resolution works for your cluster. When you query cluster hostnames outside the VPC that contains your PrivateLink endpoint, DNS may return private IP addresses that aren't reachable from your location. + +To resolve cluster hostnames from other VPCs or on-premise networks, set up DNS forwarding using Route 53 Resolver: + +. In the VPC that contains your PrivateLink endpoint, create a Route 53 Resolver inbound endpoint. ++ +Ensure that the inbound endpoint's security group allows inbound UDP/TCP port 53 from each VPC or on-prem network that will forward queries. + +. In each other VPC that must resolve the cluster domain, create a Resolver outbound endpoint and a forwarding rule for `` that targets the inbound endpoint IPs from the previous step. Associate the rule to those VPCs. ++ +The cluster domain is the suffix after the seed hostname. For example, if your bootstrap server URL is: cki01qgth38kk81ard3g.any.us-east-1.aw.priv.prd.cloud.redpanda.com:9092, then `cluster_domain` is: `cki01qgth38kk81ard3g.any.us-east-1.aw.priv.prd.cloud.redpanda.com`. +. For on-premises DNS, create a conditional forwarder for `` that forwards to the inbound endpoint IPs from the earlier step (over VPN/Direct Connect). ++ +[IMPORTANT] +==== +Do not configure forwarding rules to target the VPC's Amazon-provided DNS resolver (VPC base CIDR + 2). Rules must target the IP addresses of Route 53 Resolver endpoints. +==== + +== Enable endpoint service for existing clusters + +. In the Redpanda Cloud Console, select your https://cloud.redpanda.com/clusters[cluster^], and go to the *Cluster settings* page. +. Under Networking, select **Private Access** and then select an existing private link. + +NOTE: For help with issues when enabling PrivateLink, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda support^]. + +== Access Redpanda services through VPC endpoint + +After you have enabled PrivateLink for your cluster, your connection URLs are available in the *How to Connect* section of the cluster overview in the Redpanda Cloud Console. + +include::networking:partial$private-links-access-rp-service-serverless.adoc[] + +== Test the connection + +You can test the connection to the endpoint service from any VM or container in the consumer VPC. If configuring a client isn't possible right away, you can do these checks using `rpk` or cURL: + +include::networking:partial$private-links-test-connection-serverless.adoc[] + +== Disable endpoint service + +On the Cluster Settings page, deselect **Private Access**. Existing connections are closed after the AWS PrivateLink service is disabled. + +NOTE: Disabling private access in Redpanda Cloud does not delete the PrivateLink endpoint in your AWS account or the PrivateLink resource in Redpanda Cloud. Both remain provisioned and continue to incur charges until you explicitly delete them. + +include::shared:partial$suggested-reading.adoc[] + +* xref:networking:serverless/aws/privatelink-api.adoc[] diff --git a/modules/networking/pages/serverless/index.adoc b/modules/networking/pages/serverless/index.adoc new file mode 100644 index 000000000..8c63441c2 --- /dev/null +++ b/modules/networking/pages/serverless/index.adoc @@ -0,0 +1,3 @@ += Networking: Serverless +:description: Learn how to configure private networking with AWS PrivateLink. +:page-layout: index \ No newline at end of file diff --git a/modules/networking/partials/private-links-access-rp-service-serverless.adoc b/modules/networking/partials/private-links-access-rp-service-serverless.adoc new file mode 100644 index 000000000..e77bd0062 --- /dev/null +++ b/modules/networking/partials/private-links-access-rp-service-serverless.adoc @@ -0,0 +1,43 @@ +You can access Redpanda services such as the Kafka API and Schema Registry from the client VPC or virtual network; for example, from a compute instance in the VPC or network. + +The bootstrap server hostname is unique to each cluster. The service attachment exposes a set of bootstrap ports for access to Redpanda services. These ports load balance requests among brokers. Make sure you use the following ports for initiating a connection from a consumer: + +|=== +| Redpanda service | Default bootstrap port + +| Kafka API | 9092 +| Schema Registry | 8081 +|=== + +=== Access Kafka API seed service + +Use port `9092` to access the Kafka API seed service. + +[,bash] +---- +export RPK_BROKERS=':9092' +rpk cluster info -X tls.enabled=true -X user= -X pass= +---- + +When successful, the `rpk` output should look like the following: + +[,bash,role=no-copy] +---- +CLUSTER +redpanda.rp-cki01qgth38kk81ard3g + +BROKERS +ID HOST PORT RACK +0* 0-cki01qgth38kk81ard3g.any.us-east-1.aw.priv.prd.cloud.redpanda.com 9092 use1-az1 +1 1-cki01qgth38kk81ard3g.any.us-east-1.aw.priv.prd.cloud.redpanda.com 9092 use1-az1 +2 2-cki01qgth38kk81ard3g.any.us-east-1.aw.priv.prd.cloud.redpanda.com 9092 use1-az1 +---- + +=== Access Schema Registry seed service + +Use port `8081` to access the Schema Registry seed service. + +[,bash] +---- +curl -vv -u : -H "Content-Type: application/vnd.schemaregistry.v1+json" --sslv2 --http2 :8081/subjects +---- diff --git a/modules/networking/partials/private-links-test-connection-serverless.adoc b/modules/networking/partials/private-links-test-connection-serverless.adoc new file mode 100644 index 000000000..e83ff58e6 --- /dev/null +++ b/modules/networking/partials/private-links-test-connection-serverless.adoc @@ -0,0 +1,31 @@ +. Set the following environment variables. ++ +[,bash] +---- +export RPK_BROKERS=':9092' +export RPK_TLS_ENABLED=true +export RPK_SASL_MECHANISM="" +export RPK_USER= +export RPK_PASS= +---- + +. Create a test topic. ++ +[,bash] +---- +rpk topic create test-topic +---- + +. Produce to the test topic. ++ +[,bash] +---- +echo 'hello world' | rpk topic produce test-topic +---- + +. Consume from the test topic. ++ +[,bash] +---- +rpk topic consume test-topic -n 1 +---- diff --git a/modules/networking/partials/vnet-peering.adoc b/modules/networking/partials/vnet-peering.adoc index e190edffc..6ffce0fde 100644 --- a/modules/networking/partials/vnet-peering.adoc +++ b/modules/networking/partials/vnet-peering.adoc @@ -27,7 +27,7 @@ To create a peering connection between your Azure VNet and Redpanda VPC: . In the Redpanda Cloud UI, go to the *Overview* page for your cluster. . In the Details section, click the name of the *Redpanda network*. -. On the Networks page for your cluster, click *VPC peering walkthrough*. +. On the Networking page for your cluster, click *VPC peering walkthrough*. . For *Connection name*, enter a name. For example, the name could refer to your Azure VNet ID. . For *Azure account number*, enter the account number associated with the VNet you want to connect to. . For *Azure VNet ID*, enter the VNet ID.