2540 - Additional conditioning and single sourcing for Cloud RPCN lau…

…nch (#35)

Co-authored-by: JakeSCahill <[email protected]>

local-antora-playbook.yml

@@ -7,7 +7,6 @@ urls:
   html_extension_style: indexify
 content:
   sources:
-  # Fetches autogenerated docs.
   - url: .
     branches: HEAD
   - url: https://github.com/redpanda-data/documentation
@@ -32,8 +31,8 @@ asciidoc:
   - '@redpanda-data/docs-extensions-and-macros/asciidoc-extensions/add-line-numbers-highlights'
 antora:
   extensions:
-  - require: '@redpanda-data/docs-extensions-and-macros/extensions/generate-rp-connect-info'
   - require: '@redpanda-data/docs-extensions-and-macros/extensions/generate-rp-connect-categories'
+  - require: '@redpanda-data/docs-extensions-and-macros/extensions/generate-rp-connect-info'
   - require: '@redpanda-data/docs-extensions-and-macros/extensions/unpublish-pages'
   - require: '@redpanda-data/docs-extensions-and-macros/extensions/unlisted-pages'
   - require: '@redpanda-data/docs-extensions-and-macros/extensions/add-global-attributes'

modules/components/pages/about.adoc

@@ -33,7 +33,12 @@ These are the main components within Redpanda Connect and they provide the major
 
 == Observability components
 
-There are also the observability components xref:components:http/about.adoc[http], xref:components:logger/about.adoc[logger], xref:components:metrics/about.adoc[metrics], and xref:components:tracers/about.adoc[tracing], which allow you to specify how Redpanda Connect exposes observability data:
+ifndef::env-cloud[]
+There are also the observability components: xref:components:http/about.adoc[http], xref:components:logger/about.adoc[logger], xref:components:metrics/about.adoc[metrics], and xref:components:tracers/about.adoc[tracing], which allow you to specify how Redpanda Connect exposes observability data.
+endif::[]
+ifdef::env-cloud[]
+There are also the observability components: xref:components:logger/about.adoc[logger], xref:components:metrics/about.adoc[metrics], and xref:components:tracers/about.adoc[tracing], which allow you to specify how Redpanda Connect exposes observability data.
+endif::[]
 
 [source,yaml]
 ----

modules/components/pages/caches/about.adoc

@@ -32,6 +32,8 @@ pipeline:
 
 For the simple case where you wish to store messages in a cache as an output destination for your pipeline check out the xref:components:outputs/cache.adoc[`cache` output]. To see examples of more advanced uses of caches such as hydration and deduplication check out the xref:components:processors/cache.adoc[`cache` processor].
 
+ifndef::env-cloud[]
 You can find out more about resources xref:configuration:resources.adoc[in this document.]
+endif::[]
 
 // end::single-source[]

modules/components/pages/caches/gcp_cloud_storage.adoc

@@ -1,4 +1,5 @@
 = gcp_cloud_storage
+// tag::single-source[]
 :type: cache
 :status: beta
 
@@ -37,5 +38,4 @@ Optional field to explicitly set the Content-Type.
 
 *Type*: `string`
 
-
-
+// end::single-source[]

modules/components/pages/inputs/about.adoc

@@ -24,7 +24,12 @@ input:
        root.link_count = this.links.length()
 ----
 
+ifndef::env-cloud[]
 Some inputs have a logical end, for example a xref:components:inputs/csv.adoc[`csv` input] ends once the last row is consumed, when this happens the input gracefully terminates and Redpanda Connect will shut itself down once all messages have been processed fully.
+endif::[]
+ifdef::env-cloud[]
+Some inputs have a logical end, when this happens the input gracefully terminates and Redpanda Connect will shut itself down once all messages have been processed fully.
+endif::[]
 
 It's also possible to specify a logical end for an input that otherwise doesn't have one with the xref:components:inputs/read_until.adoc[`read_until` input], which checks a condition against each consumed message in order to determine whether it should be the last.
 

modules/components/pages/inputs/azure_blob_storage.adoc

@@ -1,4 +1,5 @@
 = azure_blob_storage
+// tag::single-source[]
 :type: input
 :status: beta
 :categories: ["Services","Azure"]
@@ -11,8 +12,9 @@ component_type_dropdown::[]
 
 Downloads objects within an Azure Blob Storage container, optionally filtered by a prefix.
 
+ifndef::env-cloud[]
 Introduced in version 3.36.0.
-
+endif::[]
 
 [tabs]
 ======
@@ -158,7 +160,10 @@ The xref:components:scanners/about.adoc[scanner] by which the stream of bytes co
 *Type*: `scanner`
 
 *Default*: `{"to_the_end":{}}`
+
+ifndef::env-cloud[]
 Requires version 4.25.0 or newer
+endif::[]
 
 === `delete_objects`
 
@@ -180,7 +185,9 @@ This requires setting up https://learn.microsoft.com/en-gb/azure/event-grid/even
 
 *Type*: `input`
 
+ifndef::env-cloud[]
 Requires version 4.27.0 or newer
+endif::[]
 
 [tabs]
 ======
@@ -239,4 +246,7 @@ targets_input:
 ```
 
 --
-======
+======
+
+// end::single-source[]
+

modules/components/pages/inputs/azure_cosmosdb.adoc

@@ -1,4 +1,5 @@
 = azure_cosmosdb
+// tag::single-source[]
 :type: input
 :status: experimental
 :categories: ["Azure"]
@@ -11,8 +12,9 @@ component_type_dropdown::[]
 
 Executes a SQL query against https://learn.microsoft.com/en-us/azure/cosmos-db/introduction[Azure CosmosDB^] and creates a batch of messages from each page of items.
 
+ifndef::env-cloud[]
 Introduced in version v4.25.0.
-
+endif::[]
 
 [tabs]
 ======
@@ -288,4 +290,4 @@ Additionally, instead of installing the container self-signed certificate which
 
 Then you can access the CosmosDB UI via `http://localhost:8080/_explorer/index.html` and use `http://localhost:8080` as the CosmosDB endpoint.
 
-
+// end::single-source[]

modules/components/pages/inputs/azure_queue_storage.adoc

@@ -1,4 +1,5 @@
 = azure_queue_storage
+// tag::single-source[]
 :type: input
 :status: beta
 :categories: ["Services","Azure"]
@@ -11,8 +12,9 @@ component_type_dropdown::[]
 
 Dequeue objects from an Azure Storage Queue.
 
+ifndef::env-cloud[]
 Introduced in version 3.42.0.
-
+endif::[]
 
 [tabs]
 ======
@@ -118,7 +120,10 @@ The timeout duration until a dequeued message gets visible again, 30s by default
 *Type*: `string`
 
 *Default*: `"30s"`
+
+ifndef::env-cloud[]
 Requires version 3.45.0 or newer
+endif::[]
 
 === `max_in_flight`
 
@@ -138,4 +143,4 @@ If set to `true` the queue is polled on each read request for information such a
 
 *Default*: `false`
 
-
+// end::single-source[]

modules/components/pages/inputs/azure_table_storage.adoc

@@ -1,4 +1,5 @@
 = azure_table_storage
+// tag::single-source[]
 :type: input
 :status: beta
 :categories: ["Services","Azure"]
@@ -11,8 +12,9 @@ component_type_dropdown::[]
 
 Queries an Azure Storage Account Table, optionally with multiple filters.
 
+ifndef::env-cloud[]
 Introduced in version 4.10.0.
-
+endif::[]
 
 [tabs]
 ======
@@ -56,7 +58,9 @@ input:
 ======
 
 Queries an Azure Storage Account Table, optionally with multiple filters.
+
 == Metadata
+
 This input adds the following metadata fields to each message:
 
 - table_storage_name
@@ -155,4 +159,4 @@ Maximum number of records to return on each page.
 
 *Default*: `1000`
 
-
+// end::single-source[]

modules/components/pages/inputs/gcp_bigquery_select.adoc

@@ -1,4 +1,5 @@
 = gcp_bigquery_select
+// tag::single-source[]
 :type: input
 :status: beta
 :categories: ["Services","GCP"]
@@ -12,7 +13,9 @@ component_type_dropdown::[]
 
 Executes a `SELECT` query against BigQuery and creates a message for each row received.
 
+ifndef::env-cloud[]
 Introduced in version 3.63.0.
+endif::[]
 
 ```yml
 # Config fields, showing default values
@@ -168,5 +171,4 @@ An optional suffix to append to the select query.
 
 *Type*: `string`
 
-
-
+// end::single-source[]

modules/components/pages/inputs/gcp_cloud_storage.adoc

@@ -1,4 +1,5 @@
 = gcp_cloud_storage
+// tag::single-source[]
 :type: input
 :status: beta
 :categories: ["Services","GCP"]
@@ -11,8 +12,9 @@ component_type_dropdown::[]
 
 Downloads objects within a Google Cloud Storage bucket, optionally filtered by a prefix.
 
+ifndef::env-cloud[]
 Introduced in version 3.43.0.
-
+endif::[]
 
 [tabs]
 ======
@@ -98,7 +100,10 @@ The xref:components:scanners/about.adoc[scanner] by which the stream of bytes co
 *Type*: `scanner`
 
 *Default*: `{"to_the_end":{}}`
+
+ifndef::env-cloud[]
 Requires version 4.25.0 or newer
+endif::[]
 
 === `delete_objects`
 
@@ -109,4 +114,4 @@ Whether to delete downloaded objects from the bucket once they are processed.
 
 *Default*: `false`
 
-
+// end::single-source[]

modules/components/pages/inputs/gcp_pubsub.adoc

@@ -1,4 +1,5 @@
 = gcp_pubsub
+// tag::single-source[]
 :type: input
 :status: stable
 :categories: ["Services","GCP"]
@@ -156,4 +157,4 @@ Defines the topic that the subscription should be vinculated to.
 
 *Default*: `""`
 
-
+// end::single-source[]

modules/components/pages/inputs/inproc.adoc

@@ -18,8 +18,15 @@ input:
   inproc: ""
 ```
 
+ifndef::env-cloud[]
 Directly connect to an output within a Redpanda Connect process by referencing it by a chosen ID. This allows you to hook up isolated streams whilst running Redpanda Connect in xref:guides:streams_mode/about.adoc[streams mode], it is NOT recommended that you connect the inputs of a stream with an output of the same stream, as feedback loops can lead to deadlocks in your message flow.
 
 It is possible to connect multiple inputs to the same inproc ID, resulting in messages dispatching in a round-robin fashion to connected inputs. However, only one output can assume an inproc ID, and will replace existing outputs if a collision occurs.
+endif::[]
+ifdef::env-cloud[]
+Directly connect to an output within a Redpanda Connect process by referencing it by a chosen ID.
+
+It is possible to connect multiple inputs to the same inproc ID, resulting in messages dispatching in a round-robin fashion to connected inputs. However, only one output can assume an inproc ID, and will replace existing outputs if a collision occurs.
+endif::[]
 
 // end::single-source[]

modules/components/pages/inputs/resource.adoc

@@ -57,6 +57,8 @@ input_resources:
 
 Resources also allow you to reference a single input in multiple places, such as multiple streams mode configs, or multiple entries in a broker input. However, when a resource is referenced more than once the messages it produces are distributed across those references, so each message will only be directed to a single reference, not all of them.
 
+ifndef::env-cloud[]
 You can find out more about resources in xref:configuration:resources.adoc[].
+endif::[]
 
 // end::single-source[]

modules/components/pages/metrics/about.adoc

@@ -11,7 +11,12 @@ metrics:
     flush_period: 100ms
 ----
 
+ifndef::env-cloud[]
 The default metrics configuration is to expose Prometheus metrics on the xref:components:http/about.adoc[service-wide HTTP endpoint] at the endpoints `/metrics` and `/stats`.
+endif::[]
+ifdef::env-cloud[]
+Redpanda Connect automatically xref:develop:connect/configuration/monitor-connect.adoc[exports detailed metrics] for each component of your data pipeline to a Prometheus endpoint.
+endif::[]
 
 == Timings
 
@@ -101,7 +106,12 @@ The `label` label contains the unique label configured for a component emitting
 
 === `stream`
 
+ifndef::env-cloud[]
 The `stream` label is present in a metric series emitted from a stream config executed when Redpanda Connect is running in xref:guides:streams_mode/about.adoc[streams mode], and is populated with the stream name.
+endif::[]
+ifdef::env-cloud[]
+The `stream` label is present in a metric series emitted from a stream config executed when Redpanda Connect is running in streams mode, and is populated with the stream name.
+endif::[]
 
 == Example
 

modules/components/pages/outputs/azure_blob_storage.adoc

@@ -1,4 +1,5 @@
 = azure_blob_storage
+// tag::single-source[]
 :type: output
 :status: beta
 :categories: ["Services","Azure"]
@@ -11,8 +12,9 @@ component_type_dropdown::[]
 
 Sends message parts as objects to an Azure Blob Storage Account container. Each object is uploaded with the filename specified with the `container` field.
 
+ifndef::env-cloud[]
 Introduced in version 3.36.0.
-
+endif::[]
 
 [tabs]
 ======
@@ -191,4 +193,4 @@ The maximum number of messages to have in flight at a given time. Increase this
 
 *Default*: `64`
 
-
+// end::single-source[]

modules/components/pages/outputs/azure_cosmosdb.adoc

@@ -1,4 +1,5 @@
 = azure_cosmosdb
+// tag::single-source[]
 :type: output
 :status: experimental
 :categories: ["Azure"]
@@ -11,8 +12,9 @@ component_type_dropdown::[]
 
 Creates or updates messages as JSON documents in https://learn.microsoft.com/en-us/azure/cosmos-db/introduction[Azure CosmosDB^].
 
+ifndef::env-cloud[]
 Introduced in version v4.25.0.
-
+endif::[]
 
 [tabs]
 ======
@@ -517,4 +519,4 @@ Additionally, instead of installing the container self-signed certificate which
 
 Then you can access the CosmosDB UI via `http://localhost:8080/_explorer/index.html` and use `http://localhost:8080` as the CosmosDB endpoint.
 
-
+// end::single-source[]