2540 Autogenerated files moved to rp-connect-docs (#28)

Co-authored-by: JakeSCahill <[email protected]>

Author: asimms41

.gitignore

@@ -1,3 +1,4 @@
 node_modules
 .vscode
-docs
+docs
+*.DS_Store

local-antora-playbook.yml

@@ -8,13 +8,6 @@ urls:
 content:
   sources:
   # Fetches autogenerated docs.
-  - url: https://github.com/redpanda-data/connect
-    # The latest tag is fetched from GitHub during the build in:
-    # https://github.com/redpanda-data/docs-extensions-and-macros/blob/main/README.adoc#redpanda-connect-tag-modifier.
-    # This is a fallback version.
-    tags: ['v4.29.0']
-    branches: ~
-    start_paths: [docs]
   - url: .
     branches: HEAD
   - url: https://github.com/redpanda-data/documentation
@@ -40,7 +33,6 @@ asciidoc:
 antora:
   extensions:
   - require: '@redpanda-data/docs-extensions-and-macros/extensions/generate-rp-connect-categories'
-  - require: '@redpanda-data/docs-extensions-and-macros/extensions/modify-connect-tag-playbook'
   - require: '@redpanda-data/docs-extensions-and-macros/extensions/unpublish-pages'
   - require: '@redpanda-data/docs-extensions-and-macros/extensions/unlisted-pages'
   - require: '@redpanda-data/docs-extensions-and-macros/extensions/add-global-attributes'

modules/ROOT/nav.adoc

@@ -212,6 +212,7 @@
 *** xref:components:outputs/pinecone.adoc[]
 *** xref:components:outputs/pulsar.adoc[]
 *** xref:components:outputs/pusher.adoc[]
+*** xref:components:outputs/qdrant.adoc[]
 *** xref:components:outputs/redis_hash.adoc[]
 *** xref:components:outputs/redis_list.adoc[]
 *** xref:components:outputs/redis_pubsub.adoc[]

modules/components/pages/buffers/memory.adoc

@@ -0,0 +1,172 @@
+= memory
+:type: buffer
+:status: stable
+:categories: ["Utility"]
+
+// © 2024 Redpanda Data Inc.
+
+
+component_type_dropdown::[]
+
+
+Stores consumed messages in memory and acknowledges them at the input level. During shutdown Redpanda Connect will make a best attempt at flushing all remaining messages before exiting cleanly.
+
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Common config fields, showing default values
+buffer:
+  memory:
+    limit: 524288000
+    batch_policy:
+      enabled: false
+      count: 0
+      byte_size: 0
+      period: ""
+      check: ""
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All config fields, showing default values
+buffer:
+  memory:
+    limit: 524288000
+    batch_policy:
+      enabled: false
+      count: 0
+      byte_size: 0
+      period: ""
+      check: ""
+      processors: [] # No default (optional)
+```
+
+--
+======
+
+This buffer is appropriate when consuming messages from inputs that do not gracefully handle back pressure and where delivery guarantees aren't critical.
+
+This buffer has a configurable limit, where consumption will be stopped with back pressure upstream if the total size of messages in the buffer reaches this amount. Since this calculation is only an estimate, and the real size of messages in RAM is always higher, it is recommended to set the limit significantly below the amount of RAM available.
+
+== Delivery guarantees
+
+This buffer intentionally weakens the delivery guarantees of the pipeline and therefore should never be used in places where data loss is unacceptable.
+
+== Batching
+
+It is possible to batch up messages sent from this buffer using a xref:configuration:batching.adoc#batch-policy[batch policy].
+
+== Fields
+
+=== `limit`
+
+The maximum buffer size (in bytes) to allow before applying backpressure upstream.
+
+
+*Type*: `int`
+
+*Default*: `524288000`
+
+=== `batch_policy`
+
+Optionally configure a policy to flush buffered messages in batches.
+
+
+*Type*: `object`
+
+
+=== `batch_policy.enabled`
+
+Whether to batch messages as they are flushed.
+
+
+*Type*: `bool`
+
+*Default*: `false`
+
+=== `batch_policy.count`
+
+A number of messages at which the batch should be flushed. If `0` disables count based batching.
+
+
+*Type*: `int`
+
+*Default*: `0`
+
+=== `batch_policy.byte_size`
+
+An amount of bytes at which the batch should be flushed. If `0` disables size based batching.
+
+
+*Type*: `int`
+
+*Default*: `0`
+
+=== `batch_policy.period`
+
+A period in which an incomplete batch should be flushed regardless of its size.
+
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+period: 1s
+
+period: 1m
+
+period: 500ms
+```
+
+=== `batch_policy.check`
+
+A xref:guides:bloblang/about.adoc[Bloblang query] that should return a boolean value indicating whether a message should end a batch.
+
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+check: this.type == "end_of_transaction"
+```
+
+=== `batch_policy.processors`
+
+A list of xref:components:processors/about.adoc[processors] to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op.
+
+
+*Type*: `array`
+
+
+```yml
+# Examples
+
+processors:
+  - archive:
+      format: concatenate
+
+processors:
+  - archive:
+      format: lines
+
+processors:
+  - archive:
+      format: json_array
+```
+
+

modules/components/pages/buffers/none.adoc

@@ -0,0 +1,23 @@
+= none
+:type: buffer
+:status: stable
+
+// © 2024 Redpanda Data Inc.
+
+
+component_type_dropdown::[]
+
+
+Do not buffer messages. This is the default and most resilient configuration.
+
+```yml
+# Config fields, showing default values
+buffer:
+  none: {}
+```
+
+Selecting no buffer means the output layer is directly coupled with the input layer. This is the safest and lowest latency option since acknowledgements from at-least-once protocols can be propagated all the way from the output protocol to the input protocol.
+
+If the output layer is hit with back pressure it will propagate all the way to the input layer, and further up the data stream. If you need to relieve your pipeline of this back pressure consider using a more robust buffering solution such as Kafka before resorting to alternatives.
+
+

modules/components/pages/buffers/sqlite.adoc

@@ -0,0 +1,93 @@
+= sqlite
+:type: buffer
+:status: stable
+:categories: ["Utility"]
+
+// © 2024 Redpanda Data Inc.
+
+
+component_type_dropdown::[]
+
+
+Stores messages in an SQLite database and acknowledges them at the input level.
+
+```yml
+# Config fields, showing default values
+buffer:
+  sqlite:
+    path: "" # No default (required)
+    pre_processors: [] # No default (optional)
+    post_processors: [] # No default (optional)
+```
+
+Stored messages are then consumed as a stream from the database and deleted only once they are successfully sent at the output level. If the service is restarted Redpanda Connect will make a best attempt to finish delivering messages that are already read from the database, and when it starts again it will consume from the oldest message that has not yet been delivered.
+
+== Delivery guarantees
+
+Messages are not acknowledged at the input level until they have been added to the SQLite database, and they are not removed from the SQLite database until they have been successfully delivered. This means at-least-once delivery guarantees are preserved in cases where the service is shut down unexpectedly. However, since this process relies on interaction with the disk (wherever the SQLite DB is stored) these delivery guarantees are not resilient to disk corruption or loss.
+
+== Batching
+
+Messages that are logically batched at the point where they are added to the buffer will continue to be associated with that batch when they are consumed. This buffer is also more efficient when storing messages within batches, and therefore it is recommended to use batching at the input level in high-throughput use cases even if they are not required for processing.
+
+
+== Fields
+
+=== `path`
+
+The path of the database file, which will be created if it does not already exist.
+
+
+*Type*: `string`
+
+
+=== `pre_processors`
+
+An optional list of processors to apply to messages before they are stored within the buffer. These processors are useful for compressing, archiving or otherwise reducing the data in size before it's stored on disk.
+
+
+*Type*: `array`
+
+
+=== `post_processors`
+
+An optional list of processors to apply to messages after they are consumed from the buffer. These processors are useful for undoing any compression, archiving, etc that may have been done by your `pre_processors`.
+
+
+*Type*: `array`
+
+
+== Examples
+
+[tabs]
+======
+Batching for optimization::
++
+--
+
+Batching at the input level greatly increases the throughput of this buffer. If logical batches aren't needed for processing add a xref:components:processors/split.adoc[`split` processor] to the `post_processors`.
+
+```yaml
+input:
+  batched:
+    child:
+      sql_select:
+        driver: postgres
+        dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable
+        table: footable
+        columns: [ '*' ]
+    policy:
+      count: 100
+      period: 500ms
+
+buffer:
+  sqlite:
+    path: ./foo.db
+    post_processors:
+      - split: {}
+```
+
+--
+======
+
+