Skip to content

Commit

Permalink
Add more docs
Browse files Browse the repository at this point in the history
  • Loading branch information
edgarrmondragon committed Nov 27, 2024
1 parent cecdc07 commit a89a2bc
Show file tree
Hide file tree
Showing 2 changed files with 121 additions and 4 deletions.
24 changes: 20 additions & 4 deletions docs/builtin.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,26 @@ The Singer SDK library provides a number of built-in settings and capabilities.
.. autodata:: ADD_RECORD_METADATA
:no-value:

.. autoattribute:: ADD_RECORD_METADATA.schema

.. autodata:: BATCH
:no-value:

.. autoattribute:: BATCH.schema
.. autoattribute:: BATCH.capability
.. autodata:: FLATTENING
:no-value:

.. autodata:: STREAM_MAPS
:no-value:

.. autodata:: TARGET_BATCH_SIZE_ROWS
:no-value:

.. autodata:: TARGET_HARD_DELETE
:no-value:

.. autodata:: TARGET_LOAD_METHOD
:no-value:

.. autodata:: TARGET_SCHEMA
:no-value:

.. autodata:: TARGET_VALIDATE_RECORDS
:no-value:
101 changes: 101 additions & 0 deletions singer_sdk/helpers/capabilities/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,29 +45,130 @@

#: For taps, support emitting BATCH messages. For targets, support consuming BATCH
#: messages.
#:
#: Example:
#:
#: .. code-block:: json
#:
#: {
#: "batch_config": {
#: "encoding": {
#: "format": "jsonl",
#: "compression": "gzip"
#: },
#: "storage": {
#: "type": "root",
#: "root": "file:///path/to/batch/files",
#: "prefix": "batch-"
#: }
#: }
#: }
#:
BATCH = Builtin(
schema=schema.BATCH_CONFIG,
capability=PluginCapabilities.BATCH,
)

#: Support schema flattening, aka de-nesting of complex properties.
#:
#: Example:
#:
#: .. code-block:: json
#:
#: {
#: "flattening_enabled": true,
#: "flattening_max_depth": 3
#: }
#:
FLATTENING = Builtin(
schema=schema.FLATTENING_CONFIG,
capability=PluginCapabilities.FLATTENING,
)

#: Support inline stream map transforms.
#:
#: Example:
#:
#: .. code-block:: json
#:
#: {
#: "stream_maps": {
#: "users": {
#: "id": "id",
#: "fields": "[f for f in fields if f['key'] != 'age']"
#: }
#: }
#: }
#:
STREAM_MAPS = Builtin(
schema.STREAM_MAPS_CONFIG,
capability=PluginCapabilities.STREAM_MAPS,
)

#: Target batch size in rows.
#:
#: Example:
#:
#: .. code-block:: json
#:
#: {
#: "batch_size_rows": 10000
#: }
#:
TARGET_BATCH_SIZE_ROWS = Builtin(schema=schema.TARGET_BATCH_SIZE_ROWS_CONFIG)

#: Support hard delete capability.
#:
#: Example:
#:
#: .. code-block:: json
#:
#: {
#: "hard_delete": true
#: }
#:
TARGET_HARD_DELETE = Builtin(
schema=schema.TARGET_HARD_DELETE_CONFIG,
capability=TargetCapabilities.HARD_DELETE,
)

#: Target load method.
#:
#: Example:
#:
#: .. code-block:: json
#:
#: {
#: "load_method": "upsert"
#: }
#:
TARGET_LOAD_METHOD = Builtin(schema=schema.TARGET_LOAD_METHOD_CONFIG)

#: Allow setting the target schema.
#:
#: Example:
#:
#: .. code-block:: json
#:
#: {
#: "default_target_schema": "my_schema"
#: }
#:
TARGET_SCHEMA = Builtin(
schema=schema.TARGET_SCHEMA_CONFIG,
capability=TargetCapabilities.TARGET_SCHEMA,
)

#: Validate incoming records against their declared schema.
#:
#: Example:
#:
#: .. code-block:: json
#:
#: {
#: "validate_records": true
#: }
#:
TARGET_VALIDATE_RECORDS = Builtin(
schema=schema.TARGET_VALIDATE_RECORDS_CONFIG,
capability=TargetCapabilities.VALIDATE_RECORDS,
Expand Down

0 comments on commit a89a2bc

Please sign in to comment.