Add Plugin: tap-planetscale

[visch]

What is the plugin type?

extractor

Repository URL

https://github.com/planetscale/singer-tap/

Extension Repository URL

No response

Description

PlanetScale is the world’s most advanced MySQL platform

Domain URL

https://planetscale.com/docs

Logo Image

Any other notes?

This tap is written in Go.

Note that with some settings set automatically MeltanoLabs tap-mysql tap can support this as well (After MeltanoLabs/tap-mysql#45 is merged)

[visch]

I hit an issue with their tap (I was trying to write Meltano instructions to place in the planetscale readme) see https://github.com/planetscale/singer-tap/issues/51

[edgarrmondragon]

The problem with having this in the Hub would be that meltano add wouldn't be sufficient to set up the tap: users would still need to download or build the right binary, and then point executable to it. I don't know if we have another non-Python tap in the Hub.

[tayloramurphy]

@edgarrmondragon I have a memory of another non-python tap, but I don't think we added it to the Hub. Maybe @pnadolny13 remembers? It may have been this mysql one to be honest.

[edgarrmondragon]

There's https://github.com/MeltanoLabs/target-jsonl-blob which is also Golang-based, and we never added it to the Hub.

[pnadolny13]

@edgarrmondragon @tayloramurphy yeah I dont know of any that exist. I wonder if its helpful to still list it but exclude it from the API so its not accessible by meltano automatically. The hub page can explain how to install it manually if needed.

I'd have to look into this but we could also list it as a docker airbyte variant https://hub.docker.com/r/planetscale/airbyte-source/tags.

[visch]

We could probably do something better than this but here's a "working" example. Note that their tap doesn't follow the singer spec so I don't think we can add this anyway :/ See https://github.com/planetscale/singer-tap/issues/51 maybe they'll adjust it though!

  extractors:
  - name: tap-planetscale
    namespace: tap_planetscale
    pip_url: cowsay # I don't think we need to do this, could just have no pip url as you don't have to run install?
    executable: /home/visch/git/tap-mysql/ps-singer-tap
    capabilities:
    - state
    - catalog
    - discover
    - about
    - stream-maps
    settings:
    - name: host
    - name: username
    - name: database
    - name: password
      sensitive: true
    config:
      host: aws.connect.psdb.cloud
      username: 0fiqne6txvcqtjbdywan
      database: tap-mysql
    select:
    - "*.*"

[visch]

I played around a bit with the latest version of the planet scale tap (they fixed the catalog output)

I can't get meltano to properly select the fields I have, or add the proper metadata. If I manually add a select, and update the replication-method to FULL_TABLE the tap works as advertised.

Not sure exactly what's going on here, my best guess is the stream_id having a : in the name (right now it's tap-mysql:testtable ) might be throwing off something maybe a regex strips them internally.

visch@DESKTOP-9BDPA9T:~/git/tap-mysql$ meltano --log-level=debug invoke --dump=catalog tap-planetscale
2024-02-22T20:03:56.095140Z [debug    ] meltano 3.3.0, Linux
2024-02-22T20:03:56.098361Z [debug    ] /etc/timezone found, contents:
 America/New_York

2024-02-22T20:03:56.098817Z [debug    ] /etc/localtime found
2024-02-22T20:03:56.099580Z [debug    ] 2 found:
 {'/etc/timezone': 'America/New_York', '/etc/localtime is a symlink to': 'America/New_York'}
2024-02-22T20:03:56.101959Z [info     ] Environment 'test' is active
2024-02-22T20:03:56.119210Z [debug    ] Creating DB engine for project at '/home/visch/git/tap-mysql' with DB URI 'sqlite://home/visch/git/tap-mysql/.meltano/meltano.db'
2024-02-22T20:03:56.243347Z [debug    ] Created configuration at /home/visch/git/tap-mysql/.meltano/run/tap-planetscale/tap.b204835c-78f5-4ac4-a5af-d6f5a295f714.config.json
2024-02-22T20:03:56.243643Z [debug    ] Could not find tap.properties.json in /home/visch/git/tap-mysql/.meltano/extractors/tap-planetscale/tap.properties.json, skipping.
2024-02-22T20:03:56.243844Z [debug    ] Could not find tap.properties.cache_key in /home/visch/git/tap-mysql/.meltano/extractors/tap-planetscale/tap.properties.cache_key, skipping.
2024-02-22T20:03:56.244042Z [debug    ] Could not find state.json in /home/visch/git/tap-mysql/.meltano/extractors/tap-planetscale/state.json, skipping.
2024-02-22T20:03:56.245698Z [debug    ] Invoking: ['/home/visch/git/tap-mysql/ps-singer-tap', '--config', '/home/visch/git/tap-mysql/.meltano/run/tap-planetscale/tap.b204835c-78f5-4ac4-a5af-d6f5a295f714.config.json', '--discover']
PlanetScale Tap : INFO : PlanetScale Singer Tap : version ["0.27.0"], commit ["29d23da"], published on ["2024-02-21T19:10:50Z"]
PlanetScale Tap : INFO : running in discovery mode
PlanetScale Tap : INFO : Discovering Schema for PlanetScale database : tap-mysql
2024-02-22T20:03:56.924387Z [debug    ] Visiting CatalogNode.STREAM at '.streams[0]'.
2024-02-22T20:03:56.924620Z [debug    ] Unknown node type 'CatalogNode.STREAM'.
2024-02-22T20:03:56.924774Z [debug    ] Skipping node at '.streams[0].type'
2024-02-22T20:03:56.924897Z [debug    ] Skipping node at '.streams[0].stream'
2024-02-22T20:03:56.925024Z [debug    ] Skipping node at '.streams[0].tap_stream_id'
2024-02-22T20:03:56.925159Z [debug    ] Skipping node at '.streams[0].schema.type[0]'
2024-02-22T20:03:56.925283Z [debug    ] Skipping node at '.streams[0].schema.type[1]'
2024-02-22T20:03:56.925441Z [debug    ] Skipping node at '.streams[0].schema.additionalProperties'
2024-02-22T20:03:56.925622Z [debug    ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.test'.
2024-02-22T20:03:56.925752Z [debug    ] Unknown node type 'CatalogNode.PROPERTY'.
2024-02-22T20:03:56.925878Z [debug    ] Skipping node at '.streams[0].table-name'
2024-02-22T20:03:56.926016Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.selected'
2024-02-22T20:03:56.926139Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.replication-method'
2024-02-22T20:03:56.926261Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.inclusion'
2024-02-22T20:03:56.926402Z [debug    ] Skipping node at '.streams[0].metadata[1].metadata.selected'
2024-02-22T20:03:56.926537Z [debug    ] Skipping node at '.streams[0].metadata[1].metadata.inclusion'
2024-02-22T20:03:56.926662Z [debug    ] Skipping node at '.streams[0].metadata[1].metadata.breadcrumb[0]'
2024-02-22T20:03:56.926786Z [debug    ] Skipping node at '.streams[0].metadata[1].metadata.breadcrumb[1]'
2024-02-22T20:03:56.926935Z [debug    ] Skipping node at '.streams[0].key_properties'
2024-02-22T20:03:56.927061Z [debug    ] Skipping node at '.streams[0].bookmark_properties'
2024-02-22T20:03:56.929199Z [debug    ] Invoking: ['/home/visch/git/tap-mysql/ps-singer-tap', '--config', '/home/visch/git/tap-mysql/.meltano/run/tap-planetscale/tap.b204835c-78f5-4ac4-a5af-d6f5a295f714.config.json', '--catalog', '/home/visch/git/tap-mysql/.meltano/run/tap-planetscale/tap.properties.json']
{
  "streams": [
    {
      "type": "",
      "stream": "testtable",
      "tap_stream_id": "tap-mysql:testtable",
      "schema": {
        "type": [
          "null",
          "object"
        ],
        "additionalProperties": false,
        "properties": {
          "test": {
            "type": [
              "null",
              "string"
            ]
          }
        }
      },
      "table-name": "testtable",
      "metadata": [
        {
          "metadata": {
            "selected": false,
            "replication-method": "INCREMENTAL",
            "inclusion": "available",
            "breadcrumb": []
          }
        },
        {
          "metadata": {
            "selected": false,
            "inclusion": "available",
            "breadcrumb": [
              "properties",
              "test"
            ]
          }
        }
      ],
      "key_properties": null,
      "bookmark_properties": null
    }
  ]
}

Meltano.yml

  extractors:
  - name: tap-planetscale
    namespace: tap_planetscale
    pip_url: cowsay
    executable: /home/visch/git/tap-mysql/ps-singer-tap
    capabilities:
    - state
    - catalog
    - discover
    settings:
    - name: host
    - name: username
    - name: database
    - name: password
      sensitive: true
    config:
      host: aws.connect.psdb.cloud
      username: 0fiqne6txvcqtjbdywan
      database: tap-mysql
    select:
    - '*.*'
    metadata:
      '*':
        replication-method: FULL_TABLE