feat: Mock capability for taps

[ReubenFrankel]

Feature scope

Taps (catalog, state, tests, etc.)

Description

Do you see value in adding a mock data capability to SDK taps? In the spirit of #1009 and MeltanoLabs/Singer-Most-Wanted#90 (comment), the idea would be to generate mock records for each stream based on their schemas (JSON Schema property type, format, enum, etc) using Faker - possibly reusing/refactoring the current stream maps implementation. This is obviously already possible with stream maps today, but would require stubbing out each stream property individually - the aim is to streamline this process; reduce the amount of necessary config and assume sensible mock values. Another benefit would be for taps with statically defined schemas that would be able to run without any config or connection to the source as a result, allowing for quick onboarding/prototyping. This would also work for taps that perform dynamic discovery, although config would still be required (in most cases).

Spec

• New MOCK (or similar) capability, which exposes some SDK-native settings

  capabilities:
  - MOCK

  # perhaps these could be declared per-stream, although some overlap with stream maps
  settings:
  - name: mock_enabled
    kind: boolean
    default: false
  - name: mock_records
    kind: integer
    default: 10

• Underlying Faker instance seed and locale configurable with faker_config (as with stream maps)

  config:
    faker_config:
      seed: 0
      locale:
      - en_US
      - en_GB

Considerations

• Inference of Faker provider from just JSON Schema property type/format is fairly straightforward - it may be possible to resolve a more accurate provider in a more granular context when taking the property name into account (this would be more of a "best guess", however)
• With Meltano: warn if running meltano config <tap> test with mock_enabled: true, as this may otherwise indicate a false-positive

[edgarrmondragon]

Hey @ReubenFrankel 👋

• How do you imagine invoking a tap in "mock" mode? Something like MELTANO_EXTRACTOR_MOCK_ENABLED meltano run ...?
• Other than your comment about meltano config <tap> test, do you think most of the work to enable this would be done in the SDK? Do you think non-SDK taps are out of scope for this feature?

[ReubenFrankel]

How do you imagine invoking a tap in "mock" mode? Something like MELTANO_EXTRACTOR_MOCK_ENABLED meltano run ...?

Yes, given that mock_enabled is a valid tap setting.

Other than your comment about meltano config <tap> test, do you think most of the work to enable this would be done in the SDK?

Yes.

Do you think non-SDK taps are out of scope for this feature?

Potentially not, given that meltano-map-transformer or another SDK-powered mapper could bundle a mock mapping.