Proposal discussion: Specialized RESTSink class to aid in writing SaaS targets

[aaronsteers]

As first discussed in Slack between myself, @edgarrmondragon, and Bernard Wezeman:

You are right to call out the SDK's relatively less mature helper classes for API/SaaS targets. The corollary for the class RESTStream or GraphQLStream would be something like a RESTSink but as of now we only have RecordSink and BatchSink. Those classes are capable of supporting SaaS targets, but not really optimized for them.

In theory, RESTSink could be created and added to the SDK, and if so, I imagine that would have the same kind of streamlining as RESTStream has (around the requests library specifically). Also, some RESTSinks would use POST for one more more records at a time, and some might use PUT for exactly one record at a time. Those could represent separate subclasses or just implementation options on the same class. :thinking_face:

Opening this discussion to begin exploring what this interface might look like.

PRs are welcome, but because spec discussion may be a long-running conversation, this discussion should give space to tackle various aspects of exploring the implications around spec and validating use cases.

[edgarrmondragon]

My 2c on a few aspects the API should cover:

• Authenticator classes currently used for taps should be reusable (I think this is already the case).
• Schema validation should be built-in and run before pushing any data to the target.
• SaaS targets may have hardcoded sinks just like SaaS taps usually have hardcoded streams. E.g. SubscribersSink, CampaignsSink, etc.
• The scenario in the previous point may require an explicit stream-to-entity mapping. This configuration option should be built-in (similar to stream maps).

Anything I'm missing?

[aaronsteers]

@edgarrmondragon - A great list, thanks!

A couple things that come to mind...

Primary Keys, deduping, and distinguishing between update vs create verbs for each incoming record

The target may need logic to know on what keys to dedupe and perhaps how to distinguish an UPDATE versus a CREATE (for APIs that need to). This can vary from very simple to extremely complex depending on the API.

A simple case is "customer ID" is part of the path and the API doesn't will update or create as needed without us knowing ahead of time whether it exists.

A complex case is that the primary key on the target API is a surrogate key assigned after POST. The "expected" solution from the target API's perspective is that the source system holds only the mapping of business keys to surrogate keys in order to properly update - not create duplicates - in future.

We won't solve all applications with one solution so I think we have to be guided by which APIs we are solving for first and try (as well as possible) to solve those use cases using a reasonable amount of reusability.

Backoff / retry profiles

As with Auth classes, we can probably reuse backoff and retry implementation options from the tap SDK. Still, worth mentioning and thinking about.

Singleton vs Batch, and implications for failures and retries

Some SaaS targets will only be able to accept one record at a time and some may be more performant with writing 100s or 1000s of records at once in one REST call.

If multiple records are accepted at once, do we simply hard fail the whole load if one of the batch fails? Should the developer have an interface to manage errors, retrying with a subset of records or performing some other recovery/retry attempts.

Dead letter queues

There are many SaaS target use cases where the expected action will be to continue on failure, and for some use cases a dead letter queue would be necessary to track which records have failed. Dead letter queue data could be saved for future retry, could be used as a disallow list for future records of the same key, or could be used for logging/escalation when a human needs to take some action based on the failed upsert operation.

[JulesHuisman]

I think these are great points @edgarrmondragon and @aaronsteers, there is a lot of overlap with the points Bernard and I came up with.

The authentication should be an easy port as the functionality will be the same. One thing we ran into is the fact that the Authenticators are currently coupled with streams. As the Authenticators use the tap_name for example.

Schema validation would be great, and could have the same design as the Streams currently do, just the other way around.

Design wise, all the Saas targets we need to build have the same "upsert" requirement. We need to 1) check if a record already exists, 2) if it does not, insert it, otherwise 3) update it.

Here is a rough sketch of the idea we came up with, let me know what you guys think about this.

class CustomerSink:
    schema = th.PropertiesList(
        th.Property('xxx', th.StringType),
        ...
    ).to_dict()

    def process_record(self, record: dict) -> None:
        if self.record_exists(record):
            self.update_record(record)
        else:
            self.create_record(record)

    def record_exists(self, record: dict) -> bool:
        """
        In our example the customer id needs to be inserted into the url.
        E.g. /API/V3/Account/{accountID}/Customer/{customerID}.json
        Not sure how this would work yet.
        """
        ...

    def create_record(self, record: dict) -> None:
        """
        This could default to a POST of the record on a predifined url.
        """
        ...

    def update_record(self, record: dict) -> None:
        """
        This could default to a PUT of the record.
        """
        ...

This would also allow for different behaviour of inserting the records:

If you do not need to update:

def process_record(self, record: dict) -> None:
    if not self.record_exists(record):
        self.create_record(record)

Or if the API supports upserting themselves:

def process_record(self, record: dict) -> None:
    self.update_record(record)