feature: Support for Descending Replication Keys in Stream Resumption

[JulioWhile]

Feature scope

Other

Description

Feature Scope

Core Functionality (Stream Resumption, Replication Keys)

Description

Problem

In the current Meltano implementation, we have an is_sorted property that supports resuming streams when unexpectedly interrupted. This works well when the replication_key is ascending, typically dates. However, there are use cases that deal with descending replication_keys, and the existing configuration does not accommodate these scenarios.

Proposal

Introduce support for descending replication_keys in Meltano. This would involve adding logic similar to the existing logic for upstream replication keys. To do this, I propose to add a new boolean field is_descending similar to is_sorted, which would allow the system to handle flows sorted in a descending fashion. (or maybe we can do something to make it work with is_sorted??)

Once this is implemented, the updated code could look something like this:

if self.is_sorted:
    start_date = self.get_starting_replication_key(context)
    if start_date:
        params["start_date"] = start_date

if self.is_descending:
   end_date = self.get_starting_replication_key(context)
   if end_date:
       params["end_date"] = end_date

Let me know what you think or how we can implement this idea, or if you are already working on something for this. ✌️

[edgarrmondragon]

Thanks for logging @JulioWhile!

What would happen when more recent records are added to the source? It's unlike the more common and currently supported reverse case where older records should not appear in the source under normal circumstances.

Or did you have a specific source or use case in mind that doesn't have that risk?