Merge pr (#18)

* Add lookback for loan transactions * creating pr * use default for get Co-authored-by: scott.coleman <[email protected]>
singer-io · Sep 28, 2020 · 7651a31 · 7651a31
1 parent b9f8900
commit 7651a31
Show file tree

Hide file tree

Showing 4 changed files with 17 additions and 5 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,8 @@
 # Changelog
 
+## 1.1.2
+  * Added lookback window for loan transactions stream.
+
 ## 1.1.1
 
  * Fix two fields typed incorrectly in schema reported by bug ticket [#16](https://github.com/singer-io/tap-mambu/pull/16)

diff --git a/README.md b/README.md
@@ -150,6 +150,7 @@ This tap:
   - Bookmark query field: creationDate
   - Sort by: creationDate:ASC
   - Bookmark: creation_date (date-time)
+  - Lookback window: 14 day default, overridden by config
 - Transformations: Fields camelCase to snake_case
 
 [**tasks (GET v2)**](https://api.mambu.com/?http#tasks-getAll)
@@ -210,14 +211,15 @@ This tap:
     - [singer-tools](https://github.com/singer-io/singer-tools)
     - [target-stitch](https://github.com/singer-io/target-stitch)
 
-3. Create your tap's `config.json` file. The `subdomain` is everything before `.mambu.com` in the Mambu instance URL.  For the URL: `https://stitch.sandbox.mambu.com`, the subdomain would be `stitch.sandbox`.
+3. Create your tap's `config.json` file. The `subdomain` is everything before `.mambu.com` in the Mambu instance URL.  For the URL: `https://stitch.sandbox.mambu.com`, the subdomain would be `stitch.sandbox`. Lookback window applies only to `loan transactions` stream.
 
     ```json
     {
         "username": "YOUR_USERNAME",
         "password": "YOUR_PASSWORD",
         "subdomain": "YOUR_SUBDOMAIN",
         "start_date": "2019-01-01T00:00:00Z",
+        "lookback_window: 30,
         "user_agent": "tap-mambu <api_user_email@your_company.com>",
         "page_size": "500"
     }

diff --git a/setup.py b/setup.py
@@ -3,7 +3,7 @@
 from setuptools import setup, find_packages
 
 setup(name='tap-mambu',
-      version='1.1.1',
+      version='1.1.2',
       description='Singer.io tap for extracting data from the Mambu 2.0 API',
       author='[email protected]',
       classifiers=['Programming Language :: Python :: 3 :: Only'],

diff --git a/tap_mambu/sync.py b/tap_mambu/sync.py
@@ -1,11 +1,13 @@
+from _datetime import timedelta
 from datetime import datetime
+
 import singer
-from singer import metrics, metadata, Transformer, utils
-from singer.utils import strptime_to_utc, strftime
+from singer import Transformer, metadata, metrics, utils
+from singer.utils import strftime, strptime_to_utc
 from tap_mambu.transform import transform_json
 
 LOGGER = singer.get_logger()
-
+LOOKBACK_DEFAULT = 14
 
 def write_schema(catalog, stream_name):
     stream = catalog.get_stream(stream_name)
@@ -363,6 +365,11 @@ def sync(client, config, catalog, state):
 
     loan_transactions_dttm_str = get_bookmark(state, 'loan_transactions', 'self', start_date)
     loan_transactions_dt_str = transform_datetime(loan_transactions_dttm_str)[:10]
+    loan_transactions_dttm = strptime_to_utc(loan_transactions_dt_str)
+    lookback_days = int(config.get('lookback_window', LOOKBACK_DEFAULT))
+    lookback_date = utils.now() - timedelta(lookback_days)
+    if loan_transactions_dttm > lookback_date:
+        loan_transactions_dt_str = transform_datetime(strftime(lookback_date))[:10]
     # LOGGER.info('loan_transactions bookmark_date = {}'.format(loan_transactions_dt_str))
 
     # endpoints: API URL endpoints to be called