Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix/visitor activity 504 #8

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Conversation

dmosorast
Copy link
Collaborator

Description of change

Pardot's API will return a 504 on large requests with certain parameters for the visitorActivity endpoint. There's an inactive topic on the Salesforce community boards that suggests changing the request parameters could cause this to be more efficient. https://trailblazers.salesforce.com/answers?id=9063A000000DlRUQA0

In testing, sorting by updated_at descending proved to be the only consistent option, so this PR adds a stream type to handle that pattern (bookmarking based on tap-trello's implementation).

This PR also adds a method to translate existing "id-based" state to the current "updated_at" based state, using a request for the activity object whose ID is bookmarked.

Of note: The API seems to treat datetimes formatted as YYYY-mm-DD HH:MM:SS as the user's time zone, and datetimes formatted as YYYY-mm-DDTHH:MM:SSZ as UTC. A critical part of this PR is to never change the formatting of values so that these types can be mixed effectively for consistency.

Manual QA steps

  • Ran through cases of the state to confirm that:
    • The expected count of records is returned using this method
    • The pagination works and aligns using mixed datetime format (UTC vs user, in this case EDT)
    • The state translation works and successfully overrides a reasonable start_date (e.g., 2019-04-12T00:00:00.000000Z)

Risks

  • Medium, this is the replacement of an existing pattern, and testing will need to be performed end to end with a live connection to validate the resulting data.

Rollback steps

  • revert this branch, bump patch version, and release

@chrisgonzalez0
Copy link

Will this be merged anytime soon? I am using Stitch to try to backfill some historical data from Pardot and this same issue seems to be occurring on the Stitch side. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants