Skip to content

Conversation

pengyu-hou
Copy link
Collaborator

Summary

This PR will add a flag to exclude backfill start date in the join part in versioning check.

Why / Goal

Exclude the backfill start date in the join part, it is intended to trigger separate group by backfill jobs, and semantic hash is only used in join for versioning check, therefore, exclude it for more flexibility of group by backfill jobs

Test Plan

  • Added Unit Tests
  • Covered by existing CI
  • Integration tested

Checklist

  • Documentation update

Reviewers

@airbnb/airbnb-chronon-maintainers

@pengyu-hou pengyu-hou requested a review from hzding621 April 5, 2025 01:42
Copy link
Collaborator

@hzding621 hzding621 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this change we must introduce a new entry in semantic_hash_options because this is not backward compatible and will cause all joins that use GroupBys with backfill_start_date to have a new semantic hash and trigger an unexpected archive. You also need to handle the semantic hash migration, similar to #801

@pengyu-hou
Copy link
Collaborator Author

For this change we must introduce a new entry in semantic_hash_options because this is not backward compatible and will cause all joins that use GroupBys with backfill_start_date to have a new semantic hash and trigger an unexpected archive. You also need to handle the semantic hash migration, similar to #801

Oh yes, thanks for the pointer. I was about to ask the PR reference. Will address it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants