Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backfill the EnrollmentEvents table from Amplitude/Metabase data #2436

Closed
3 tasks done
thekaveman opened this issue Oct 8, 2024 · 4 comments
Closed
3 tasks done

Backfill the EnrollmentEvents table from Amplitude/Metabase data #2436

thekaveman opened this issue Oct 8, 2024 · 4 comments
Assignees
Labels
analytics App event tracking, instrumentation, logging: Metabase, Amplitude deliverable Work that ends with a non-code deliverable (e.g. Google doc)

Comments

@thekaveman
Copy link
Member

thekaveman commented Oct 8, 2024

This is blocked until cal-itp/data-infra#3468 is complete and merged.

This is blocked until #2521 is complete and merged.

We want to take this historical enrollment record from Amplitude/Metabase and backfill our newly created local EnrollmentEvent table in the app.

The backfill should be for any enrollment that occurred prior to when the EnrollmentEvent table went into prod: 09/18/2024, version 2024.09.2

The general approach could be something like:

  1. Download all enrollment events from Metabase as JSON
  2. Transform into a Django fixture JSON structure
  3. Download the production database locally
  4. Use manage.py loaddata <fixtures> to load the fixtures in
  5. Re-upload and overwrite the production database

Acceptance criteria

  • All historical enrollments (~153 as of 09/22/2024) are represented accurately within the prod Admin interface EnrollmentEvents table.
  • The Amplitude/Metabase property event_time is used for the EnrollmentEvent.enrollment_datetime value
  • EnrollmentEvent.enrollment_method is DIGITAL for all backfilled events
@thekaveman thekaveman converted this from a draft issue Oct 8, 2024
@thekaveman thekaveman moved this from Stretch to Blocked in Digital Services Oct 10, 2024
@thekaveman thekaveman moved this from Blocked to Todo in Digital Services Oct 25, 2024
@thekaveman thekaveman moved this from Todo to Blocked in Digital Services Oct 31, 2024
@thekaveman
Copy link
Member Author

Blocked until cal-itp/data-infra#3518 is merged.

@thekaveman thekaveman moved this from Blocked to Todo in Digital Services Nov 4, 2024
@thekaveman thekaveman self-assigned this Nov 4, 2024
@thekaveman thekaveman added chore Chores and tasks for code cleanup, dev experience, admin/configuration settings, etc. analytics App event tracking, instrumentation, logging: Metabase, Amplitude labels Nov 4, 2024
@thekaveman thekaveman added this to the Admin tool: agency users milestone Nov 4, 2024
@thekaveman thekaveman moved this from Todo to In progress in Digital Services Nov 12, 2024
@thekaveman thekaveman moved this from In progress to Blocked in Digital Services Nov 13, 2024
@thekaveman
Copy link
Member Author

#2521 should be completed first; blocking this.

@thekaveman thekaveman moved this from Blocked to In progress in Digital Services Nov 13, 2024
@thekaveman
Copy link
Member Author

Updates to warehouse pipeline merged. Awaiting warehouse rebuild/resync with Metabase.

@thekaveman
Copy link
Member Author

Historical enrollment events have been back-filled into benefits.calitp.org, which is now showing a total of 193 enrollments going back to December 2021:

Image

Note that Metabase, as of 11/15/2024 at 13:00 Pacific, is showing a total of 192 enrollments. There was an enrollment this morning that has not yet made it from Amplitude --> Metabase 😉

@github-project-automation github-project-automation bot moved this from In progress to Done in Digital Services Nov 15, 2024
@thekaveman thekaveman added deliverable Work that ends with a non-code deliverable (e.g. Google doc) and removed chore Chores and tasks for code cleanup, dev experience, admin/configuration settings, etc. labels Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
analytics App event tracking, instrumentation, logging: Metabase, Amplitude deliverable Work that ends with a non-code deliverable (e.g. Google doc)
Projects
Status: Done
Development

No branches or pull requests

1 participant