Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new daily job to build Covid Recovery Dash data.json #66

Open
idreyn opened this issue Jan 12, 2024 · 0 comments · May be fixed by #112
Open

Add a new daily job to build Covid Recovery Dash data.json #66

idreyn opened this issue Jan 12, 2024 · 0 comments · May be fixed by #112
Labels
enhancement New feature or request

Comments

@idreyn
Copy link
Contributor

idreyn commented Jan 12, 2024

Some plans are forming around the Covid Recovery Dash:

  • In the immediate term, we'd like to automatically update it daily
  • In the medium term, we want to port its functionality over to the System Ridership tab of the Data Dashboard

In either case, the first step is to generate the CRD's data.json file from our ScheduledServiceDaily and Ridership Dynamo tables. An example of the file is here, and the code that generates it is here. With the daily ridership and service level data already available in Dynamo, there's a lot of logic from the CRD codebase that can be dropped or ignored — the important thing is that the structure of the file should remain the same.

In the end, we should have a cron job in this repo that automatically updates this file daily, running shortly after the tables mentioned above are updated, and drops it somewhere in S3. We will probably want to beef up the lambda_timeout and lambda_memory_size for the job to make sure it can reliably complete — it's got a lot of work to do. We should also consider parallelizing Dynamo reads if possible, since it'll need to look up values for ~100 routes.

Nice-to-haves:

  • We'd prefer to have the file contain pre-covid history too, perhaps back to 2016
  • If the file is getting very large, we should consider chunking it by route
  • If you can add ridership and service breakdowns by mode while you're in here, please do! There's already a spot for system-wide values that this can slot into nearby.
@idreyn idreyn added the enhancement New feature or request label Jan 12, 2024
@idreyn idreyn linked a pull request Sep 2, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant