Port Covid Recovery Dash generation code to ingestor #112
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
idreyn
changed the title
idreyn
commented
Sep 2, 2024
| @@ -99,6 +109,7 @@ def ingest_feed_to_dynamo( | |||
| "lineId": total.line_id, | |||
| "count": total.count, | |||
| "serviceMinutes": total.service_minutes, | |||
| "hasServiceExceptions": total.has_service_exceptions, | |||
There was a problem hiding this comment.
The "recovery dash" code needs to know, in general, whether the service levels on a given day include any service exceptions (usually additions/removals for holidays). It was easiest just to add this as a column to the ScheduledServiceDaily table — the migration has already run.
devinmatte
approved these changes
Sep 3, 2024
| @@ -112,6 +123,7 @@ def ingest_feeds( | |||
| force_rebuild_feeds: bool = False, | |||
| ): | |||
| for feed in feeds: | |||
| feed.use_compact_only() | |||
There was a problem hiding this comment.
Is this okay for all the s3 uploads? This is what mbta-performance will use every half hour
| @@ -148,3 +149,9 @@ def store_landing_data(event): | |||
| ridership_data = landing.get_ridership_data() | |||
| landing.upload_to_s3(json.dumps(trip_metrics_data), json.dumps(ridership_data)) | |||
| landing.clear_cache() | |||
|
|
|||
|
|
|||
| # 9:00 UTC -> 4:30/5:30am ET every day (after GTFS and ridership have bene ingested) | |||
There was a problem hiding this comment.
Suggested change
| # 9:00 UTC -> 4:30/5:30am ET every day (after GTFS and ridership have bene ingested) | |
| # 9:00 UTC -> 4:30/5:30am ET every day (after GTFS and ridership have been ingested) |
| res = {} | ||
| if isinstance(key_getter, str): | ||
| key_getter_as_str = key_getter | ||
| key_getter = lambda dict: dict[key_getter_as_str] # noqa: E731 |
There was a problem hiding this comment.
Instead of disabling each line, we can disable this rule repo wise if it's not helpful
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR finally ports the Covid Recovery Dash generation code to the data ingestor so we can run it daily and feed the resulting JSON into a CRD-like page on the Data Dashboard. The new code is very similar to what the CRD has, with a few major changes:
RidershipandScheduledServiceDailydynamo tables instead of reading from raw CSV files 🎉 Luckily most of that logic has long since been ported from CRD -> ingestor.dict[str, float]format instead of daily values inlist[float]form — this tends to be easier to work with on the Data Dashboard side.Going forward, the formats will continue to diverge, to remove covid-specific benchmarks and add by-mode totals, etc. But I've plugged this data into my local Covid Recovery Dash and confirmed that it's basically working.
The resulting JSON files are stored in the new
tm-service-ridership-dashbucket (though surely a better name exists), indexed by date. It also always writes tolatest.jsonfor easy retrieval.