dag to scrape and save the current ridership URL from the NTD portal #3545
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
charlie-costanzo
requested review from
evansiroky and
vevetron
as code owners
mjumbewu
reviewed
Nov 14, 2024
airflow/dags/sync_ntd_data_xlsx/scrape_ntd_ridership_xlsx_url.py
Outdated
Show resolved
Hide resolved
charlie-costanzo
requested review from
tiffanychu90 and
hunterowens
as code owners
charlie-costanzo
force-pushed
the
scrape-ntd-ridership-xlsx-url
branch
2 times, most recently
from
38f0deb
to
5806a3d
Compare
| @@ -15,9 +15,11 @@ repos: | |||
| rev: 6.0.0 | |||
| hooks: | |||
| - id: flake8 | |||
| args: ["--ignore=E501,W503"] # line too long and line before binary operator (black is ok with these) | |||
| args: ["--ignore=E501,W503,E231"] # line too long and line before binary operator (black is ok with these) and explicitly ignore the whitespace after colon error | |||
There was a problem hiding this comment.
Added --ignore=E231 to flake8 args to ignore "missing whitespace after colon" errors in lambda functions, which were causing CI failures despite being properly formatted locally
| types: | ||
| - python | ||
| # Suppress SyntaxWarning about invalid escape sequence from calitp-data-infra dependency without modifying source | ||
| entry: env PYTHONWARNINGS="ignore::SyntaxWarning" flake8 |
There was a problem hiding this comment.
Added entry: env PYTHONWARNINGS="ignore::SyntaxWarning" flake8 to suppress a SyntaxWarning about invalid escape sequence from the calitp-data-infra dependency without modifying its source code
charlie-costanzo
force-pushed
the
scrape-ntd-ridership-xlsx-url
branch
from
90dc13c
to
605196c
Compare
charlie-costanzo
added
the
data-pipeline-ingestion-and-modeling
charlie-costanzo
force-pushed
the
scrape-ntd-ridership-xlsx-url
branch
from
605196c
to
16d7592
Compare
evansiroky
approved these changes
Nov 26, 2024
charlie-costanzo
force-pushed
the
scrape-ntd-ridership-xlsx-url
branch
from
16d7592
to
7eefb94
Compare
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
When we created Airflow operator to ingest the NTD monthly ridership tables every month in XLSX format, we didn't realize that DOT changes the file name every time it publishes the data.
Instead of relying on someone to remember to manually run the data ingestion every month with the new link, this PR dynamically scrapes the webpage for the data download to retrieve the new link, save it in airflow as an environment variable, and allows the request library to utilize the updated link. To do this, it also makes the ingest airflow task dependent on the success of the URL scraper.
Type of change
How has this been tested?
local airflow
Post-merge follow-ups
Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.
monitor for expected behavior