Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Load data from the OpenAQ data archive on AWS Open Data #237

Draft
wants to merge 17 commits into
base: develop
Choose a base branch
from
Draft

Load data from the OpenAQ data archive on AWS Open Data #237

wants to merge 17 commits into from

Conversation

zmoon
Copy link
Member

@zmoon zmoon commented Mar 7, 2025 •
edited
Loading

  • add imports
  • more tests (e.g. loading data for more than one site)
  • wide format conversion (maybe use the func from the v3 web API once OpenAQ web API v3 #198 is merged)
  • it would be nice to have more site metadata, which get_locations from the v3 web API reader could provide

zmoon added 17 commits September 4, 2024 11:55
@zmoon
Initial file reader
fb620ee
@zmoon
Initial add_data
5c038ee 
very slow if going for all locations though
@zmoon
Seems like using exists is a bit faster
871f5fa 
in the case of one month for two locs...
@zmoon
Get loc IDs from directory structure
cd9254e
@zmoon
By provider
2291ed3 
testing with aqdc, 2017-07, takes a few min to find files,
though after the first date query the next ones seem much faster

found almost 4k files, loaded in a few min with 4 proc
@zmoon
API consistency
e8de0db
@zmoon
By country
7c76b68 
tried mx (Mexico), which seems to have had just 40 unique locs
in 2017-07, still took 5-ish minutes to discover the 700-ish files
@zmoon
Funcs to help load data from a list of sites
e98d309 
without pre-checking if the files exist
@zmoon
Try discovering all site-days
ca7fd16 
took 18 hours to find 17_455_910 site-days,
so wouldn't want to do it often

.csv.gz at 41 MB
@zmoon
Tweak name
9c53d78
@zmoon
Some name tweaks
5a0b723
@zmoon
unit -> units
6453b03 
this is the standard name
going this way with the v3 API reader too
@zmoon
More discovery options
94f5520
@zmoon
Add naive build-url option for add-data
709a1ba
@zmoon
Return df from get-locations showing country and provider
14a1683
@zmoon
Add some tests
b05c255
@zmoon
Test add-data for one site-day
0bc2612
zmoon
zmoon commented Mar 7, 2025
View reviewed changes
monetio/obs/openaq_aws.py

https://registry.opendata.aws/openaq/

https://docs.openaq.org/docs/accessing-openaq-archive-data
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
https://docs.openaq.org/docs/accessing-openaq-archive-data
https://docs.openaq.org/aws/about

it was changed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@zmoon