Skip to content

ETL for loading weather data into lake weather API database

License

Notifications You must be signed in to change notification settings

kgmcquate/weather-etl

Repository files navigation

ETL for loading weather data from open-meteo API to database

System diagram is located in the readme of this repo: (https://github.com/kgmcquate/lake-freeze-frontend)

TODO:

  • Create publicly-available job monitoring dashboard from CloudWatch logs
  • Create publicly-available dashboard for weather data
  • Add Data Quality step and alert
  • Evaluate cost of running this in Azure instead of AWS EMR Serverless
  • Run airflow on EC2 (with nat gateway) and make the UI publcly available
  • Add unit tests

About

ETL for loading weather data into lake weather API database

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published