Function for ETL job #11
Conversation
python/ETL/transfer.py
Outdated
| import argparse | ||
| import datetime | ||
|
|
||
| DATABASE_URL="postgresql://doadmin:l5al4hwte8qmj6x8@db-postgresql-sfo2-nextgen-do-user-1067699-0.db.ondigitalocean.com:25060/treetracker?ssl=true" |
There was a problem hiding this comment.
Please don't paste database_url here, it will compromise our db.
There was a problem hiding this comment.
Oh sorry for that. Didn't realize that's a public one.
python/ETL/transfer.py
Outdated
| if __name__ == "__main__": | ||
| parser = argparse.ArgumentParser(description="Transfer data from source to target database.") | ||
| parser.add_argument("-t","--target_db", default=None, help="URI for the target PostgreSQL database.") | ||
| parser.add_argument("-s","--source_db", default=DATABASE_URL, help="URI for the source PostgreSQL database.") | ||
| parser.add_argument("-o","--org_id", required=True, type=int, help="ID of the target organization.") | ||
| parser.add_argument("-a","--action", default = False,type=bool, help="Whether to update the database.") | ||
| args = parser.parse_args() | ||
| transfer_data(target=args.target_db, source=args.source_db,organization_id=args.org_id, action=args.action) |
There was a problem hiding this comment.
Can you change this to the pattern as here?
https://github.com/Greenstand/treetracker-airflow-dags/blob/360580afffd51021661c3e88a78f8bae1462ed72/lib/capture_export.py
https://github.com/Greenstand/treetracker-airflow-dags/blob/360580afffd51021661c3e88a78f8bae1462ed72/lib/capture_export_test.py
Here is a simple guide for this test pattern:
https://github.com/Greenstand/treetracker-airflow-dags#option-1-develop-without-installing-airflow
By doing this, we can:
- Make this function platform-agnostic, so it is easy to be reused by: cli, faas, airflow
- Easier to be tested, unit test lib used in the sample code is a good tool for testing and developing, you can easily check your code even hot-refresh your code once you change sometime in your sourcecode.
There was a problem hiding this comment.
Sure. I'll add the test file later.
|
@Xinyihe123 thank you for the contribution! Above is some suggestion from my side. |
Xinyihe123
force-pushed
the
main
branch
3 times, most recently
from
796962e to
ab32668
Compare
|
@Xinyihe123 your code looks good, I will run your script to try to do a migration, will back to you with my result. |
|
@Xinyihe123 all looks good, I will try to run your script to do a migration, will back to you with my feedback. |
|
@Xinyihe123 I can not run the test, did you run it on your side? This is my command and result: |
Yes I can run it on my side. It should be the import format, I've fixed it, it should be able to run on your side now. |
Fixed: #8
The file transfer.py:
Given an organization ID, find the target organization, related planters, trees, and species and insert them into the target database.
Need to check the mentioned table names and column names are correct.