Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Further revision to the new ingestion workflow #16

Merged
merged 74 commits into from
Nov 8, 2023
Merged
Changes from 1 commit
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
37de655
Re-architecting the loader classes
dchandan Aug 24, 2023
12305af
updating gitignore
dchandan Aug 24, 2023
ca45cc3
further developing the ingestion loop
dchandan Aug 24, 2023
6e500d8
moving post_stac_item to stac_utils
dchandan Aug 24, 2023
f93b10d
renaming post_collection to post_stac_collection
dchandan Aug 24, 2023
d5a3d2d
moving collection creation to seaparate function
dchandan Aug 25, 2023
946e72a
moving create_stac_collection to STACpopulatorBase
dchandan Aug 25, 2023
3ee1e6e
moving all STAC API calls to separate file
dchandan Aug 25, 2023
db008a5
Create pydantic data model for CMIP6 CV
huard Aug 28, 2023
92c5b74
Merge branch 'arch-changes' into validation
huard Aug 28, 2023
fc2daf3
create STAC item from TDS NcML response - untested due to missing STA…
huard Aug 28, 2023
ab09580
Suggestions from Deepak. Use xncml 0.3 from_text
huard Aug 28, 2023
86dddd4
black
huard Aug 28, 2023
1c7aa53
Fix errors in metadata parsing
huard Aug 28, 2023
dd171a6
change dchandan user to testuser
huard Aug 28, 2023
7e2903a
Merge pull request #17 from Ouranosinc/validation
dchandan Aug 30, 2023
99d17b5
adding type hints to collection2enum
dchandan Aug 30, 2023
94a43a5
implementation for post_stac_item + logger changes
dchandan Aug 30, 2023
e193c28
removing validator from CMIP6populator
dchandan Aug 30, 2023
a0770a7
makefile
dchandan Aug 30, 2023
b481f37
simplifying the ingestion logic
dchandan Aug 31, 2023
d28ab74
comments and small changes to the posting functions
dchandan Aug 31, 2023
933d003
fixing issue with thredds metadata for attributes with type tag
dchandan Sep 1, 2023
f697be9
Replaced enums by literal for CMIP6 CV
huard Sep 5, 2023
74ad594
Implemented post_stac_item, using a hash of the item attributes as th…
huard Sep 20, 2023
ccf1c88
added assets
huard Sep 20, 2023
e7b67a1
added Datacube extension to CMIP6. Did some code clean-up
huard Sep 21, 2023
ac441f7
refactoring to support multiple extensions later
huard Sep 21, 2023
6719b10
fixed bugs in datacube logic.
huard Sep 21, 2023
7133cb6
revised pydantic data model
dchandan Sep 21, 2023
8fd1825
docstrings
huard Sep 21, 2023
a316d4d
merge
huard Sep 22, 2023
ce268cd
work on cmip6 extension
huard Sep 26, 2023
dba86a5
fixes to CMIP6 Properties model to allow serialize
fmigneault Sep 27, 2023
ee7a339
fix prefix addition. test cmip6 extension
huard Sep 28, 2023
6d32e14
merge
huard Sep 28, 2023
0c829a1
harmonized datacube and cmip extensions
huard Sep 29, 2023
71ce3e0
adding numpy types to python types conversion for metadata
dchandan Oct 5, 2023
350b4f4
removing collection2enum
dchandan Oct 5, 2023
2a445f0
black
dchandan Oct 5, 2023
2728ce6
extracting pydantic base models to models.py
dchandan Oct 5, 2023
b47d613
removing cmip6 extension code
dchandan Oct 5, 2023
2f5dc39
Breaking CFJsonItem part 1: extracting STAC item creation
dchandan Oct 6, 2023
3f821ce
Breaking CFJsonItem part 2: extracting datacube extension code
dchandan Oct 6, 2023
3c584cc
updating geometry structure
dchandan Oct 12, 2023
b7a7ed9
moving np datatype conversion to a separate function
dchandan Oct 12, 2023
48598ae
modifications to datacube extension helper functions as per Francis's…
dchandan Oct 12, 2023
94eb521
code cleanup
dchandan Oct 12, 2023
a64a226
change how prefix is applied
dchandan Oct 12, 2023
f22c1a2
PR changes
dchandan Oct 13, 2023
efd9230
fixing output media type and roles output for assets
dchandan Oct 17, 2023
3e88591
adding magpie resource link
dchandan Oct 17, 2023
8d66fba
adding collection resource link for Magpie
dchandan Oct 18, 2023
00a968a
posting items fixes
dchandan Oct 19, 2023
2c3b49d
removing function no longer in use
dchandan Oct 19, 2023
6908d55
implemented updating stac collection and items
dchandan Oct 19, 2023
0c959ea
removing need to pass yml file to app on command line
dchandan Oct 19, 2023
73b2773
code cleanup
dchandan Oct 19, 2023
9e919c2
adding __init__ files
dchandan Oct 19, 2023
c62fb80
fix
dchandan Oct 19, 2023
10db128
more fixes
dchandan Oct 19, 2023
25985db
diagnostics
dchandan Oct 23, 2023
6d675bc
removing unused code
dchandan Oct 23, 2023
65bd5bb
refactoring to allow more flexibility
dchandan Oct 23, 2023
f540dbe
fix datacube extension
dchandan Oct 26, 2023
323c945
pr changes
dchandan Oct 27, 2023
0581c61
reverting to old way to read thredds access links
dchandan Oct 27, 2023
37a26e1
adding ability to get single file from THREDDS loader
dchandan Nov 8, 2023
e55591d
making make_cmip6_item_id a staticmethod
dchandan Nov 8, 2023
f1e28db
wrapping call to make STAC item with a try-exepcet block
dchandan Nov 8, 2023
8bb21e1
fixing commit e55591dd0b7f7db6cd4ee7256512d5693d282145
dchandan Nov 8, 2023
3055afc
more fixes to previous commits
dchandan Nov 8, 2023
3f1d284
making tracking_id optional in CMIP6ItemProperties
dchandan Nov 8, 2023
26fe4ad
Merge pull request #25 from crim-ca/arch-finalization-proposal
dchandan Nov 8, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
makefile
dchandan committed Aug 30, 2023
commit a0770a72ea6626da91a493fc46990e9a6ab08ba0
17 changes: 17 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
IMP_DIR = /Users/dchandan/DACCS/Codes/stac-populator/implementations
STAC_HOST = http://localhost:8880/stac

testcmip6:
python $(IMP_DIR)/CMIP6-UofT/add_CMIP6.py $(STAC_HOST) https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/catalog/birdhouse/testdata/xclim/cmip6/catalog.html $(IMP_DIR)/CMIP6-UofT/CMIP6.yml


starthost:
docker compose up

stophost:
docker compose down

del_docker_volume: stophost
docker volume rm stac-populator_stac-db

resethost: del_docker_volume starthost