Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Glob Support in S3 find is weird #566

Open
omad opened this issue Jun 9, 2023 · 1 comment
Open

Glob Support in S3 find is weird #566

omad opened this issue Jun 9, 2023 · 1 comment

Comments

@omad
Copy link
Member

omad commented Jun 9, 2023

I don't understand what globs are supported and what aren't. Wasted so much time on this.

$ s3-find s3://dea-public-data-dev/ausbath_09_v4/ga_multi_\*json 
$ s3-find s3://dea-public-data-dev/ausbath_09_v4/ga_multi_\*.json 
$ s3-find s3://dea-public-data-dev/ausbath_09_v4/\*.json 
s3://dea-public-data-dev/ausbath_09_v4/ga_multi_ausbath_0.stac-item.json

Which was made more inscrutiable by the indefinite hang (see #105 )

s3-to-dc --stac --update-if-exists --skip-check s3://dea-public-data-dev/ausbath_09_v4/ga_multi_ausbath_0.stac-item.json ga_multi_ausbath_0
/usr/local/lib/python3.8/dist-packages/datacube/drivers/postgres/_api.py:48: RemovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
  select([
Traceback (most recent call last):
  File "/usr/local/bin/s3-to-dc", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/s3_to_dc.py", line 169, in cli
    added, failed, skipped = dump_to_odc(
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/s3_to_dc.py", line 93, in dump_to_odc
    raise IndexingException("Doc stream was empty")
odc.apps.dc_tools.utils.IndexingException: Doc stream was empty

#### HANGS HERE ####
@Ariana-B
Copy link
Contributor

Indefinite hang resolved by #562

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants