Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enforce Dataset Filters via Configuration File #171

Open
omad opened this issue Aug 28, 2023 · 0 comments
Open

Enforce Dataset Filters via Configuration File #171

omad opened this issue Aug 28, 2023 · 0 comments

Comments

@omad
Copy link
Member

omad commented Aug 28, 2023

At the moment, DEA is filtering datasets that get sent to the Queue which Alchemist is processing. For example, datasets with low geometric accuracy, or low maturity aren't supposed to be processed and aren't added to the queue.

While this works okay most of the time, it has several flaws:

  • The configuration is split between the config file, and the infrastructure which is stored somewhere completely different.
  • It's hard to reprocess, since the filtering must be done by hand.
  • It's possible to process datasets which shouldn't be processed!

We need to allow configuring a dataset filter inside the configuration file.

A possible way of doing this is using the same --predicate python form as is used on the add-missing-to-queue CLI mode.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant