Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a FileExistsSensor before creating the foreign table, using pg_stat_file or similar #6455

Open
greenape opened this issue Feb 21, 2024 · 1 comment
Labels
enhancement New feature or request FlowETL

Comments

@greenape
Copy link
Member

          The approach makes sense. Although this still leaves us in the situation of using the fail/retry mechanism in lieu of a sensor; it's just that it will now be the `CreateForeignStagingTableOperator` task that fails and retries until the file appears, rather than the "wait for data" task. I wonder whether we could add a `FileExistsSensor` before creating the foreign table, using `pg_stat_file` or similar?

Originally posted by @jc-harrison in #6448 (review)

@greenape
Copy link
Member Author

My suggestion from that thread is that we do something using pg_ls_dir to allow for checking for multiple files.

@greenape greenape added enhancement New feature or request FlowETL labels Feb 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request FlowETL
Projects
None yet
Development

No branches or pull requests

1 participant