Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

promdump: prevent concurrent executions from the same working directory #145

Open
ionthegeek opened this issue Sep 9, 2024 · 0 comments
Labels
enhancement New feature or request promdump Issues related to the prometheus-remote-backfill tools

Comments

@ionthegeek
Copy link
Contributor

ionthegeek commented Sep 9, 2024

If two instances of the promdump binary manage to pass the export file existence check before writing out their first batch of metrics, they will step all over each other and produce a dump that is -- at best -- inconsistent.

The easiest way to prevent this is probably creating a lock file and having each instance lock it before attempting to dump any metrics. Locks are automatically cleaned up if the process exits, so we wouldn't have to worry about them going stale even if the lock file is left around.

Unfortunately Golang doesn't have a native library for advisory locks, so we might have to roll our own or vet a third party package.

@ionthegeek ionthegeek added enhancement New feature or request promdump Issues related to the prometheus-remote-backfill tools labels Sep 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request promdump Issues related to the prometheus-remote-backfill tools
Projects
None yet
Development

No branches or pull requests

1 participant