Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve store get_info with collection data for VersionStore #544

Open
dimosped opened this issue Apr 20, 2018 · 1 comment
Open

Improve store get_info with collection data for VersionStore #544

dimosped opened this issue Apr 20, 2018 · 1 comment

Comments

@dimosped
Copy link
Contributor

dimosped commented Apr 20, 2018

We need to improve the available meta-data we keep for the data usage of a given library.

Extend capabilities of get_info with :

  • collection size
  • size of old versions
  • size of orphaned segments
  • last access time the collection was accessed
  • number (maybe size) of versions accessed each day

The user may create standardized Arctic Prometheus metrics for the above, with option to start prometheus web server ready for scraping.

The goal is to help understanding space usage providing insights for data eligible for deletion.

This could also help with solutions of multi tier storage (e.g. hybrid mongo(where most recently accessed data go) and nfs/s3(slower easier to maintain, slower retrieval))

@dimosped
Copy link
Contributor Author

Use Prometheus in your Python code:
https://github.com/prometheus/client_python

Spin up your own local Prometheus server:
https://prometheus.io/docs/prometheus/latest/getting_started/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant