the problem of out of memory

[bruse-peng]

Feature request

Use case. Why is this important?

“Nice to have” is not a good use case. :)

Bug Report

What did you do?

What did you expect to see?

What did you see instead? Under which circumstances?

Environment

• System information:

    Insert output of `uname -srm` here.
  

• Pushgateway version:

    Insert output of `pushgateway --version` here.
  

• Pushgateway command line:

    Insert full command line.
  

• Logs:

Insert Pushgateway logs relevant to the issue here.

when i run pushgateway accept the metric for a few hours, it throw out error message 'out of memory' how can i remove the old metrics release the memory,below is my running command:

/usr/local/pushgateway-1.4.2.linux-amd64/pushgateway --persistence.file=/usr/local/prometheus-pushgateway-data/pushgateway.data --persistence.interval=10s --log.level=error

[bruse-peng]

when i persistence the metics to disk if the metrics will be delete from memory?

[beorn7]

Persistence is only used to restore the pushed metrics after a restart. All pushed metrics are always kept in memory. The Pushgateway is designed for just a few metrics (usually up to hundreds, thousands would be a stretch). It's not a fully fledged database like the Prometheus itself (which can manage many millions of metrics).

You can delete metrics with the DELETE method: https://github.com/prometheus/pushgateway#delete-method
Or you can push them in a suitable way so that old metrics are overwritten. See documentation for PUT and POST: https://github.com/prometheus/pushgateway#delete-method

[bruse-peng]

how i can guarantee the metrics pulled by prometheus beforce if i delete metrics with the DELETE method: https://github.com/prometheus/pushgateway#delete-method ? if can't the metrics data will lost

[beorn7]

You cannot. The Prometheus scrape model is stateless. A target exposes metrics, and any number of scrapers can scrape it as often as they want (which includes zero). Metrics are exposed with their most current value. In the clean case, they are only ever updated and never deleted. Deletion may make sense if the metrics are simply out of date, so you don't want them to be scraped anymore.

If you require a "scrape at least once" semantics, it's a strong hint that you are in fact doing something like event logging, for which Prometheus is a huge misfit.