Audit Metric Consolidation

[amydevs]

Specification

It has been decided that the metrics stored in Polykey should better match the types of metrics found in Prometheus, within the computed metrics such as averages and such, being produced by a Mimir server offering up a PromQL interface.

This will mean that the AuditMetrics as they are need to be reworked into 4 distinct types:

• Counter - A metric value that can only increase of reset (Each value cannot be lower than the previous value). It should be used for metrics such as number of connections, number of errors, etc. If these metrics are fed into Mimir, PromQL is able to aggregate more complex metrics such as the rate of connections sampled at some period of time as seen below:
• Gage - This is a number that can either go up or down, it should be used for metrics like the the number of seednodes, CPU/RAM usage, etc. If these metrics are fed into Mimir, PromQL is able to aggregate these metrics into max/min/avg over certain time periods:
• Histogram - Histograms are buckets that contain counts of values that make up that bucket. For example, a histogram would have the following structure if it were to receive a value of 0.4:

Bucket	Count
0 - 0.1	0
0 - 0.5	1
0 - 1	1
0 - +inf	1

• Summary - TBD

Storage

We have our key-value Rocks-DB database. This means that our metrics will have to be similarly layed out like our events, being that we have metric paths, with the subpaths of those containing AuditPointId keys that point to metrics in Audit/points/${id}.

Additional context

• https://prometheus.io/docs/tutorials/understanding_metric_types/
• https://prometheus.io/docs/concepts/metric_types/
• https://prometheus.io/docs/practices/histograms/
• https://github.com/prometheus/prom2json
• audit Domain #634

Tasks

1. Figure out data structure for storing metrics compatible with our KV RocksDB.
2. Amend event handlers in Audit domain to allow for insertion of metric data points based on events.
3. ...

[CMCDragonkai]

To be precise, we won't use Prometheus software at all, in fact use "Open Metrics" format as that's compatible with Prometheus metrics.

However we won't even bother with Open Metrics format in PK RPC, we continue to expose data as normal as JSON.

It's only PKNS that will expose Open Metrics format which then has to be stored in Mimir.

When PKNS queries the database from Mimir, it still isn't Open Metrics format, it's actually just JSON again. That data is then read by PKND and loaded into ChartJS.

[CMCDragonkai]

So the only time Open Metrics format is used at all, would be just when PKNS reads PK metrics data and puts it into Mimir. That's the only time the format appears.

However audit metrics and operational metrics should only produce those 4 types of data: gauges, counters, histograms and summaries.

Create functions that accumulate such data - and create the relevant schemas in the audit sublevel that can store such things.