From 1d313cca5d5a07d17d87b89c1ed68b17eedf745f Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Sat, 14 Dec 2024 21:46:43 +0530 Subject: [PATCH 01/22] feat: init ct proposal Signed-off-by: Manik Rana --- ...created-timestamps-for-openmetrics-text.md | 57 +++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md new file mode 100644 index 0000000..8cbe7b0 --- /dev/null +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -0,0 +1,57 @@ +# Update Created Timestamps syntax for OpenMetrics + +* **Owners:** + * Manik Rana [@Maniktherana](https://github.com/Maniktherana) + * Arthur Silva Sens [@ArthurSens](https://github.com/ArthurSens) + * Bartłomiej Płotka [@bwplotka](https://github.com/bwplotka) + +* **Implementation Status:** Not implemented + +* **Related Issues and PRs:** + * [github.com/prometheus/prometheus/issues/14217](https://github.com/prometheus/prometheus/issues/14217) + * [github.com/prometheus/prometheus/issues/14823](https://github.com/prometheus/prometheus/issues/14823) + +* **Other docs or links:** + * [github.com/prometheus/proposals/blob/main/proposals/2023-06-13_created-timestamp.md](https://github.com/prometheus/proposals/blob/main/proposals/2023-06-13_created-timestamp.md) + +> TL;DR: We propose an updated syntax for handling created timestamps in the OpenMetrics exposition format. This syntax allows for more efficient parsing of created lines and eliminates confusion on naming metrics that support `_created` lines by placing the created timestamp inline with the metric it is attached to. + +## Why + +Created Timestamps (CTs) were proposed in the summer of 2023 with support for the OpenMetrics (OM) exposition format following in summer of 2024. Once CT lines could be parsed by Prometheus in OM text it felt as though the syntax could be improved upon to optimize how the parser picks up these CTs. CT lines can be characterized as followed: + +* They are represented as a standalone line in the OM text exposition format +* It is denoted with a metric name and `_created` suffix +* It can appear immediately after its associated metric line or between other metric lines and can be placed several lines apart while sharing labels where applicable. + +These characterisitics, specifically the final one means that the parser must search or "peek" ahead to find the `_created` line for a given metric with the same label set and thus, requires additional CPU/memory resources when it can be saved. + +This search operation can be specifically taxing when the CT line is the very last line in a large MetricFamily such as that of a histogram. + +### Pitfalls of the current solution + +As stated above, `_created` lines can appear anywhere after its associated metric line. This means parser must is required to store the current position of the lexer before we start searching for the created timestamp. We arrived at a somewhat acceptable implementation that caches the timestamp and minimizes data stored in memory. Before, we used to make a deep copy of the parser at every line whenever the `CreatedTimestamp` function was called which lead to [Prometheus consuming Gigabytes more memory](https://github.com/prometheus/prometheus/issues/14808) than needed. + +## Goals + +* Prometheus can efficiently parse Created Timestamps without peeking forward. +* OpenMetrics has an updated specification with a clear and precise syntax. + +### Audience + +* Developers maintaining the OpenMetrics text parser. +* Users that utilize the OpenMetrics Text format with Created Timestamps enabled. + +## Non-Goals + +* Update other aspects of OpenMetrics. While this can be a step towards an OM 2.0 this proposal only deals with created timestamps which is small subset of the specification. +* Storing CTs in metadata storage or the WAL (we're only dealing with parsing here). +* Add CT support to additional metric types like guages. + +## How + +## Alternatives + +## Action Plan + + From 9531e661c8a542a11aa83c4d1ac3c558c9af12a5 Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Sat, 14 Dec 2024 21:52:39 +0530 Subject: [PATCH 02/22] chore: minor updates to content before "How" section Signed-off-by: Manik Rana --- ...date-created-timestamps-for-openmetrics-text.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index 8cbe7b0..5f728a9 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -8,13 +8,13 @@ * **Implementation Status:** Not implemented * **Related Issues and PRs:** - * [github.com/prometheus/prometheus/issues/14217](https://github.com/prometheus/prometheus/issues/14217) - * [github.com/prometheus/prometheus/issues/14823](https://github.com/prometheus/prometheus/issues/14823) + * [github.com/prometheus/prometheus/issues/14217](https://github.com/prometheus/prometheus/issues/14217) + * [github.com/prometheus/prometheus/issues/14823](https://github.com/prometheus/prometheus/issues/14823) * **Other docs or links:** - * [github.com/prometheus/proposals/blob/main/proposals/2023-06-13_created-timestamp.md](https://github.com/prometheus/proposals/blob/main/proposals/2023-06-13_created-timestamp.md) + * [github.com/prometheus/proposals/blob/main/proposals/2023-06-13_created-timestamp.md](https://github.com/prometheus/proposals/blob/main/proposals/2023-06-13_created-timestamp.md) -> TL;DR: We propose an updated syntax for handling created timestamps in the OpenMetrics exposition format. This syntax allows for more efficient parsing of created lines and eliminates confusion on naming metrics that support `_created` lines by placing the created timestamp inline with the metric it is attached to. +> TL;DR: We propose an updated syntax for handling created timestamps in the OpenMetrics exposition format. This syntax allows for more efficient parsing of created lines and eliminates confusion on naming metrics that support `_created` lines by placing the created timestamp inline with the metric it is attached to. ## Why @@ -26,11 +26,11 @@ Created Timestamps (CTs) were proposed in the summer of 2023 with support for th These characterisitics, specifically the final one means that the parser must search or "peek" ahead to find the `_created` line for a given metric with the same label set and thus, requires additional CPU/memory resources when it can be saved. -This search operation can be specifically taxing when the CT line is the very last line in a large MetricFamily such as that of a histogram. +This search operation can be specifically taxing when the CT line, if it exists, is the very last line in a large MetricFamily such as that of a histogram with many buckets. ### Pitfalls of the current solution -As stated above, `_created` lines can appear anywhere after its associated metric line. This means parser must is required to store the current position of the lexer before we start searching for the created timestamp. We arrived at a somewhat acceptable implementation that caches the timestamp and minimizes data stored in memory. Before, we used to make a deep copy of the parser at every line whenever the `CreatedTimestamp` function was called which lead to [Prometheus consuming Gigabytes more memory](https://github.com/prometheus/prometheus/issues/14808) than needed. +As stated above, `_created` lines can appear anywhere after its associated metric line. This means parser is required to store the current position of the lexer before we start searching for the created timestamp. Currently, we cache the timestamp and minimize data stored in memory. Before, we used to make a deep copy of the parser at every line whenever the `CreatedTimestamp` function was called which lead to [Prometheus consuming Gigabytes more memory](https://github.com/prometheus/prometheus/issues/14808) than needed. ## Goals @@ -44,7 +44,7 @@ As stated above, `_created` lines can appear anywhere after its associated metr ## Non-Goals -* Update other aspects of OpenMetrics. While this can be a step towards an OM 2.0 this proposal only deals with created timestamps which is small subset of the specification. +* Require backwards compatibility. While this can be a step towards an OM 2.0 this proposal only deals with created timestamps which is small subset of the specification. * Storing CTs in metadata storage or the WAL (we're only dealing with parsing here). * Add CT support to additional metric types like guages. From 768d1f8091fbdcb9e66de5c9f64e7a7b288b43ae Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Sat, 14 Dec 2024 22:11:09 +0530 Subject: [PATCH 03/22] feat: add an "Alternatives" section Signed-off-by: Manik Rana --- ...created-timestamps-for-openmetrics-text.md | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index 5f728a9..9422e39 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -52,6 +52,31 @@ As stated above, `_created` lines can appear anywhere after its associated metr ## Alternatives +### Do nothing + +Continue using the OM text format in its current state and further optimize CPU usage through PRs that build on [github.com/prometheus/prometheus/issues/14823](https://github.com/prometheus/prometheus/issues/14823) . This is desirable if we wish to keep backwards compatibility but we would have to live with an inefficient solution. + +### Storing CTs Using a `# HELP`-Like Syntax + +In addition to the `TYPE`, `UNIT`, and `HELP` fields, we can introduce a `# CREATED` line for metrics that have an associated creation timestamp (CT). This approach allows us to quickly determine whether a CT exists for a given metric, eliminating the need for a more time-consuming search process. By parsing the `# CREATED` line, we can associate it with a specific hash corresponding to the metric's label set, thereby mapping each CT to the correct metric. + +However, this method still involves the overhead of storing the CT until we encounter the relevant metric line. A more efficient solution would be to place the CT inline with the metric itself, streamlining the process and reducing the need for intermediate storage. + +Furthermore the `CREATED` line itself might look somewhat convoluted compared to `TYPE` , `UNIT` , and `HELP` which are very human readable. This new `CREATED` line can end up looking something like this: + +``` +# HELP foo Counter with and without labels to certify CT is parsed for both cases +# TYPE foo counter +# CREATED 1520872607.123; {a="b"} 1520872607.123 +foo_total 17.0 1520879607.789 # {id="counter-test"} 5 +foo_total{a="b"} 17.0 1520879607.789 # {id="counter-test"} 5 +foo_total{le="c"} 21.0 +foo_created{le="c"} 1520872621.123 +foo_total{le="1"} 10.0 +``` + +When the same MetricFamily has multiple label sets with their own CTs we'd have to cram all the timestamps with the additional labels with a delimiter in between. + ## Action Plan From be917f1147fa197bc6fd7837757fb6c7c8426f35 Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Sat, 14 Dec 2024 22:15:47 +0530 Subject: [PATCH 04/22] feat: add action plan Signed-off-by: Manik Rana --- ...24-12-14_update-created-timestamps-for-openmetrics-text.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index 9422e39..fab537e 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -79,4 +79,8 @@ When the same MetricFamily has multiple label sets with their own CTs we'd have ## Action Plan +In no particular order: +* [ ] Update the OpenMetrics specification to include the new syntax for Created Timestamps. +* [ ] Update Prometheus's OpenMetrics text parser to support the new syntax. +* [ ] Update tooling and libraries that expose OpenMetrics text. From 6f7889d49cce1e74eca29f5f69699e36f032a640 Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Sat, 14 Dec 2024 22:18:03 +0530 Subject: [PATCH 05/22] chore: update audience Signed-off-by: Manik Rana --- .../2024-12-14_update-created-timestamps-for-openmetrics-text.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index fab537e..1553743 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -40,6 +40,7 @@ As stated above, `_created` lines can appear anywhere after its associated metr ### Audience * Developers maintaining the OpenMetrics text parser. +* Developers maintaining client libraries. * Users that utilize the OpenMetrics Text format with Created Timestamps enabled. ## Non-Goals From e24e60de6f2cee727d768c898141d0af721a484b Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Sat, 14 Dec 2024 22:21:42 +0530 Subject: [PATCH 06/22] fix: alternative example Signed-off-by: Manik Rana --- ...024-12-14_update-created-timestamps-for-openmetrics-text.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index 1553743..b3590a1 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -68,11 +68,10 @@ Furthermore the `CREATED` line itself might look somewhat convoluted compared t ``` # HELP foo Counter with and without labels to certify CT is parsed for both cases # TYPE foo counter -# CREATED 1520872607.123; {a="b"} 1520872607.123 +# CREATED 1520872607.123; {a="b"} 1520872607.123; {le="c"} 1520872621.123 foo_total 17.0 1520879607.789 # {id="counter-test"} 5 foo_total{a="b"} 17.0 1520879607.789 # {id="counter-test"} 5 foo_total{le="c"} 21.0 -foo_created{le="c"} 1520872621.123 foo_total{le="1"} 10.0 ``` From 73a57fa7eba19e549534a97a2f3b73c8fb511aca Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Sat, 14 Dec 2024 23:00:54 +0530 Subject: [PATCH 07/22] feat: add a how section Signed-off-by: Manik Rana --- ...created-timestamps-for-openmetrics-text.md | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index b3590a1..6116da3 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -51,6 +51,67 @@ As stated above, `_created` lines can appear anywhere after its associated metr ## How +We store the Created Timestamp inline with the metric it is attached to and we remove `_created` lines. This will allow the parser to immediately associate the CT with the metric without having to search for it. For counters its straightforward as we can just add the timestamp after the metric value like an exemplar. To separate it from a traditional timstamp we can prefix the created timestamp with something like `ct@`, although this portion of the syntax is not final and can be changed. Furthermore, we can order the created timestamp such that it is after the metric value + timestamp and before an exemplar. + +Lets look at some examples to illustrate the difference. + +### Counters + +``` +# HELP foo Counter with and without labels to certify CT is parsed for both cases +# TYPE foo counter +foo_total 17.0 1520879607.789 ct@1520872607.123 # {id="counter-test"} 5 +foo_total{a="b"} 17.0 1520879607.789 ct@1520872607.123 # {id="counter-test"} 5 +foo_total{le="c"} 21.0 ct1520872621.123 +foo_total{le="1"} 10.0 +``` + +vs the current syntax: + +``` +# HELP foo Counter with and without labels to certify CT is parsed for both cases +# TYPE foo counter +foo_total 17.0 1520879607.789 # {id="counter-test"} 5 +foo_created 1520872607.123 +foo_total{a="b"} 17.0 1520879607.789 # {id="counter-test"} 5 +foo_created{a="b"} 1520872607.123 +foo_total{le="c"} 21.0 +foo_created{le="c"} 1520872621.123 +foo_total{le="1"} 10.0 +``` + +### Summaries and Histograms + +Summaries and histograms are a bit more complex as they have quantiles and buckets respectively. Moreoever, there is no defacto line like in a counter where we can place the CT. Thus, we can opt to place the CT on the first line of the metric with the same label set. We can then cache this timestamp with a hash of the label set and use it for all subsequent lines with the same label set. This is something we somewhat do already with the current syntax. + +A diff example (for brevity) of a summary metric with current vs proposed syntax: + +```diff +# HELP rpc_durations_seconds RPC latency distributions. +# TYPE rpc_durations_seconds summary ++rpc_durations_seconds{service="exponential",quantile="0.5"} 7.689368882420941e-07 ct@1.7268398130168908e+09 +-rpc_durations_seconds{service="exponential",quantile="0.5"} 7.689368882420941e-07 +rpc_durations_seconds{service="exponential",quantile="0.9"} 1.6537614174305048e-06 +rpc_durations_seconds{service="exponential",quantile="0.99"} 2.0965499063061924e-06 +rpc_durations_seconds_sum{service="exponential"} 2.0318666372575776e-05 +rpc_durations_seconds_count{service="exponential"} 22 +-rpc_durations_seconds_created{service="exponential"} 1.7268398130168908e+09 +``` + +With the current syntax `_created` line can be anywhere after the `quantile` lines. A histogram would look similar to the summary but with `le` labels instead of `quantile` labels. + +Another option is to simply place the CT on every line of a summary or histogram metric. This would be more verbose but would be more explicit and easier to parse and avoids storing the CT completely: + +``` +# HELP rpc_durations_seconds RPC latency distributions. +# TYPE rpc_durations_seconds summary ++rpc_durations_seconds{service="exponential",quantile="0.5"} 7.689368882420941e-07 ct@1.7268398130168908e+09 +rpc_durations_seconds{service="exponential",quantile="0.9"} 1.6537614174305048e-06 ct@1.7268398130168908e+09 +rpc_durations_seconds{service="exponential",quantile="0.99"} 2.0965499063061924e-06 ct@1.7268398130168908e+09 +rpc_durations_seconds_sum{service="exponential"} 2.0318666372575776e-05 ct@1.7268398130168908e+09 +rpc_durations_seconds_count{service="exponential"} 22 ct@1.7268398130168908e+09 +``` + ## Alternatives ### Do nothing From 696b466b8798d4b1e1182684a5d52e2acd9d7744 Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Sat, 14 Dec 2024 23:16:03 +0530 Subject: [PATCH 08/22] feat: updates and add a backwards compatibilty section Signed-off-by: Manik Rana --- ...te-created-timestamps-for-openmetrics-text.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index 6116da3..f89f322 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -45,8 +45,8 @@ As stated above, `_created` lines can appear anywhere after its associated metr ## Non-Goals -* Require backwards compatibility. While this can be a step towards an OM 2.0 this proposal only deals with created timestamps which is small subset of the specification. -* Storing CTs in metadata storage or the WAL (we're only dealing with parsing here). +* Require backwards compatibility. While this can be a step towards an OM 2.0 this proposal only deals with created timestamps which is a small subset of the specification. +* Dealing with the storage of CTs. * Add CT support to additional metric types like guages. ## How @@ -112,19 +112,23 @@ rpc_durations_seconds_sum{service="exponential"} 2.0318666372575776e-05 ct@1.726 rpc_durations_seconds_count{service="exponential"} 22 ct@1.7268398130168908e+09 ``` +### Backwards compatibility and semantic versioning + +This change is not backwards compatible and would break existing parsers that expect the `_created` line. OpenMetrics 1.x parsers that support `_created` lines would not be able to parse the new syntax. This would require a new major version of the OpenMetrics specification to be released, i.e, OpenMetrics 2.0. Any client libraries or tools that expose OpenMetrics text would also need to be updated to support the new syntax. + ## Alternatives ### Do nothing -Continue using the OM text format in its current state and further optimize CPU usage through PRs that build on [github.com/prometheus/prometheus/issues/14823](https://github.com/prometheus/prometheus/issues/14823) . This is desirable if we wish to keep backwards compatibility but we would have to live with an inefficient solution. +Continue using the OM text format in its current state and further optimize CPU usage through PRs that build on [github.com/prometheus/prometheus/issues/14823](https://github.com/prometheus/prometheus/issues/14823). This is desirable if we wish to keep backwards compatibility and avoids breaking changes but we would have to live with an inefficient solution. ### Storing CTs Using a `# HELP`-Like Syntax In addition to the `TYPE`, `UNIT`, and `HELP` fields, we can introduce a `# CREATED` line for metrics that have an associated creation timestamp (CT). This approach allows us to quickly determine whether a CT exists for a given metric, eliminating the need for a more time-consuming search process. By parsing the `# CREATED` line, we can associate it with a specific hash corresponding to the metric's label set, thereby mapping each CT to the correct metric. -However, this method still involves the overhead of storing the CT until we encounter the relevant metric line. A more efficient solution would be to place the CT inline with the metric itself, streamlining the process and reducing the need for intermediate storage. +However, this method still involves the overhead of storing potentially multiple CTs in memory until we encounter the relevant metric line. Furthermore the `CREATED` line itself might look somewhat convoluted compared to `TYPE` , `UNIT` , and `HELP` which are very human readable. -Furthermore the `CREATED` line itself might look somewhat convoluted compared to `TYPE` , `UNIT` , and `HELP` which are very human readable. This new `CREATED` line can end up looking something like this: +An example of how this might look: ``` # HELP foo Counter with and without labels to certify CT is parsed for both cases @@ -136,7 +140,7 @@ foo_total{le="c"} 21.0 foo_total{le="1"} 10.0 ``` -When the same MetricFamily has multiple label sets with their own CTs we'd have to cram all the timestamps with the additional labels with a delimiter in between. +When the same MetricFamily has multiple label sets with their own CTs we'd have to place all the timestamps in a single line with the additional labels and separate them with delimiters. This does mitigate the increased verbosity described in the [Summaries and Histograms section](#summaries-and-histograms) but it still requires storing all the timestamps in memory for the lifetime of the MetricFamily. Even if we store the CT only for the first label set in a MetricFamily, we only need to keep a single timestamp in memory at a time for each label set in a summary or histogram until the next label set is encountered. ## Action Plan From bbf069f0b2b1abed901b465c48799c7a1f1bfd40 Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Sat, 14 Dec 2024 23:25:36 +0530 Subject: [PATCH 09/22] chore: minor fixes Signed-off-by: Manik Rana --- ...24-12-14_update-created-timestamps-for-openmetrics-text.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index f89f322..9349221 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -82,7 +82,7 @@ foo_total{le="1"} 10.0 ### Summaries and Histograms -Summaries and histograms are a bit more complex as they have quantiles and buckets respectively. Moreoever, there is no defacto line like in a counter where we can place the CT. Thus, we can opt to place the CT on the first line of the metric with the same label set. We can then cache this timestamp with a hash of the label set and use it for all subsequent lines with the same label set. This is something we somewhat do already with the current syntax. +Summaries and histograms are a bit more complex as they have quantiles and buckets respectively. Moreoever, there is no defacto line like in a counter where we can place the CT. Thus, we can opt to place the CT on the first line of the metric with the same label set. We can then cache this timestamp with a hash of the label set and use it for all subsequent lines with the same label set. This is something we already do with the current syntax. A diff example (for brevity) of a summary metric with current vs proposed syntax: @@ -105,7 +105,7 @@ Another option is to simply place the CT on every line of a summary or histogram ``` # HELP rpc_durations_seconds RPC latency distributions. # TYPE rpc_durations_seconds summary -+rpc_durations_seconds{service="exponential",quantile="0.5"} 7.689368882420941e-07 ct@1.7268398130168908e+09 +rpc_durations_seconds{service="exponential",quantile="0.5"} 7.689368882420941e-07 ct@1.7268398130168908e+09 rpc_durations_seconds{service="exponential",quantile="0.9"} 1.6537614174305048e-06 ct@1.7268398130168908e+09 rpc_durations_seconds{service="exponential",quantile="0.99"} 2.0965499063061924e-06 ct@1.7268398130168908e+09 rpc_durations_seconds_sum{service="exponential"} 2.0318666372575776e-05 ct@1.7268398130168908e+09 From 5f789505f33729dbc7d01cb41510c6e75bb149f6 Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Sat, 14 Dec 2024 23:27:13 +0530 Subject: [PATCH 10/22] chore: add an assumption Signed-off-by: Manik Rana --- ...2024-12-14_update-created-timestamps-for-openmetrics-text.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index 9349221..9a272a1 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -140,7 +140,7 @@ foo_total{le="c"} 21.0 foo_total{le="1"} 10.0 ``` -When the same MetricFamily has multiple label sets with their own CTs we'd have to place all the timestamps in a single line with the additional labels and separate them with delimiters. This does mitigate the increased verbosity described in the [Summaries and Histograms section](#summaries-and-histograms) but it still requires storing all the timestamps in memory for the lifetime of the MetricFamily. Even if we store the CT only for the first label set in a MetricFamily, we only need to keep a single timestamp in memory at a time for each label set in a summary or histogram until the next label set is encountered. +When the same MetricFamily has multiple label sets with their own CTs we'd have to place all the timestamps in a single line with the additional labels and separate them with delimiters. This does mitigate the increased verbosity described in the [Summaries and Histograms section](#summaries-and-histograms) but it still requires storing all the timestamps in memory for the lifetime of the MetricFamily. Even if we store the CT only for the first label set in a MetricFamily, we only need to keep a single timestamp in memory at a time for each label set in a summary or histogram until the next label set is encountered (assuming metrics with the same label sets are grouped together). ## Action Plan From f77c8280484d10b26181a24afad12f38b5b05e73 Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Tue, 17 Dec 2024 22:45:22 +0530 Subject: [PATCH 11/22] fix: punctuation Co-authored-by: Arthur Silva Sens Signed-off-by: Manik Rana --- ...24-12-14_update-created-timestamps-for-openmetrics-text.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index 9a272a1..c16cb51 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -20,8 +20,8 @@ Created Timestamps (CTs) were proposed in the summer of 2023 with support for the OpenMetrics (OM) exposition format following in summer of 2024. Once CT lines could be parsed by Prometheus in OM text it felt as though the syntax could be improved upon to optimize how the parser picks up these CTs. CT lines can be characterized as followed: -* They are represented as a standalone line in the OM text exposition format -* It is denoted with a metric name and `_created` suffix +* They are represented as a standalone line in the OM text exposition format. +* It is denoted with a metric name and `_created` suffix. * It can appear immediately after its associated metric line or between other metric lines and can be placed several lines apart while sharing labels where applicable. These characterisitics, specifically the final one means that the parser must search or "peek" ahead to find the `_created` line for a given metric with the same label set and thus, requires additional CPU/memory resources when it can be saved. From 9b8740744a5c6cad160526799a3412a81f828c9d Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Tue, 17 Dec 2024 22:45:36 +0530 Subject: [PATCH 12/22] fix: wording Co-authored-by: Arthur Silva Sens Signed-off-by: Manik Rana --- ...2024-12-14_update-created-timestamps-for-openmetrics-text.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index c16cb51..20c9015 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -34,7 +34,7 @@ As stated above, `_created` lines can appear anywhere after its associated metr ## Goals -* Prometheus can efficiently parse Created Timestamps without peeking forward. +* Prometheus can parse Created Timestamps without holding state between lines. * OpenMetrics has an updated specification with a clear and precise syntax. ### Audience From a34f6092d5051166ee50dbd16f4c6ebc0b4d3edb Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Tue, 17 Dec 2024 22:46:01 +0530 Subject: [PATCH 13/22] fix: wording Co-authored-by: Arthur Silva Sens Signed-off-by: Manik Rana --- ...2024-12-14_update-created-timestamps-for-openmetrics-text.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index 20c9015..a09c7ff 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -26,7 +26,7 @@ Created Timestamps (CTs) were proposed in the summer of 2023 with support for th These characterisitics, specifically the final one means that the parser must search or "peek" ahead to find the `_created` line for a given metric with the same label set and thus, requires additional CPU/memory resources when it can be saved. -This search operation can be specifically taxing when the CT line, if it exists, is the very last line in a large MetricFamily such as that of a histogram with many buckets. +This search operation can be particularly taxing when the CT line, if it exists, is the very last line in a large MetricFamily, such as a histogram with many buckets. ### Pitfalls of the current solution From aea32d8d6f88048617eb9206f643555f89679978 Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Tue, 17 Dec 2024 22:46:20 +0530 Subject: [PATCH 14/22] fix: grammar I should use grammarly Co-authored-by: Arthur Silva Sens Signed-off-by: Manik Rana --- ...2024-12-14_update-created-timestamps-for-openmetrics-text.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index a09c7ff..b95d706 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -51,7 +51,7 @@ As stated above, `_created` lines can appear anywhere after its associated metr ## How -We store the Created Timestamp inline with the metric it is attached to and we remove `_created` lines. This will allow the parser to immediately associate the CT with the metric without having to search for it. For counters its straightforward as we can just add the timestamp after the metric value like an exemplar. To separate it from a traditional timstamp we can prefix the created timestamp with something like `ct@`, although this portion of the syntax is not final and can be changed. Furthermore, we can order the created timestamp such that it is after the metric value + timestamp and before an exemplar. +We store the Created Timestamp inline with the attached metric and remove `_created` lines. This will allow the parser to immediately associate the CT with the metric without having to search for it. For counters, it's straightforward, as we can just add the timestamp after the metric value like an exemplar. To separate it from a traditional timestamp, we can prefix the created timestamp with something like `ct@`, although this portion of the syntax is not final and can be changed. Furthermore, we can order the created timestamp such that it is after the metric value + timestamp and before an exemplar. Lets look at some examples to illustrate the difference. From 3a4bc9c00f03eee1dd64aad2fa090370bee8dbc9 Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Tue, 17 Dec 2024 22:47:10 +0530 Subject: [PATCH 15/22] fix: example Signed-off-by: Manik Rana --- ...2024-12-14_update-created-timestamps-for-openmetrics-text.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index b95d706..eada769 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -62,7 +62,7 @@ Lets look at some examples to illustrate the difference. # TYPE foo counter foo_total 17.0 1520879607.789 ct@1520872607.123 # {id="counter-test"} 5 foo_total{a="b"} 17.0 1520879607.789 ct@1520872607.123 # {id="counter-test"} 5 -foo_total{le="c"} 21.0 ct1520872621.123 +foo_total{le="c"} 21.0 ct@1520872621.123 foo_total{le="1"} 10.0 ``` From 8f730b70ae8013217dba376b852d070dadf48bbc Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Tue, 17 Dec 2024 22:53:50 +0530 Subject: [PATCH 16/22] fix: order action plan Signed-off-by: Manik Rana --- ...12-14_update-created-timestamps-for-openmetrics-text.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index eada769..8f6a76a 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -144,8 +144,7 @@ When the same MetricFamily has multiple label sets with their own CTs we'd have ## Action Plan -In no particular order: +* [ ] Update tooling and libraries that expose OpenMetrics text. +* [ ] Update Prometheus text parser and benchmark against current implementation. +* [ ] If performance significantly improves, update the OpenMetrics specification to include the new syntax for Created Timestamps. -* [ ] Update the OpenMetrics specification to include the new syntax for Created Timestamps. -* [ ] Update Prometheus's OpenMetrics text parser to support the new syntax. -* [ ] Update tooling and libraries that expose OpenMetrics text. From 509eb0a7127007bb70f7babe996793026ed38f61 Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Wed, 18 Dec 2024 14:47:31 +0530 Subject: [PATCH 17/22] fix: why section Signed-off-by: Manik Rana --- ...2024-12-14_update-created-timestamps-for-openmetrics-text.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index 8f6a76a..413d829 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -18,7 +18,7 @@ ## Why -Created Timestamps (CTs) were proposed in the summer of 2023 with support for the OpenMetrics (OM) exposition format following in summer of 2024. Once CT lines could be parsed by Prometheus in OM text it felt as though the syntax could be improved upon to optimize how the parser picks up these CTs. CT lines can be characterized as followed: +Prometheus integrated Created Timestamps (CTs) into its protobuf format in the summer of 2023, building on the concept originally defined by the OpenMetrics (OM) specification. Support for the OM text exposition format followed in the summer of 2024. Once CT lines could be parsed by Prometheus in OM text it felt as though the syntax could be improved upon to optimize how the parser picks up these CTs. CT lines can be characterized as followed: * They are represented as a standalone line in the OM text exposition format. * It is denoted with a metric name and `_created` suffix. From 0d3028bdd54483716b152581e44bc9c8948c0c7b Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Thu, 19 Dec 2024 22:24:34 +0530 Subject: [PATCH 18/22] fix: contributors Co-authored-by: Bartlomiej Plotka Signed-off-by: Manik Rana --- ...2024-12-14_update-created-timestamps-for-openmetrics-text.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index 413d829..a0e559e 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -2,6 +2,8 @@ * **Owners:** * Manik Rana [@Maniktherana](https://github.com/Maniktherana) + +**Contributors:** * Arthur Silva Sens [@ArthurSens](https://github.com/ArthurSens) * Bartłomiej Płotka [@bwplotka](https://github.com/bwplotka) From c7f95a0f2d75613d1e807f26bbe59b33d5159923 Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Thu, 19 Dec 2024 22:24:53 +0530 Subject: [PATCH 19/22] fix: brevity Signed-off-by: Manik Rana --- ...2024-12-14_update-created-timestamps-for-openmetrics-text.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index a0e559e..4bde2cc 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -32,7 +32,7 @@ This search operation can be particularly taxing when the CT line, if it exists, ### Pitfalls of the current solution -As stated above, `_created` lines can appear anywhere after its associated metric line. This means parser is required to store the current position of the lexer before we start searching for the created timestamp. Currently, we cache the timestamp and minimize data stored in memory. Before, we used to make a deep copy of the parser at every line whenever the `CreatedTimestamp` function was called which lead to [Prometheus consuming Gigabytes more memory](https://github.com/prometheus/prometheus/issues/14808) than needed. +As stated above, `_created` lines can appear anywhere after its associated metric line. This means parser is required to store the current position of the lexer before we start searching for the created timestamp. Currently, we cache the timestamp and minimize data stored in memory but there is still significant CPU overhead. ## Goals From f542e2d232db6e9aca1de25c24abd32236e375f4 Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Thu, 19 Dec 2024 22:30:08 +0530 Subject: [PATCH 20/22] fix: grammar + add CT unit Signed-off-by: Manik Rana --- ...2024-12-14_update-created-timestamps-for-openmetrics-text.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index 4bde2cc..40e958c 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -53,7 +53,7 @@ As stated above, `_created` lines can appear anywhere after its associated metri ## How -We store the Created Timestamp inline with the attached metric and remove `_created` lines. This will allow the parser to immediately associate the CT with the metric without having to search for it. For counters, it's straightforward, as we can just add the timestamp after the metric value like an exemplar. To separate it from a traditional timestamp, we can prefix the created timestamp with something like `ct@`, although this portion of the syntax is not final and can be changed. Furthermore, we can order the created timestamp such that it is after the metric value + timestamp and before an exemplar. +We store the Created Timestamp inline with the attached metric and remove `_created` lines. This will allow the parser to immediately associate the CT with the metric without having to search for it. For counters, it's straightforward, as we can just add the timestamp after the metric value like an exemplar. To separate it from a traditional timestamp, we can prefix the created timestamp with something like `ct@`, although this portion of the syntax is not final and can be changed. We propose to force a certain order of the created timestamp such that it is after the metric value and the timestamp and before an exemplar. This CT itself will be Unix Epoch in seconds, unchanged from the current specification Lets look at some examples to illustrate the difference. From 49045e12436bfa6d3f0a197649495496a4ee295e Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Thu, 19 Dec 2024 22:33:10 +0530 Subject: [PATCH 21/22] feat: update goals Signed-off-by: Manik Rana --- ...024-12-14_update-created-timestamps-for-openmetrics-text.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index 40e958c..491dbca 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -36,8 +36,9 @@ As stated above, `_created` lines can appear anywhere after its associated metri ## Goals -* Prometheus can parse Created Timestamps without holding state between lines. +* Prometheus can parse Created Timestamps more eficiently. * OpenMetrics has an updated specification with a clear and precise syntax. +* OpenMetics doesn't expose artifical metrics for CT semantics. ### Audience From e3182373ec121e6e4bf5d8aa062273e55684663a Mon Sep 17 00:00:00 2001 From: Manik Rana Date: Thu, 19 Dec 2024 22:48:16 +0530 Subject: [PATCH 22/22] feat: update histogram/summary section Signed-off-by: Manik Rana --- ...created-timestamps-for-openmetrics-text.md | 51 ++++++++++++------- 1 file changed, 34 insertions(+), 17 deletions(-) diff --git a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md index 491dbca..70c87be 100644 --- a/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md +++ b/proposals/2024-12-14_update-created-timestamps-for-openmetrics-text.md @@ -85,36 +85,35 @@ foo_total{le="1"} 10.0 ### Summaries and Histograms -Summaries and histograms are a bit more complex as they have quantiles and buckets respectively. Moreoever, there is no defacto line like in a counter where we can place the CT. Thus, we can opt to place the CT on the first line of the metric with the same label set. We can then cache this timestamp with a hash of the label set and use it for all subsequent lines with the same label set. This is something we already do with the current syntax. +Summaries and histograms are a bit more complex as they have quantiles and buckets respectively. Moreoever, there is no defacto line like in a counter where we can place the CT. Thus, we can opt to place the CT on every line of a summary or histogram metric. This would be more explicit, easier to parse, and avoids storing the CT between lines. -A diff example (for brevity) of a summary metric with current vs proposed syntax: +An example of a summary metric with the proposed syntax: + +``` +# HELP rpc_durations_seconds RPC latency distributions. +# TYPE rpc_durations_seconds summary +rpc_durations_seconds{service="exponential",quantile="0.5"} 7.689368882420941e-07 ct@1.7268398130168908e+09 +rpc_durations_seconds{service="exponential",quantile="0.9"} 1.6537614174305048e-06 ct@1.7268398130168908e+09 +rpc_durations_seconds{service="exponential",quantile="0.99"} 2.0965499063061924e-06 ct@1.7268398130168908e+09 +rpc_durations_seconds_sum{service="exponential"} 2.0318666372575776e-05 ct@1.7268398130168908e+09 +rpc_durations_seconds_count{service="exponential"} 22 ct@1.7268398130168908e+09 +``` + +vs current syntax: ```diff # HELP rpc_durations_seconds RPC latency distributions. # TYPE rpc_durations_seconds summary -+rpc_durations_seconds{service="exponential",quantile="0.5"} 7.689368882420941e-07 ct@1.7268398130168908e+09 --rpc_durations_seconds{service="exponential",quantile="0.5"} 7.689368882420941e-07 +rpc_durations_seconds{service="exponential",quantile="0.5"} 7.689368882420941e-07 rpc_durations_seconds{service="exponential",quantile="0.9"} 1.6537614174305048e-06 rpc_durations_seconds{service="exponential",quantile="0.99"} 2.0965499063061924e-06 rpc_durations_seconds_sum{service="exponential"} 2.0318666372575776e-05 ++rpc_durations_seconds_created{service="exponential"} 1.7268398130168908e+09 rpc_durations_seconds_count{service="exponential"} 22 --rpc_durations_seconds_created{service="exponential"} 1.7268398130168908e+09 ``` With the current syntax `_created` line can be anywhere after the `quantile` lines. A histogram would look similar to the summary but with `le` labels instead of `quantile` labels. -Another option is to simply place the CT on every line of a summary or histogram metric. This would be more verbose but would be more explicit and easier to parse and avoids storing the CT completely: - -``` -# HELP rpc_durations_seconds RPC latency distributions. -# TYPE rpc_durations_seconds summary -rpc_durations_seconds{service="exponential",quantile="0.5"} 7.689368882420941e-07 ct@1.7268398130168908e+09 -rpc_durations_seconds{service="exponential",quantile="0.9"} 1.6537614174305048e-06 ct@1.7268398130168908e+09 -rpc_durations_seconds{service="exponential",quantile="0.99"} 2.0965499063061924e-06 ct@1.7268398130168908e+09 -rpc_durations_seconds_sum{service="exponential"} 2.0318666372575776e-05 ct@1.7268398130168908e+09 -rpc_durations_seconds_count{service="exponential"} 22 ct@1.7268398130168908e+09 -``` - ### Backwards compatibility and semantic versioning This change is not backwards compatible and would break existing parsers that expect the `_created` line. OpenMetrics 1.x parsers that support `_created` lines would not be able to parse the new syntax. This would require a new major version of the OpenMetrics specification to be released, i.e, OpenMetrics 2.0. Any client libraries or tools that expose OpenMetrics text would also need to be updated to support the new syntax. @@ -125,6 +124,24 @@ This change is not backwards compatible and would break existing parsers that ex Continue using the OM text format in its current state and further optimize CPU usage through PRs that build on [github.com/prometheus/prometheus/issues/14823](https://github.com/prometheus/prometheus/issues/14823). This is desirable if we wish to keep backwards compatibility and avoids breaking changes but we would have to live with an inefficient solution. +### Place the CT next to the first line of a Summary or Histogram metric + +This involves opting the syntax described in the [counters section](#counters) but changing how we handle Summaries and Histograms. Here we place the CT on the first line of the metric with the same label set. This would be more less verbose and is something we already do with the current syntax. We can then cache this timestamp with a hash of the label set and use it for all subsequent lines with the same label set. + +A diff example (for brevity) of a summary metric with current vs this proposed syntax: + +```diff +# HELP rpc_durations_seconds RPC latency distributions. +# TYPE rpc_durations_seconds summary ++rpc_durations_seconds{service="exponential",quantile="0.5"} 7.689368882420941e-07 ct@1.7268398130168908e+09 +-rpc_durations_seconds{service="exponential",quantile="0.5"} 7.689368882420941e-07 +rpc_durations_seconds{service="exponential",quantile="0.9"} 1.6537614174305048e-06 +rpc_durations_seconds{service="exponential",quantile="0.99"} 2.0965499063061924e-06 +rpc_durations_seconds_sum{service="exponential"} 2.0318666372575776e-05 +rpc_durations_seconds_count{service="exponential"} 22 +-rpc_durations_seconds_created{service="exponential"} 1.7268398130168908e+09 +``` + ### Storing CTs Using a `# HELP`-Like Syntax In addition to the `TYPE`, `UNIT`, and `HELP` fields, we can introduce a `# CREATED` line for metrics that have an associated creation timestamp (CT). This approach allows us to quickly determine whether a CT exists for a given metric, eliminating the need for a more time-consuming search process. By parsing the `# CREATED` line, we can associate it with a specific hash corresponding to the metric's label set, thereby mapping each CT to the correct metric.