Sync to upstream Prometheus' HEAD #817

julienduchesne · 2025-01-08T19:22:51Z

No description provided.

# Conflicts: # tsdb/docs/format/wal.md

The segment size was too low for the additional NHCB data, thus it created more segments then expected. This meant that less were in the lower numbered segments, which meant more was kept. FAIL: TestCheckpoint (4.05s) FAIL: TestCheckpoint/compress=none (0.22s) checkpoint_test.go:361: Error Trace: /home/krajo/go/github.com/prometheus/prometheus/tsdb/wlog/checkpoint_test.go:361 Error: "0.8586956521739131" is not less than "0.8" Test: TestCheckpoint/compress=none Signed-off-by: György Krajcsovits <[email protected]>

Signed-off-by: György Krajcsovits <[email protected]>

Remove the 2 minute timeout as the default is 2 hours and wouldn't interfere. With the test. Otherwise the extra samples combined with race detection can push the test over 2 minutes and make it fail. Signed-off-by: György Krajcsovits <[email protected]>

Signed-off-by: György Krajcsovits <[email protected]>

…erge speed We need to create more postings entries so the merger has some work to do. Not material for the regexp ones as they match so few series. Signed-off-by: Bryan Boreham <[email protected]>

As long as we run small and big sizes, we don't need all the sizes inbetween. Signed-off-by: Bryan Boreham <[email protected]>

Now we can call it with more specific types which is more efficient than making everything go through the `Postings` interface. Benchmark the concrete type. Signed-off-by: Bryan Boreham <[email protected]>

This enables it to take advantage of a more compact data structure since all postings are known to be `*ListPostings`. Remove the `Get` member which was not used for anything else, and fix up tests. Signed-off-by: Bryan Boreham <[email protected]>

It was bumped during 3.0 with the adoption of log/slog and other dep updates. ``` ~/go/src/github.com/prometheus/prometheus (main [ ]) -> grep '^go' go.mod go 1.22.0 ``` Signed-off-by: TJ Hoplock <[email protected]>

fix(main.go): avoid closing the query engine until it is guaranteed to no longer be in use.

…733) Signed-off-by: Arve Knudsen <[email protected]>

TSDB: Optimization: Merge postings using concrete type

Signed-off-by: György Krajcsovits <[email protected]>

# Conflicts: # tsdb/tsdbutil/histogram.go

prometheus/prometheus#15467 (comment) Signed-off-by: György Krajcsovits <[email protected]>

Signed-off-by: Bryan Boreham <[email protected]>

Signed-off-by: Arve Knudsen <[email protected]>

feat(nhcb): support custom buckets in native histograms in the WAL/WBL

…l (#15786) Signed-off-by: Hélia Barroso <[email protected]>

…racters (#15664) BuildCompliantName was renamed to BuildCompliantMetricName, and it no longer takes UTF8 support into consideration. It focuses on building a metric name that follows Prometheus conventions. A new function, BuildMetricName, was added to optionally add unit and type suffixes to OTLP metric names without translating any characters to underscores(_).

* `RuleConcurrencyController`: Add `SplitGroupIntoBatches` method The concurrency implementation can now return a slice of concurrent rule batches This allows for additional concurrency as opposed to the current interface which is limited by the order in which the rules have been loaded Also, I removed the `concurrencyController` attribute from the group. That information is duplicated in the opts.RuleConcurrencyController` attribute, leading to some confusing behavior, especially in tests. Signed-off-by: Julien Duchesne <[email protected]> * Address PR comments Signed-off-by: Julien Duchesne <[email protected]> * Apply suggestions from code review Co-authored-by: gotjosh <[email protected]> Signed-off-by: Julien Duchesne <[email protected]> --------- Signed-off-by: Julien Duchesne <[email protected]> Signed-off-by: Julien Duchesne <[email protected]> Co-authored-by: gotjosh <[email protected]>

* Rules: Store dependencies instead of boolean To improve prometheus/prometheus#15681 further, we'll need to store the dependencies and dependents of each Right now, if a rule has both (at least 1) dependents and dependencies, it is not possible to determine the order to run the rules and they must all run sequentially This PR only changes the dependents and dependencies attributes of rules, it does not implement a new topological sort algorithm Signed-off-by: Julien Duchesne <[email protected]> * Store a slice of Rule instead Signed-off-by: Julien Duchesne <[email protected]> * Add `BenchmarkRuleDependencyController_AnalyseRules` for future reference Signed-off-by: Julien Duchesne <[email protected]> --------- Signed-off-by: Julien Duchesne <[email protected]>

This commit introduced two field in `/status` endpoint: - The node currently serving the request. - The current server time for debugging time drift issues. fixes #15394. Signed-off-by: sujal shah <[email protected]>

api: Add two new fields Hostname and ServerTime.

Signed-off-by: beorn7 <[email protected]>

tsdb: expose remaining ListPostings Length Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]>

docs: fix spelling

Besides making eval_ordered ignore annotations, this does the following: - Adds a test to verify that eval_ordered indeed ignores an info annotations now, while eval complains about it, eval_info recognizes it and, eval_warn flags the missing of the warn annotation. - Refactors the annotation check into its own method. - Moves closing of the query to the appropriate place where it wasn't so far. Signed-off-by: beorn7 <[email protected]>

This also improves the documentation in the following ways: - Clarifies that `eval` requires no annotations. - Clarifies that `eval_ordered` ignores annotations. - Clarifies that `eval_ordered` does not work with matrix returns (which could very well be created by instant queries). - Clarifies that there are more `eval` commands than just `eval`. - Improves wording for `eval_ordered`. - Replaces `...` by the typographical correct `…`. - Fixes a numerical error in an example. Signed-off-by: beorn7 <[email protected]>

promqltest: let eval_ordered ignore annotations and improve documentation

This test was added in the Grafana fork a while ago: #714 and has been helpful to make sure we can safely terminate rule evaluations early The new rule evaluation logic (done here: prometheus/prometheus#15681) does not have the bug, but the test was useful to verify that Signed-off-by: Julien Duchesne <[email protected]>

CLAassistant · 2025-01-08T19:23:03Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
5 out of 11 committers have signed the CLA.

✅ krajorama
✅ bboreham
✅ carrieedwards
✅ beorn7
✅ aknuds1
❌ tjhop
❌ heliapb
❌ juliusv
❌ sujalshah-bit
❌ bwplotka
❌ yeya24
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

…chesne/upstream-pt3

carrieedwards and others added 30 commits December 5, 2024 09:21

Handle custom buckets in WAL and WBL

aa144b7

Use histogram records for custom value handling

6d413fa

Remove references to custom values record

cfcd515

Attempt for record type

37df50a

Add separate handling for histograms and custom bucket histograms

454f6d3

Rename old histogram record type, use old names for new records

6684344

Update WAL doc to include native histogram encodings

f8a3976

Fix comment and histogram record string

6b44c14

Extend tsdb agent tests with custom bucket histograms

45944c1

Use new record type only for NHCB

a046417

Fix test

1933ccc

Merge branch 'main' into cedwards/nhcb-wal-wbl

efdd088

# Conflicts: # tsdb/docs/format/wal.md

fix(lint): linter errors

8f572fe

Signed-off-by: György Krajcsovits <[email protected]>

fix(test): if we are dereferencing a slice we should check its len

07276ae

Signed-off-by: György Krajcsovits <[email protected]>

Benchmark encoding classic and nhcb

d64d1c4

Signed-off-by: György Krajcsovits <[email protected]>

Fix lint errors

fdb1516

Signed-off-by: György Krajcsovits <[email protected]>

Fix unused import

cf36792

Signed-off-by: György Krajcsovits <[email protected]>

Fix lint for real

df88de5

Signed-off-by: György Krajcsovits <[email protected]>

TSDB: extend BenchmarkMemPostings_PostingsForLabelMatching to check m…

e630ffd

…erge speed We need to create more postings entries so the merger has some work to do. Not material for the regexp ones as they match so few series. Signed-off-by: Bryan Boreham <[email protected]>

TSDB BenchmarkMerge: run fewer sizes

1b22242

As long as we run small and big sizes, we don't need all the sizes inbetween. Signed-off-by: Bryan Boreham <[email protected]>

TSDB: Make mergedPostings generic

0a8779f

Now we can call it with more specific types which is more efficient than making everything go through the `Postings` interface. Benchmark the concrete type. Signed-off-by: Bryan Boreham <[email protected]>

Update benchmark test and comment

1508149

docs: update required go version in readme to 1.22 (#15447)

4cbd9ff

It was bumped during 3.0 with the adoption of log/slog and other dep updates. ``` ~/go/src/github.com/prometheus/prometheus (main [ ]) -> grep '^go' go.mod go 1.22.0 ``` Signed-off-by: TJ Hoplock <[email protected]>

Merge pull request #15723 from machine424/eng-clos

2dfb4fd

fix(main.go): avoid closing the query engine until it is guaranteed to no longer be in use.

OTLP receiver: Optimize by initializing regexps at program start (#15…

f37b5ad

…733) Signed-off-by: Arve Knudsen <[email protected]>

Merge pull request #14518 from bboreham/faster-listpostings-merge

096e2aa

TSDB: Optimization: Merge postings using concrete type

record_test.go: avoid captures, simply return test refs

a7ccc8e

Signed-off-by: György Krajcsovits <[email protected]>

krajorama and others added 18 commits January 2, 2025 12:50

Merge branch 'main' into cedwards/nhcb-wal-wbl

1e420ef

# Conflicts: # tsdb/tsdbutil/histogram.go

perf(nhcbparse): unroll recursion (#15776)

cfcb00a

prometheus/prometheus#15467 (comment) Signed-off-by: György Krajcsovits <[email protected]>

Merge 3.1 into main (#15775)

a6947a0

Signed-off-by: Bryan Boreham <[email protected]>

template: Use cases.Title instead of deprecated strings.Title (#15721)

4f67a38

Signed-off-by: Arve Knudsen <[email protected]>

Merge pull request #15467 from prometheus/cedwards/nhcb-wal-wbl

a441ad7

feat(nhcb): support custom buckets in native histograms in the WAL/WBL

[Docs] Note that scrape_timeout cannot be greater than scrape_interva…

5609419

…l (#15786) Signed-off-by: Hélia Barroso <[email protected]>

api: Add two new fields Node and ServerTime.

73a3438

This commit introduced two field in `/status` endpoint: - The node currently serving the request. - The current server time for debugging time drift issues. fixes #15394. Signed-off-by: sujal shah <[email protected]>

Merge pull request #15784 from sujalshah-bit/15394_server_name_and_time

18bb8bf

api: Add two new fields Hostname and ServerTime.

docs: fix spelling

df55e53

Signed-off-by: beorn7 <[email protected]>

Expose ListPostings Length via Len() method (#15678)

919a5b6

tsdb: expose remaining ListPostings Length Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]>

Merge pull request #15789 from prometheus/beorn7/doc2

02501e0

docs: fix spelling

Merge pull request #15795 from prometheus/beorn7/promql

1ea9b72

promqltest: let eval_ordered ignore annotations and improve documentation

julienduchesne force-pushed the julienduchesne/upstream-pt3 branch from f1280c8 to 092e23f Compare January 8, 2025 19:24

julienduchesne changed the title ~~Julienduchesne/upstream pt3~~ Sync to upstream Prometheus' HEAD Jan 8, 2025

julienduchesne force-pushed the julienduchesne/upstream-pt3 branch from 092e23f to e53efdc Compare January 8, 2025 19:30

Merge commit 'a768a3b95e65efd0338cb2db598191ecd117f362' into juliendu…

635f79f

…chesne/upstream-pt3

julienduchesne force-pushed the julienduchesne/upstream-pt3 branch from e53efdc to 635f79f Compare January 8, 2025 19:53

julienduchesne closed this Jan 9, 2025

julienduchesne deleted the julienduchesne/upstream-pt3 branch January 9, 2025 14:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sync to upstream Prometheus' HEAD #817

Sync to upstream Prometheus' HEAD #817

julienduchesne commented Jan 8, 2025

CLAassistant commented Jan 8, 2025 •

edited

Loading

Sync to upstream Prometheus' HEAD #817

Sync to upstream Prometheus' HEAD #817

Conversation

julienduchesne commented Jan 8, 2025

CLAassistant commented Jan 8, 2025 • edited Loading

CLAassistant commented Jan 8, 2025 •

edited

Loading