This repository has been archived by the owner on Apr 2, 2024. It is now read-only.
Adds support for Prometheus metric rollups #1799
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Harkishen-Singh
added
the
epic/auto-rollups
Harkishen-Singh
force-pushed
the
feature_metric_rollup
branch
3 times, most recently
from
933cabf
to
e1b7371
Compare
Harkishen-Singh
force-pushed
the
feature_metric_rollup
branch
from
fa50345
to
a2eba1d
Compare
Harkishen-Singh
commented
Jan 11, 2023
| @@ -0,0 +1,66 @@ | |||
| package database | |||
|
|
|||
There was a problem hiding this comment.
To reviewers: This file is yet to be reviewed. I had to merge the PR quickly that involved this file, hence.
Harkishen-Singh
changed the title
Signed-off-by: Harkishen-Singh <[email protected]> This commit adds support for creation or deletion of metric rollups by reading the dataset-config file. The dataset config file contains a new field `downsample:` which supports 2 options: `automatic` & `resolution`. If `automatic` is false, then connector updates in the database so that `_prom_catalog.scan_for_new_rollups` does not create any new rollups (Caggs). `resolution` is a comma separated list of resolutions for downsampling, in a way that label:resolution:retention is the format that is needed. Example: `short:5m:90d,long:1h:365d` -> The resolution works in a strict manner. Its aim is to make sure that the database contains exactly those resolutions that are mentioned in the dataset config. Like, if the database already contains `short` and `long` downsampled data and the dataset is supplied with `short:5m:90d,very_long:1d:365d` then `long` downsampled case is deleted and `very_long` is created. -> `resolution` is applied if `automatic` is `true`. If automatic is not given in dataset config, it defaults to `true` based on our plan to keep downsampling applied as a default setting.
…ollups. Signed-off-by: Harkishen-Singh <[email protected]>
Signed-off-by: Harkishen-Singh <[email protected]>
Signed-off-by: Harkishen-Singh <[email protected]>
Signed-off-by: Harkishen-Singh <[email protected]>
Signed-off-by: Harkishen-Singh <[email protected]>
Signed-off-by: Harkishen-Singh <[email protected]>
Signed-off-by: Harkishen-Singh <[email protected]>
Signed-off-by: Harkishen-Singh <[email protected]>
Signed-off-by: Harkishen-Singh <[email protected]> This commit implements the update UX for creating downsampled data for Prometheus metrics. Now, downsampling can be done by ``` metrics: downsampling: 5m:30d,1h:365d ``` The above dataset-config setting will create 2 downsampling, ds_5m (interval 5m & retention 30d) and ds_1h (interval 1h & retention 365d). If an entry is removed, like ``` metrics: downsampling: 1h:365d ``` Then we disable the ds_5m downsampling from refreshing. Deletion can only happen via SQL, `CALL _prom_catalog.delete_downsampling(<schema_name>)`. This keeps the UX clean and simple. If the removed dowmsapling is applied again later on, we simply enable the refreshing. This also means that this commit adds ability to enable or disable downsampling as well.
…mn__. Signed-off-by: Harkishen-Singh <[email protected]>
Signed-off-by: Harkishen-Singh <[email protected]>
Signed-off-by: Harkishen-Singh <[email protected]>
Harkishen-Singh
force-pushed
the
feature_metric_rollup
branch
from
a2eba1d
to
c334d34
Compare
niksajakovljevic
suggested changes
Jan 11, 2023
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
TODOs:
Action items to be implemented before marking this PR ready for review:
__schema__&__column__labelsMerge requirements
Please take into account the following non-code changes that you may need to make with your PR: