add prometheus metrics #514

ASuciuX · 2025-04-01T18:56:48Z

resolves expand prometheus metrics to cover all indexers #415

Ordinals

Health Metrics

update_chain_tip_distance
record_processing_error

Performance Metrics

record_block_processing_time
record_inscription_parsing_time
record_ordinal_computation_time
record_db_write_time

Volumetric Metrics

record_inscriptions_in_block
record_brc20_operations_in_block

BRC-20 Specific Metrics

record_brc20_deploy
record_brc20_mint
record_brc20_transfer
record_brc20_transfer_send

Runes

Health Metrics

update_chain_tip_distance
record_processing_error

Performance Metrics

record_block_processing_time
record_rune_parsing_time
record_rune_computation_time
record_db_write_time

Volumetric Metrics

record_runes_in_block
record_rune_operations_in_block

Runes Specific Metrics

record_rune_etching
record_rune_mint
record_rune_transfer
record_rune_burn

github-actions · 2025-04-01T18:57:59Z

Vercel deployment URL: https://bitcoin-indexer-ajnyrbzi5-hirosystems.vercel.app 🚀

codecov · 2025-04-01T18:58:44Z

Codecov Report

Attention: Patch coverage is 52.38095% with 80 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...c/core/pipeline/processors/inscription_indexing.rs	0.00%	35 Missing ⚠️
components/runes/src/db/index.rs	0.00%	23 Missing ⚠️
components/runes/src/utils/monitoring.rs	80.70%	11 Missing ⚠️
components/ordinals/src/utils/monitoring.rs	83.72%	7 Missing ⚠️
components/runes/src/lib.rs	0.00%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

CharlieC3

Thanks for this! Looking forward to exposing this data. Left some questions/comments.

components/runes/src/utils/monitoring.rs

CharlieC3 · 2025-04-02T18:49:21Z

components/runes/src/utils/monitoring.rs

+    pub rune_parsing_time: Histogram,
+    pub rune_computation_time: Histogram,


If I understand this correctly, parsing and computation describe two phases involved in processing a rune, is that correct? If so, are there any other phases we should track?

Yes, that's correct. I am looking into it and will note the possible phases there might be. The db_write_time is also part of this, I appended rune_ to that as well.

components/ordinals/src/utils/monitoring.rs

components/runes/src/utils/monitoring.rs

ASuciuX · 2025-04-03T16:17:32Z

components/ordinals/src/core/pipeline/processors/inscription_indexing.rs

+    // Update cache size metric
+    prometheus.metrics_update_cache_size(cache_l2.len() as u64);
+
+    // Memory usage metric - include all caches
+    let mut total_memory = cache_l2.len() as f64 * 0.1; // L2 cache estimate
+    total_memory += cache_l1.len() as f64 * 0.05; // L1 cache estimate
+    if brc20_cache.is_some() {
+        // Add BRC20 cache memory estimate based on config's lru_cache_size
+        let lru_size = config
+            .ordinals_brc20_config()
+            .map(|c| c.lru_cache_size)
+            .unwrap_or(0);
+        total_memory += lru_size as f64 * 0.02; // Estimate based on configured cache size
+    }
+    prometheus.metrics_update_memory_usage(total_memory);


Is this the best way to track memory/cache, or are there other resources we should be looking at for metrics?

Hmm I don't think this would be a good way because you'll only be measuring memory on this thread IIUC... There should be a way to sample the VM's memory use but that should already be covered by the grafana graphs (is that right @CharlieC3 ?)

components/runes/src/db/cache/index_cache.rs

ASuciuX · 2025-04-03T16:18:58Z

components/runes/src/db/index.rs

+    // TODO: is there no way to have processing errors as we have on ordinals?
+    // if so, delete the processing errors metrics


TBD before the review is finalized

rafaelcr · 2025-04-03T22:11:23Z

components/ordinals/src/core/pipeline/processors/inscription_indexing.rs

+    prometheus
+        .metrics_record_block_processing_time(process_start_time.elapsed().as_millis() as f64);


This measurement should occur once per block so we can measure it both by block (with a label for block height or something) and as a histogram to check stats on overall block processing

I reviewed the Prometheus documentation, and it seems that using labels for each block_height isn’t recommended, as it would create a new time series for every block, leading to high cardinality issues. Unless I’m misunderstanding something, this approach isn’t feasible for Prometheus:

Each labelset is an additional time series that has RAM, CPU, disk, and network costs. Usually the overhead is negligible, but in scenarios with lots of metrics and hundreds of labelsets across hundreds of servers, this can add up quickly.

As a general guideline, try to keep the cardinality of your metrics below 10, and for metrics that exceed that, aim to limit them to a handful across your whole system. The vast majority of your metrics should have no labels.

If you have a metric that has a cardinality over 100 or the potential to grow that large, investigate alternate solutions such as reducing the number of dimensions or moving the analysis away from monitoring and to a general-purpose processing system.

rafaelcr · 2025-04-03T22:15:27Z

components/ordinals/src/core/pipeline/processors/inscription_indexing.rs

+    // Update cache size metric
+    prometheus.metrics_update_cache_size(cache_l2.len() as u64);
+
+    // Memory usage metric - include all caches
+    let mut total_memory = cache_l2.len() as f64 * 0.1; // L2 cache estimate
+    total_memory += cache_l1.len() as f64 * 0.05; // L1 cache estimate
+    if brc20_cache.is_some() {
+        // Add BRC20 cache memory estimate based on config's lru_cache_size
+        let lru_size = config
+            .ordinals_brc20_config()
+            .map(|c| c.lru_cache_size)
+            .unwrap_or(0);
+        total_memory += lru_size as f64 * 0.02; // Estimate based on configured cache size
+    }
+    prometheus.metrics_update_memory_usage(total_memory);


Hmm I don't think this would be a good way because you'll only be measuring memory on this thread IIUC... There should be a way to sample the VM's memory use but that should already be covered by the grafana graphs (is that right @CharlieC3 ?)

rafaelcr · 2025-04-08T16:02:52Z

api/ordinals/src/api/middleware/metrics.ts

+import { FastifyInstance, FastifyReply, FastifyRequest } from 'fastify';
+import { ApiMetrics } from '../../metrics/metrics';
+
+export function registerMetricsMiddleware(fastify: FastifyInstance, metrics: ApiMetrics) {


Hey Alin, apologies, I should've explained this yesterday. The prom-client already takes care of per-request timing measurements and all kinds of default stats, so you shouldn't worry about adding these to your metrics.

The same goes for DB queries, because those get measured by another tool we use called PgHero.

You should only focus on making sure the prometheus metrics are being applied correctly to this API and the Runes API via the use of prom-client.

add(ordinals+brc20): prometheus metrics

6bad93c

add(runes): prometheus monitoring indexer

254abf4

ASuciuX changed the title ~~add(ordinals+brc20): prometheus metrics~~ add prometheus metrics Apr 2, 2025

CharlieC3 reviewed Apr 2, 2025

View reviewed changes

ASuciuX commented Apr 3, 2025

View reviewed changes

components/runes/src/db/cache/index_cache.rs Outdated Show resolved Hide resolved

ASuciuX commented Apr 3, 2025

View reviewed changes

rafaelcr reviewed Apr 3, 2025

View reviewed changes

ASuciuX and others added 8 commits April 4, 2025 14:22

renaming

b72e8b6

rune renaming db_write

4fd2173

remove resource metrics

71ad77b

Merge branch 'develop' into feat/prometheus-metrics

30a3cf9

remove processing errors

5eba757

update prometheus process block

9983a69

update histograms buckets for time related operations

c80de35

define new api ordinals metrics

a56da32

rafaelcr reviewed Apr 8, 2025

View reviewed changes

ASuciuX added 2 commits April 11, 2025 14:49

api: remove middleware and update version metrics

dbbc16c

small fixes

e65ca32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add prometheus metrics #514

add prometheus metrics #514

Uh oh!

ASuciuX commented Apr 1, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Apr 1, 2025 •

edited

Loading

Uh oh!

codecov bot commented Apr 1, 2025 •

edited

Loading

Uh oh!

CharlieC3 left a comment

Uh oh!

Uh oh!

Uh oh!

CharlieC3 Apr 2, 2025

Uh oh!

ASuciuX Apr 4, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

ASuciuX Apr 3, 2025

Uh oh!

rafaelcr Apr 3, 2025

Uh oh!

Uh oh!

ASuciuX Apr 3, 2025

Uh oh!

rafaelcr Apr 3, 2025

Uh oh!

ASuciuX Apr 7, 2025

Uh oh!

rafaelcr Apr 3, 2025

Uh oh!

rafaelcr Apr 8, 2025

Uh oh!

Uh oh!

		pub rune_parsing_time: Histogram,
		pub rune_computation_time: Histogram,

		// TODO: is there no way to have processing errors as we have on ordinals?
		// if so, delete the processing errors metrics

		prometheus
		.metrics_record_block_processing_time(process_start_time.elapsed().as_millis() as f64);

add prometheus metrics #514

Are you sure you want to change the base?

add prometheus metrics #514

Uh oh!

Conversation

ASuciuX commented Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ordinals

Health Metrics

Performance Metrics

Volumetric Metrics

BRC-20 Specific Metrics

Runes

Health Metrics

Performance Metrics

Volumetric Metrics

Runes Specific Metrics

Uh oh!

github-actions bot commented Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

CharlieC3 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ASuciuX Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ASuciuX commented Apr 1, 2025 •

edited

Loading

github-actions bot commented Apr 1, 2025 •

edited

Loading

codecov bot commented Apr 1, 2025 •

edited

Loading

ASuciuX Apr 4, 2025 •

edited

Loading