SC metrics

[Mikwiss]

Hi @jnioche !

Thanks for your great job on SC 1 and SC 2 !

I'm currently working on SC2 and i have two questions about metrics.

• The metrics provided by bolts are registered with a deprecated method (context.registerMetric(...)) and are used by a MetricsConsumer. Apache Storm v1.2 introduce metrics V2 API. Why don't use this way to provide metrics ? Lack of time ?
  Edit : found a answer here Move to metrics v2 #585

• To store metrics, SC use ElasticSearch. Did you have a technical/specific reason to not use a monitoring system like Prometheus or Graphite ?

PS : The next step of my customer's project is to set up a apache strom cluster (test mode)with a few topologies, i'll report all issues we will found.

Thanks for all.

[jnioche]

bonjour @Mikwiss

Thanks for your questions, very timely and relevant!

The metrics provided by bolts are registered with a deprecated method (context.registerMetric(...)) and are used by a MetricsConsumer. Apache Storm v1.2 introduce metrics V2 API. Why don't use this way to provide metrics ? Lack of time ?

See #585
No one has found it crucial enough to invest time in implementing it, would be a nice contribution. At a glance, V2 seems more restrictive than V1 and I am not sure whether all the metrics we provide could be implemented in v2.

To store metrics, SC use ElasticSearch.

No, we provide a way of doing it with ES but that doesn't mean you have to use it, you can use whichever connector works for Storm.

Did you have a technical/specific reason to not use a monitoring system like Prometheus or Graphite ?

No. Personally I am a fan of Grafana and would like to use it with Prometheus. The reason I implemented a connector for ES was that we often use it with Storm and with Kibana it offers a nice way of displaying the metrics. But as I pointed out, this is by no means the only way of doing.

See https://twitter.com/digitalpebble/status/1385153811220676608

PS : The next step of my customer's project is to set up a apache strom cluster (test mode)with a few topologies, i'll report all issues we will found.

That would be great, thanks! The 2.x branch is likely to become the master one soon, best to work on that even though 1.x might be more stable.

Best of luck with your project.

[Mikwiss]

Thanks for your (very) quick reply @jnioche !

Personally I am a fan of Grafana and would like to use it with Prometheus.

I'm a fan of Grafana + Prometheus too. That's why, i'm thinking how to integrate Prometheus to Apache Storm and SC.

For now, I can't work on the metrics V2 implementation. So I created a MetricConsumer which forward all SC metrics to registered Codehale metrics. With some configuration and a PrometheusStormReporter (ScheduledStormReporter implementation) it's "working" for the simplest counter.

V2 seems more restrictive than V1 and I am not sure whether all the metrics we provide could be implemented in v2.

That's my main problem, i'm working for a workaround...

I'll keep you in touch if I can work on the metrics V2 implementation, that's a better solution according to me.