Estimating how many messages are left to consume

[VitorVilela7]

Hi! I currently wrote a project which constantly consumers message at very high throughput. I noticed that unlike regular queues, there is not an easy way to figure out the current "lag" i.e. how many messages the consumer is behind to process - the web management UI only show which consumers are connected on - which is understandable given the differences to regular queues.

So I thought of using the Java client to get the last message offset and then subtract with the current processing message, but looks like there is no such option and that would be cool specially for monitoring application consumer performance. Any clue if such thing is possible under the current version? Maybe getting the first or last message offset?

For now, I'm using the time spent since the message was queued, but having an estimate amount of messages behind consumption would be cool.

Thanks in advance!

[acogoluegnes]

Sorry for the late reply, somehow this went under the radar :-(

What would need this for exactly? (you mentioned it'd be "cool", but I'd like to know the exact reason :-) )

This is metrics is likely to change a lot (fast publishers, fast consumer) and will not be always relevant: some consumers may start at the beginning of a stream and stop at some point, because they are meant to process some part of the data, not everything. For such cases, the metrics would not mean anything. This is why I'd like to know more about your use case.

Note the REST API provides the "offset lag": the difference between the offset of the last message dispatched to the consumer and the last stored offset. So this is only for consumers that use offset tracking. This is not exactly what you're asking for, so this is just FYI.

[VitorVilela7]

No worries! I ended up leaving the project which I was working on but the idea was basically being able to monitor and generate alarms (through logging) when the 'lag' (amount of messages left to be processed) gets higher than a certain threshold so a sysadmin could look what's going on with the service. I ended up using the message age as reference in the project which worked as monitoring for the specific case.