MonitoringHandler blocking EventLoop

[hiddenalpha]

(internal reference for resource planning: SDCISA-16194)

Just seen in our logs. Didn't have time yet to further investigate. Seen with houston 02.01.52.01-SNAPSHOT.

2024-06-07T11:29:44,484 snapshot houston WARN BlockedThreadChecker - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 98 ms
io.vertx.core.VertxException: Thread blocked
        at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1012) ~[?:?]
        at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006) ~[?:?]
        at io.vertx.core.impl.VertxImpl.scheduleTimeout(VertxImpl.java:657) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.impl.VertxImpl.scheduleTimeout(VertxImpl.java:676) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.impl.VertxImpl.setTimer(VertxImpl.java:443) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.eventbus.impl.ReplyHandler.<init>(ReplyHandler.java:42) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.eventbus.impl.EventBusImpl.createReplyHandler(EventBusImpl.java:408) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.eventbus.impl.EventBusImpl.request(EventBusImpl.java:126) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.eventbus.EventBus.request(EventBus.java:106) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.eventbus.EventBus.request(EventBus.java:85) ~[vertx-core-4.5.1.jar:4.5.1]
        at org.swisspush.gateleen.monitoring.MonitoringHandler.updateLastUsedQueueSizeInformation(MonitoringHandler.java:389) ~[gateleen-monitoring-2.1.6.jar:?]
        at org.swisspush.gateleen.queue.queuing.QueueClient.lambda$enqueue$4(QueueClient.java:219) ~[gateleen-queue-2.1.6.jar:?]
        at org.swisspush.gateleen.queue.queuing.QueueClient$$Lambda$2648/0x00000008409b4440.handle(Unknown Source) ~[?:?]
        at io.vertx.core.impl.future.FutureImpl$4.onSuccess(FutureImpl.java:176) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:60) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.impl.future.FutureBase$$Lambda$1475/0x00000008405e9840.run(Unknown Source) ~[?:?]
        at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:305) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.impl.DuplicatedContext.execute(DuplicatedContext.java:181) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:57) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:246) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.Promise.complete(Promise.java:66) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.eventbus.impl.ReplyHandler.dispatch(ReplyHandler.java:97) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.eventbus.impl.HandlerRegistration$InboundDeliveryContext.execute(HandlerRegistration.java:137) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.eventbus.impl.DeliveryContextBase.next(DeliveryContextBase.java:80) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.eventbus.impl.DeliveryContextBase.dispatch(DeliveryContextBase.java:43) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.eventbus.impl.HandlerRegistration.dispatch(HandlerRegistration.java:98) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.eventbus.impl.ReplyHandler.doReceive(ReplyHandler.java:81) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.eventbus.impl.HandlerRegistration.lambda$receive$0(HandlerRegistration.java:49) ~[vertx-core-4.5.1.jar:4.5.1]
        at io.vertx.core.eventbus.impl.HandlerRegistration$$Lambda$1769/0x00000008407ed440.run(Unknown Source) ~[?:?]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.103.Final.jar:4.1.103.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.103.Final.jar:4.1.103.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.103.Final.jar:4.1.103.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[netty-transport-4.1.103.Final.jar:4.1.103.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.103.Final.jar:4.1.103.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.103.Final.jar:4.1.103.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.103.Final.jar:4.1.103.Final]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]

[hiddenalpha]

Unlikely that this is a bug on its own. I suspect it to be a side effect of https://github.com/swisspost/vertx-redisques/issues/#194, https://github.com/swisspost/vertx-redisques/issues/#199, https://github.com/swisspost/vertx-redisques/issues/#188, https://github.com/swisspost/vertx-redisques/issues/#186, https://github.com/swisspost/vertx-redisques/issues/#177, https://github.com/swisspost/vertx-redisques/issues/#175, https://github.com/swisspost/vertx-redisques/issues/#169 and/or https://github.com/swisspost/vertx-redisques/issues/#163. Therefore closing it for now.