Threads blocked at RemoteJWKSet.java:433

[giuliodema]

Describe the bug
Spring Cloud microservice, with spring-security-oauth2-jose:6.0.8 with nimbus-jose-jwt:9.24.4
Sometimes (i.e. in the last 12 hours it happened 7 times), we are experiencing an increment of the java thread in state blocked, with the following stacktrace, taken from a thread dump generated during the issue:
"qtp1120531149-1608" - Thread t@1608 java.lang.Thread.State: BLOCKED at app//com.nimbusds.jose.jwk.source.RemoteJWKSet.get(RemoteJWKSet.java:433) - waiting to lock <2e2f22fa> (a com.nimbusds.jose.jwk.source.RemoteJWKSet) owned by "qtp1120531149-1460" t@1460 at app//com.nimbusds.jose.proc.JWSVerificationKeySelector.selectJWSKeys(JWSVerificationKeySelector.java:157) at app//com.nimbusds.jwt.proc.DefaultJWTProcessor.selectKeys(DefaultJWTProcessor.java:283) at app//com.nimbusds.jwt.proc.DefaultJWTProcessor.process(DefaultJWTProcessor.java:354) at app//com.nimbusds.jwt.proc.DefaultJWTProcessor.process(DefaultJWTProcessor.java:303) at app//org.springframework.security.oauth2.jwt.NimbusJwtDecoder.createJwt(NimbusJwtDecoder.java:154) at app//org.springframework.security.oauth2.jwt.NimbusJwtDecoder.decode(NimbusJwtDecoder.java:137) at app//it.eutelsat.mcm.spring.security.SwitchClientIdAuthenticationProvider.getJwt(SwitchClientIdAuthenticationProvider.java:70) at app//it.eutelsat.mcm.spring.security.SwitchClientIdAuthenticationProvider.authenticate(SwitchClientIdAuthenticationProvider.java:44) at app//org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:182) at app//org.springframework.security.oauth2.server.resource.web.authentication.BearerTokenAuthenticationFilter.doFilterInternal(BearerTokenAuthenticationFilter.java:137) at app//org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at app//org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:187) at app//org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:174) at app//org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:135) at app//org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107) at app//org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93) at app//org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:187) at app//org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:174) at app//org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:135) at app//org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) at app//org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) at app//org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at app//org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:187) at app//org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:174) at app//org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:135) at app//org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82) at app//org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69) at app//org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:187) at app//org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:174) at app//org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:135) at app//org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62) at app//org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at app//org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:187) at app//org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:174) at app//org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:135) at app//org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) at app//org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at app//org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:187) at app//org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter$$Lambda/0x00007f303757b560.doFilter(Unknown Source) at app//org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$0(ObservationFilterChainDecorator.java:270) at app//org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation$$Lambda/0x00007f303757b7a0.doFilter(Unknown Source) at app//org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:171) at app//org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:135) at app//org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233) at app//org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191) at app//org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:362) at app//org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:278) at app//org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205) at app//org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586) at app//org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at app//org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at app//org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205) at app//org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586) at app//org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at app//org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at app//org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205) at app//org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586) at app//org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:113) at app//org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at app//org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205) at app//org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586) at app//org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at app//org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at app//org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205) at app//org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586) at app//org.springframework.web.filter.ForwardedHeaderFilter.doFilterInternal(ForwardedHeaderFilter.java:173) at app//org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at app//org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205) at app//org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586) at app//org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1547) at app//org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:819) at app//org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:436) at app//org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:464) at app//org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575) at app//org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717) at app//org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1060) at app//org.eclipse.jetty.server.Server.handle(Server.java:182) at app//org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:662) at app//org.eclipse.jetty.util.thread.Invocable$ReadyTask.run(Invocable.java:168) at app//org.eclipse.jetty.http2.server.internal.HttpStreamOverHTTP2$1.run(HttpStreamOverHTTP2.java:135) at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478) at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441) at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293) at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195) at app//org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:210) at app//org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:157) at app//org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:442) at app//org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99) at app//org.eclipse.jetty.io.ssl.SslConnection$SslEndPoint.onFillable(SslConnection.java:575) at app//org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:390) at app//org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:150) at app//org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99) at app//org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478) at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441) at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293) at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201) at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164) at [email protected]/java.lang.Thread.runWith(Thread.java:1596) at [email protected]/java.lang.Thread.run(Thread.java:1583)

The problem last 3-4 minutes, and during this period all the autenticated API calls received are stuck.
This is a graph of the jetty thread when the problem happens:

Honestly I'm not able to reproduce it systematically, but I'm pretty sure (after further analisys) that the problem occurs when there are a considerable number of bearers cached.