The broker will die when creating the topic

[henry-cheung]

Problem Description：
Add topic ，Three partitions ，Use the command to create ，The broker will die when creating the topic，command does not work

2023-02-07T10:08:16,507+0800 [pulsar-web-36-8] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:08:16 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.26.0" 0
2023-02-07T10:08:16,513+0800 [prometheus-stats-37-1] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:08:16 +0800] "GET /metrics/ HTTP/1.1" 200 860114 "http://172.19.201.155:8080/metrics" "Prometheus/2.26.0" 6
2023-02-07T10:08:31,194+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.resourcegroup.ResourceGroupNamespaceConfigListener - Metadata store notification: Path /admin/policies/mxsk_data_tenant/cmp_ns, Type Modified
2023-02-07T10:08:31,195+0800 [pulsar-web-36-6] INFO org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [pulsar-admin] Successfully granted access for role mxsk-cms: [consume, produce] - topic persistent://mxsk_data_tenant/cmp_ns/platform_msg_topic-partition-0
2023-02-07T10:08:31,195+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.BrokerService - [mxsk_data_tenant/cmp_ns] updating with Policies(auth_policies=AuthPoliciesImpl(namespaceAuthentication={}, topicAuthentication={persistent://mxsk_data_tenant/cmp_ns/customer_info_topic-partition-0={mxsk-cms=[produce], mxsk-ps=[consume]}, persistent://mxsk_data_tenant/cmp_ns/customer_info_topic-partition-1={mxsk-cms=[produce], mxsk-ps=[consume]}, persistent://mxsk_data_tenant/cmp_ns/customer_info_topic-partition-2={mxsk-cms=[produce], mxsk-ps=[consume]}, persistent://mxsk_data_tenant/cmp_ns/customer_info_topic={mxsk-cms=[produce], mxsk-ps=[consume]}, persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic-partition-0={mxsk-cms=[consume, produce]}, persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic-partition-1={mxsk-cms=[consume, produce]}, persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic-partition-2={mxsk-cms=[consume, produce]}, persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic={mxsk-cms=[consume, produce]}, persistent://mxsk_data_tenant/cmp_ns/station_message_topic-partition-0={mxsk-cms=[consume, produce]}, persistent://mxsk_data_tenant/cmp_ns/station_message_topic-partition-1={mxsk-cms=[consume, produce]}, persistent://mxsk_data_tenant/cmp_ns/pro_station_message_topic-partition-0={mxsk-cms=[consume, produce]}, persistent://mxsk_data_tenant/cmp_ns/pro_station_message_topic-partition-1={mxsk-cms=[consume, produce]}, persistent://mxsk_data_tenant/cmp_ns/station_message_topic-partition-2={mxsk-cms=[consume, produce]}, persistent://mxsk_data_tenant/cmp_ns/station_message_topic={mxsk-cms=[consume, produce]}, persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-0={mxsk-cms=[consume]}, persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-1={mxsk-cms=[consume]}, persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-2={mxsk-cms=[consume]}, persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic={mxsk-cms=[consume]}, persistent://mxsk_data_tenant/cmp_ns/platform_message_topic-partition-0={}, persistent://mxsk_data_tenant/cmp_ns/platform_message_topic-partition-1={}, persistent://mxsk_data_tenant/cmp_ns/platform_msg_topic-partition-0={mxsk-cms=[consume, produce]}}, subscriptionAuthentication={}), replication_clusters=[pulsar-cluster-zk], bundles=BundlesDataImpl(boundaries=[0x00000000, 0x40000000, 0x80000000, 0xc0000000, 0xffffffff], numBundles=4), backlog_quota_map={}, clusterDispatchRate={}, topicDispatchRate={}, subscriptionDispatchRate={}, replicatorDispatchRate={}, clusterSubscribeRate={}, persistence=null, deduplicationEnabled=true, autoTopicCreationOverride=null, autoSubscriptionCreationOverride=null, publishMaxMessageRate={}, latency_stats_sample_rate={}, message_ttl_in_seconds=null, subscription_expiration_time_minutes=null, retention_policies=RetentionPolicies{retentionTimeInMinutes=10080, retentionSizeInMB=-1}, deleted=false, encryption_required=false, delayed_delivery_policies=null, inactive_topic_policies=null, subscription_auth_mode=None, max_producers_per_topic=null, max_consumers_per_topic=null, max_consumers_per_subscription=null, max_unacked_messages_per_consumer=null, max_unacked_messages_per_subscription=null, max_subscriptions_per_topic=null, compaction_threshold=null, offload_threshold=-1, offload_deletion_lag_ms=null, max_topics_per_namespace=null, schema_auto_update_compatibility_strategy=Full, schema_compatibility_strategy=UNDEFINED, is_allow_auto_update_schema=true, schema_validation_enforced=false, offload_policies=null, deduplicationSnapshotIntervalSeconds=null, subscription_types_enabled=[], properties={}, resource_group_name=null)
2023-02-07T10:08:31,196+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.AbstractTopic - Disabling publish throttling for persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-2
2023-02-07T10:08:31,508+0800 [pulsar-web-36-1] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:08:31 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.26.0" 0
2023-02-07T10:08:46,508+0800 [pulsar-web-36-3] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:08:46 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.26.0" 0
2023-02-07T10:08:53,004+0800 [pulsar-load-manager-1-1] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Writing local data to metadata store because maximum change 26.63647792656338% exceeded threshold 10%; time since last report written is 60.0 seconds
2023-02-07T10:09:01,195+0800 [pulsar-web-36-6] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [pulsar-admin] Failed to grant permissions for topic persistent://mxsk_data_tenant/cmp_ns/platform_msg_topic-partition-1
org.apache.pulsar.metadata.api.MetadataStoreException: Failed to set data for /admin/policies/mxsk_data_tenant/cmp_ns
at org.apache.pulsar.broker.resources.BaseResources.set(BaseResources.java:106) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.resources.NamespaceResources.setPolicies(NamespaceResources.java:113) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.grantPermissions(PersistentTopicsBase.java:262) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalGrantPermissionsOnTopic(PersistentTopicsBase.java:291) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.admin.v2.PersistentTopics.grantPermissionsOnTopic(PersistentTopics.java:174) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:67) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.apache.pulsar.broker.web.AuthenticationFilter.doFilter(AuthenticationFilter.java:84) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) [org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555) [org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410) [org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) [org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386) [org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.resources.BaseResources.set(BaseResources.java:101) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
... 74 more
2023-02-07T10:09:01,197+0800 [pulsar-web-36-6] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [07/Feb/2023:10:08:31 +0800] "POST /admin/v2/persistent/mxsk_data_tenant/cmp_ns/platform_msg_topic/permissions/mxsk-cms HTTP/1.1" 500 7235 "-" "Pulsar-Java-v2.9.1" 30030
2023-02-07T10:09:01,197+0800 [metadata-store-6-1] WARN org.apache.pulsar.broker.authorization.AuthorizationService - Consumer-client with Role - mxsk-cms failed to get permissions for topic - persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-2. null
2023-02-07T10:09:01,197+0800 [metadata-store-6-1] WARN org.apache.pulsar.broker.service.Consumer - [mxsk-cms] Get unexpected error while autorizing [persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-2] null
java.util.concurrent.TimeoutException: null
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.authorization.AuthorizationService.canConsume(AuthorizationService.java:247) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.Consumer.checkPermissions(Consumer.java:659) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807) ~[?:?]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$onPoliciesUpdate$108(PersistentTopic.java:2423) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.onPoliciesUpdate(PersistentTopic.java:2422) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$84(BrokerService.java:1830) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$85(BrokerService.java:1830) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753) ~[?:?]
at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731) ~[?:?]
at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$86(BrokerService.java:1825) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$87(BrokerService.java:1821) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$get$7(ZKMetadataStore.java:139) ~[org.apache.pulsar-pulsar-metadata-2.9.1.jar:2.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-02-07T10:09:01,198+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.Consumer - [mxsk-cms] is not allowed to consume from topic [persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-2] anymore
2023-02-07T10:09:01,198+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.Consumer - Disconnecting consumer: Consumer{subscription=PersistentSubscription{topic=persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-2, name=oasPushCoreRiskReport}, consumerId=5, consumerName=6699f, address=/112.74.187.85:48008}
2023-02-07T10:09:01,199+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers - Removed consumer Consumer{subscription=PersistentSubscription{topic=persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-2, name=oasPushCoreRiskReport}, consumerId=5, consumerName=6699f, address=/112.74.187.85:48008} with pending 0 acks
2023-02-07T10:09:01,508+0800 [pulsar-web-36-5] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:09:01 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.26.0" 0
2023-02-07T10:09:01,509+0800 [pulsar-web-36-6] WARN org.apache.pulsar.broker.web.ResponseHandlerFilter - Http request Request[GET //172.19.201.155:8080/metrics/]@10686fe2 async context timeout.
2023-02-07T10:09:01,510+0800 [pulsar-web-36-6] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:08:31 +0800] "GET /metrics/ HTTP/1.1" 500 552 "http://172.19.201.155:8080/metrics" "Prometheus/2.26.0" 30002
2023-02-07T10:09:01,510+0800 [prometheus-stats-37-1] WARN org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read policies data, will apply the default backlog quota: namespace=mxsk_data_tenant/cmp_ns
org.apache.pulsar.metadata.api.MetadataStoreException: Failed to get data from /admin/policies/mxsk_data_tenant/cmp_ns
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:91) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.resources.NamespaceResources.getPolicies(NamespaceResources.java:101) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:82) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.getBacklogQuota(PersistentTopic.java:2461) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.getTopicStats(NamespaceStatsAggregator.java:127) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$0(NamespaceStatsAggregator.java:76) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$1(NamespaceStatsAggregator.java:75) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$2(NamespaceStatsAggregator.java:74) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.generate(NamespaceStatsAggregator.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator.generate(PrometheusMetricsGenerator.java:110) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet.lambda$doGet$0(PrometheusMetricsServlet.java:77) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.2.jar:4.14.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:86) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
... 26 more
2023-02-07T10:09:16,508+0800 [pulsar-web-36-3] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:09:16 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.26.0" 0
2023-02-07T10:09:16,509+0800 [pulsar-web-36-6] WARN org.apache.pulsar.broker.web.ResponseHandlerFilter - Http request Request[GET //172.19.201.155:8080/metrics/]@52d51223 async context timeout.
2023-02-07T10:09:16,510+0800 [pulsar-web-36-6] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:08:46 +0800] "GET /metrics/ HTTP/1.1" 500 552 "http://172.19.201.155:8080/metrics" "Prometheus/2.26.0" 30002
2023-02-07T10:09:19,825+0800 [pulsar-backlog-quota-checker-26-1] WARN org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read policies data, will apply the default backlog quota: namespace=mxsk_data_tenant/cmp_ns
org.apache.pulsar.metadata.api.MetadataStoreException: Failed to get data from /admin/policies/mxsk_data_tenant/cmp_ns
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:91) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.resources.NamespaceResources.getPolicies(NamespaceResources.java:101) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:82) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuotaLimitInSize(BacklogQuotaManager.java:101) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.isSizeBacklogExceeded(PersistentTopic.java:2494) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$72(BrokerService.java:1634) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$71(BrokerService.java:1622) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1620) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1631) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.2.jar:4.14.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:86) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
... 21 more
2023-02-07T10:09:31,200+0800 [metadata-store-6-1] WARN org.apache.pulsar.broker.authorization.AuthorizationService - Consumer-client with Role - mxsk-cms failed to get permissions for topic - persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-2. null
2023-02-07T10:09:31,200+0800 [metadata-store-6-1] WARN org.apache.pulsar.broker.service.Consumer - [mxsk-cms] Get unexpected error while autorizing [persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-2] null
java.util.concurrent.TimeoutException: null
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.authorization.AuthorizationService.canConsume(AuthorizationService.java:247) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.Consumer.checkPermissions(Consumer.java:659) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807) ~[?:?]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$onPoliciesUpdate$108(PersistentTopic.java:2423) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.onPoliciesUpdate(PersistentTopic.java:2422) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$84(BrokerService.java:1830) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$85(BrokerService.java:1830) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753) ~[?:?]
at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731) ~[?:?]
at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$86(BrokerService.java:1825) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$87(BrokerService.java:1821) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$get$7(ZKMetadataStore.java:139) ~[org.apache.pulsar-pulsar-metadata-2.9.1.jar:2.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-02-07T10:09:31,200+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.Consumer - [mxsk-cms] is not allowed to consume from topic [persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-2] anymore
2023-02-07T10:09:31,200+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.Consumer - Disconnecting consumer: Consumer{subscription=PersistentSubscription{topic=persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-2, name=oasPushCoreRiskReport}, consumerId=5, consumerName=3b221, address=/39.108.17.76:6419}
2023-02-07T10:09:31,200+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers - Removed consumer Consumer{subscription=PersistentSubscription{topic=persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-2, name=oasPushCoreRiskReport}, consumerId=5, consumerName=3b221, address=/39.108.17.76:6419} with pending 0 acks
2023-02-07T10:09:31,202+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.AbstractTopic - Disabling publish throttling for persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic-partition-2
2023-02-07T10:09:31,508+0800 [pulsar-web-36-7] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:09:31 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.26.0" 0
2023-02-07T10:09:31,508+0800 [pulsar-web-36-6] WARN org.apache.pulsar.broker.web.ResponseHandlerFilter - Http request Request[GET //172.19.201.155:8080/metrics/]@73b582da async context timeout.
2023-02-07T10:09:31,509+0800 [pulsar-web-36-6] INFO org.eclipse.jetty.server.RequestLog - - - - [07/Feb/2023:10:09:01 +0800] "GET /metrics/ HTTP/1.1" 500 0 "http://172.19.201.155:8080/metrics" "Prometheus/2.26.0" 30001
2023-02-07T10:09:31,510+0800 [prometheus-stats-37-1] WARN org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read policies data, will apply the default backlog quota: namespace=mxsk_data_tenant/cmp_ns
org.apache.pulsar.metadata.api.MetadataStoreException: Failed to get data from /admin/policies/mxsk_data_tenant/cmp_ns
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:91) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.resources.NamespaceResources.getPolicies(NamespaceResources.java:101) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:82) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.getBacklogQuota(PersistentTopic.java:2461) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.getTopicStats(NamespaceStatsAggregator.java:129) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$0(NamespaceStatsAggregator.java:76) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$1(NamespaceStatsAggregator.java:75) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$2(NamespaceStatsAggregator.java:74) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.generate(NamespaceStatsAggregator.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator.generate(PrometheusMetricsGenerator.java:110) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet.lambda$doGet$0(PrometheusMetricsServlet.java:77) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.2.jar:4.14.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:86) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
... 26 more
2023-02-07T10:09:46,508+0800 [pulsar-web-36-8] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:09:46 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.26.0" 0
2023-02-07T10:09:46,509+0800 [pulsar-web-36-3] WARN org.apache.pulsar.broker.web.ResponseHandlerFilter - Http request Request[GET //172.19.201.155:8080/metrics/]@13800470 async context timeout.
2023-02-07T10:09:46,510+0800 [pulsar-web-36-3] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:09:16 +0800] "GET /metrics/ HTTP/1.1" 500 552 "http://172.19.201.155:8080/metrics" "Prometheus/2.26.0" 30002
2023-02-07T10:09:49,826+0800 [pulsar-backlog-quota-checker-26-1] WARN org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read policies data, will apply the default backlog quota: namespace=mxsk_data_tenant/cmp_ns
org.apache.pulsar.metadata.api.MetadataStoreException: Failed to get data from /admin/policies/mxsk_data_tenant/cmp_ns
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:91) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.resources.NamespaceResources.getPolicies(NamespaceResources.java:101) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:82) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuotaLimitInTime(BacklogQuotaManager.java:105) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.isTimeBacklogExceeded(PersistentTopic.java:2517) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$72(BrokerService.java:1637) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$71(BrokerService.java:1622) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1620) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1631) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.2.jar:4.14.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:86) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
... 21 more
2023-02-07T10:10:01,202+0800 [metadata-store-6-1] WARN org.apache.pulsar.broker.authorization.AuthorizationService - Consumer-client with Role - mxsk-cms failed to get permissions for topic - persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic-partition-2. null
2023-02-07T10:10:01,202+0800 [metadata-store-6-1] WARN org.apache.pulsar.broker.service.Consumer - [mxsk-cms] Get unexpected error while autorizing [persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic-partition-2] null
java.util.concurrent.TimeoutException: null
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.authorization.AuthorizationService.canConsume(AuthorizationService.java:247) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.Consumer.checkPermissions(Consumer.java:659) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807) ~[?:?]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$onPoliciesUpdate$108(PersistentTopic.java:2423) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.onPoliciesUpdate(PersistentTopic.java:2422) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$84(BrokerService.java:1830) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$85(BrokerService.java:1830) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753) ~[?:?]
at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731) ~[?:?]
at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$86(BrokerService.java:1825) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$87(BrokerService.java:1821) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$get$7(ZKMetadataStore.java:139) ~[org.apache.pulsar-pulsar-metadata-2.9.1.jar:2.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-02-07T10:10:01,202+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.Consumer - [mxsk-cms] is not allowed to consume from topic [persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic-partition-2] anymore
2023-02-07T10:10:01,202+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.Consumer - Disconnecting consumer: Consumer{subscription=PersistentSubscription{topic=persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic-partition-2, name=mxsk-credit-trade-notice-topic-sub}, consumerId=2, consumerName=DEFAULT_CONSUMER_dev_persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic_370638649129779200, address=/120.24.41.133:40346}
2023-02-07T10:10:01,203+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers - Removed consumer Consumer{subscription=PersistentSubscription{topic=persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic-partition-2, name=mxsk-credit-trade-notice-topic-sub}, consumerId=2, consumerName=DEFAULT_CONSUMER_dev_persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic_370638649129779200, address=/120.24.41.133:40346} with pending 0 acks
2023-02-07T10:10:01,204+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.AbstractTopic - Disabling publish throttling for persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic-partition-0
2023-02-07T10:10:01,508+0800 [pulsar-web-36-3] WARN org.apache.pulsar.broker.web.ResponseHandlerFilter - Http request Request[GET //172.19.201.155:8080/metrics/]@15d93a79 async context timeout.
2023-02-07T10:10:01,509+0800 [pulsar-web-36-6] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:10:01 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.26.0" 0
2023-02-07T10:10:01,510+0800 [pulsar-web-36-6] INFO org.eclipse.jetty.server.RequestLog - - - - [07/Feb/2023:10:09:31 +0800] "GET /metrics/ HTTP/1.1" 500 0 "http://172.19.201.155:8080/metrics" "Prometheus/2.26.0" 30002
2023-02-07T10:10:01,511+0800 [prometheus-stats-37-1] WARN org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read policies data, will apply the default backlog quota: namespace=mxsk_data_tenant/cmp_ns
org.apache.pulsar.metadata.api.MetadataStoreException: Failed to get data from /admin/policies/mxsk_data_tenant/cmp_ns
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:91) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.resources.NamespaceResources.getPolicies(NamespaceResources.java:101) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:82) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.getBacklogQuota(PersistentTopic.java:2461) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.getTopicStats(NamespaceStatsAggregator.java:127) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$0(NamespaceStatsAggregator.java:76) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$1(NamespaceStatsAggregator.java:75) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$2(NamespaceStatsAggregator.java:74) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.generate(NamespaceStatsAggregator.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator.generate(PrometheusMetricsGenerator.java:110) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet.lambda$doGet$0(PrometheusMetricsServlet.java:77) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.2.jar:4.14.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:86) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
... 26 more
2023-02-07T10:10:16,509+0800 [pulsar-web-36-7] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:10:16 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.26.0" 1
2023-02-07T10:10:16,509+0800 [pulsar-web-36-8] WARN org.apache.pulsar.broker.web.ResponseHandlerFilter - Http request Request[GET //172.19.201.155:8080/metrics/]@3c606fcf async context timeout.
2023-02-07T10:10:16,509+0800 [pulsar-web-36-7] INFO org.eclipse.jetty.server.RequestLog - - - - [07/Feb/2023:10:09:46 +0800] "GET /metrics/ HTTP/1.1" 500 0 "http://172.19.201.155:8080/metrics" "Prometheus/2.26.0" 30001
2023-02-07T10:10:19,827+0800 [pulsar-backlog-quota-checker-26-1] WARN org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read policies data, will apply the default backlog quota: namespace=mxsk_data_tenant/cmp_ns
org.apache.pulsar.metadata.api.MetadataStoreException: Failed to get data from /admin/policies/mxsk_data_tenant/cmp_ns
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:91) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.resources.NamespaceResources.getPolicies(NamespaceResources.java:101) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:82) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuotaLimitInSize(BacklogQuotaManager.java:101) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.isSizeBacklogExceeded(PersistentTopic.java:2494) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$72(BrokerService.java:1634) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$71(BrokerService.java:1622) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1620) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1631) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.2.jar:4.14.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:86) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
... 21 more
2023-02-07T10:10:31,204+0800 [metadata-store-6-1] WARN org.apache.pulsar.broker.authorization.AuthorizationService - Consumer-client with Role - mxsk-cms failed to get permissions for topic - persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic-partition-0. null
2023-02-07T10:10:31,204+0800 [metadata-store-6-1] WARN org.apache.pulsar.broker.service.Consumer - [mxsk-cms] Get unexpected error while autorizing [persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic-partition-0] null
java.util.concurrent.TimeoutException: null
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.authorization.AuthorizationService.canConsume(AuthorizationService.java:247) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.Consumer.checkPermissions(Consumer.java:659) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807) ~[?:?]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$onPoliciesUpdate$108(PersistentTopic.java:2423) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.onPoliciesUpdate(PersistentTopic.java:2422) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$84(BrokerService.java:1830) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$85(BrokerService.java:1830) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753) ~[?:?]
at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731) ~[?:?]
at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$86(BrokerService.java:1825) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$87(BrokerService.java:1821) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$get$7(ZKMetadataStore.java:139) ~[org.apache.pulsar-pulsar-metadata-2.9.1.jar:2.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-02-07T10:10:31,204+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.Consumer - [mxsk-cms] is not allowed to consume from topic [persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic-partition-0] anymore
2023-02-07T10:10:31,204+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.Consumer - Disconnecting consumer: Consumer{subscription=PersistentSubscription{topic=persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic-partition-0, name=mxsk-credit-trade-notice-topic-sub}, consumerId=0, consumerName=DEFAULT_CONSUMER_dev_persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic_370638649129779200, address=/120.24.41.133:40346}
2023-02-07T10:10:31,204+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers - Removed consumer Consumer{subscription=PersistentSubscription{topic=persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic-partition-0, name=mxsk-credit-trade-notice-topic-sub}, consumerId=0, consumerName=DEFAULT_CONSUMER_dev_persistent://mxsk_data_tenant/cmp_ns/credit_trade_notice_topic_370638649129779200, address=/120.24.41.133:40346} with pending 0 acks
2023-02-07T10:10:31,205+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.AbstractTopic - Disabling publish throttling for persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-0
2023-02-07T10:10:31,509+0800 [pulsar-web-36-6] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:10:31 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.26.0" 1
2023-02-07T10:10:31,510+0800 [pulsar-web-36-8] WARN org.apache.pulsar.broker.web.ResponseHandlerFilter - Http request Request[GET //172.19.201.155:8080/metrics/]@4eec295b async context timeout.
2023-02-07T10:10:31,511+0800 [pulsar-web-36-8] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:10:01 +0800] "GET /metrics/ HTTP/1.1" 500 552 "http://172.19.201.155:8080/metrics" "Prometheus/2.26.0" 30001
2023-02-07T10:10:31,511+0800 [prometheus-stats-37-1] WARN org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read policies data, will apply the default backlog quota: namespace=mxsk_data_tenant/cmp_ns
org.apache.pulsar.metadata.api.MetadataStoreException: Failed to get data from /admin/policies/mxsk_data_tenant/cmp_ns
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:91) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.resources.NamespaceResources.getPolicies(NamespaceResources.java:101) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:82) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.getBacklogQuota(PersistentTopic.java:2461) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.getTopicStats(NamespaceStatsAggregator.java:129) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$0(NamespaceStatsAggregator.java:76) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$1(NamespaceStatsAggregator.java:75) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$2(NamespaceStatsAggregator.java:74) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.generate(NamespaceStatsAggregator.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator.generate(PrometheusMetricsGenerator.java:110) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet.lambda$doGet$0(PrometheusMetricsServlet.java:77) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.2.jar:4.14.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:86) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
... 26 more
2023-02-07T10:10:46,508+0800 [pulsar-web-36-3] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:10:46 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.26.0" 0
2023-02-07T10:10:46,510+0800 [pulsar-web-36-8] WARN org.apache.pulsar.broker.web.ResponseHandlerFilter - Http request Request[GET //172.19.201.155:8080/metrics/]@6bd0f803 async context timeout.
2023-02-07T10:10:46,510+0800 [pulsar-web-36-8] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:10:16 +0800] "GET /metrics/ HTTP/1.1" 500 552 "http://172.19.201.155:8080/metrics" "Prometheus/2.26.0" 30001
2023-02-07T10:10:49,827+0800 [pulsar-backlog-quota-checker-26-1] WARN org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read policies data, will apply the default backlog quota: namespace=mxsk_data_tenant/cmp_ns
org.apache.pulsar.metadata.api.MetadataStoreException: Failed to get data from /admin/policies/mxsk_data_tenant/cmp_ns
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:91) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.resources.NamespaceResources.getPolicies(NamespaceResources.java:101) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:82) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuotaLimitInTime(BacklogQuotaManager.java:105) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.isTimeBacklogExceeded(PersistentTopic.java:2517) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$72(BrokerService.java:1637) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$71(BrokerService.java:1622) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1620) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1631) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.2.jar:4.14.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:86) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
... 21 more
2023-02-07T10:11:01,205+0800 [metadata-store-6-1] WARN org.apache.pulsar.broker.authorization.AuthorizationService - Consumer-client with Role - mxsk-cms failed to get permissions for topic - persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-0. null
2023-02-07T10:11:01,205+0800 [metadata-store-6-1] WARN org.apache.pulsar.broker.service.Consumer - [mxsk-cms] Get unexpected error while autorizing [persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-0] null
java.util.concurrent.TimeoutException: null
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.authorization.AuthorizationService.canConsume(AuthorizationService.java:247) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.Consumer.checkPermissions(Consumer.java:659) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807) ~[?:?]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$onPoliciesUpdate$108(PersistentTopic.java:2423) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.onPoliciesUpdate(PersistentTopic.java:2422) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$84(BrokerService.java:1830) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$85(BrokerService.java:1830) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753) ~[?:?]
at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731) ~[?:?]
at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$86(BrokerService.java:1825) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$87(BrokerService.java:1821) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$get$7(ZKMetadataStore.java:139) ~[org.apache.pulsar-pulsar-metadata-2.9.1.jar:2.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-02-07T10:11:01,205+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.Consumer - [mxsk-cms] is not allowed to consume from topic [persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-0] anymore
2023-02-07T10:11:01,205+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.Consumer - Disconnecting consumer: Consumer{subscription=PersistentSubscription{topic=persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-0, name=oasPushCoreRiskReport}, consumerId=3, consumerName=6699f, address=/112.74.187.85:48008}
2023-02-07T10:11:01,205+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers - Removed consumer Consumer{subscription=PersistentSubscription{topic=persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-0, name=oasPushCoreRiskReport}, consumerId=3, consumerName=6699f, address=/112.74.187.85:48008} with pending 0 acks
2023-02-07T10:11:01,509+0800 [pulsar-web-36-7] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:11:01 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.26.0" 1
2023-02-07T10:11:01,509+0800 [pulsar-web-36-3] WARN org.apache.pulsar.broker.web.ResponseHandlerFilter - Http request Request[GET //172.19.201.155:8080/metrics/]@815d46f async context timeout.
2023-02-07T10:11:01,510+0800 [pulsar-web-36-3] INFO org.eclipse.jetty.server.RequestLog - - - - [07/Feb/2023:10:10:31 +0800] "GET /metrics/ HTTP/1.1" 500 0 "http://172.19.201.155:8080/metrics" "Prometheus/2.26.0" 30001
2023-02-07T10:11:01,512+0800 [prometheus-stats-37-1] WARN org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read policies data, will apply the default backlog quota: namespace=mxsk_data_tenant/cmp_ns
org.apache.pulsar.metadata.api.MetadataStoreException: Failed to get data from /admin/policies/mxsk_data_tenant/cmp_ns
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:91) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.resources.NamespaceResources.getPolicies(NamespaceResources.java:101) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:82) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.getBacklogQuota(PersistentTopic.java:2461) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.getTopicStats(NamespaceStatsAggregator.java:127) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$0(NamespaceStatsAggregator.java:76) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$1(NamespaceStatsAggregator.java:75) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$2(NamespaceStatsAggregator.java:74) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.generate(NamespaceStatsAggregator.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator.generate(PrometheusMetricsGenerator.java:110) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet.lambda$doGet$0(PrometheusMetricsServlet.java:77) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.2.jar:4.14.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:86) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
... 26 more
2023-02-07T10:11:14,706+0800 [pulsar-web-36-8] INFO org.eclipse.jetty.server.RequestLog - 198.98.57.210 - - [07/Feb/2023:10:11:14 +0800] "GET / HTTP/1.1" 404 491 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" 2
2023-02-07T10:11:16,508+0800 [pulsar-web-36-6] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:11:16 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.26.0" 0
2023-02-07T10:11:16,509+0800 [pulsar-web-36-3] WARN org.apache.pulsar.broker.web.ResponseHandlerFilter - Http request Request[GET //172.19.201.155:8080/metrics/]@40d390b8 async context timeout.
2023-02-07T10:11:16,509+0800 [pulsar-web-36-3] INFO org.eclipse.jetty.server.RequestLog - - - - [07/Feb/2023:10:10:46 +0800] "GET /metrics/ HTTP/1.1" 500 0 "http://172.19.201.155:8080/metrics" "Prometheus/2.26.0" 30000
2023-02-07T10:11:19,827+0800 [pulsar-backlog-quota-checker-26-1] WARN org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read policies data, will apply the default backlog quota: namespace=mxsk_data_tenant/cmp_ns
org.apache.pulsar.metadata.api.MetadataStoreException: Failed to get data from /admin/policies/mxsk_data_tenant/cmp_ns
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:91) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.resources.NamespaceResources.getPolicies(NamespaceResources.java:101) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:82) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuotaLimitInSize(BacklogQuotaManager.java:101) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.isSizeBacklogExceeded(PersistentTopic.java:2494) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$72(BrokerService.java:1634) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$71(BrokerService.java:1622) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1620) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1631) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.2.jar:4.14.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:86) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
... 21 more
2023-02-07T10:11:31,205+0800 [metadata-store-6-1] WARN org.apache.pulsar.broker.authorization.AuthorizationService - Consumer-client with Role - mxsk-cms failed to get permissions for topic - persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-0. null
2023-02-07T10:11:31,205+0800 [metadata-store-6-1] WARN org.apache.pulsar.broker.service.Consumer - [mxsk-cms] Get unexpected error while autorizing [persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-0] null
java.util.concurrent.TimeoutException: null
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.authorization.AuthorizationService.canConsume(AuthorizationService.java:247) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.Consumer.checkPermissions(Consumer.java:659) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807) ~[?:?]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$onPoliciesUpdate$108(PersistentTopic.java:2423) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.onPoliciesUpdate(PersistentTopic.java:2422) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$84(BrokerService.java:1830) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$85(BrokerService.java:1830) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753) ~[?:?]
at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731) ~[?:?]
at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$86(BrokerService.java:1825) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$87(BrokerService.java:1821) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$get$7(ZKMetadataStore.java:139) ~[org.apache.pulsar-pulsar-metadata-2.9.1.jar:2.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-02-07T10:11:31,206+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.Consumer - [mxsk-cms] is not allowed to consume from topic [persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-0] anymore
2023-02-07T10:11:31,206+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.Consumer - Disconnecting consumer: Consumer{subscription=PersistentSubscription{topic=persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-0, name=oasPushCoreRiskReport}, consumerId=3, consumerName=3b221, address=/39.108.17.76:6419}
2023-02-07T10:11:31,206+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers - Removed consumer Consumer{subscription=PersistentSubscription{topic=persistent://mxsk_data_tenant/cmp_ns/sync_core_risk_report_topic-partition-0, name=oasPushCoreRiskReport}, consumerId=3, consumerName=3b221, address=/39.108.17.76:6419} with pending 0 acks
2023-02-07T10:11:31,206+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.AbstractTopic - Disabling publish throttling for persistent://mxsk_data_tenant/cmp_ns/customer_info_topic-partition-2
2023-02-07T10:11:31,508+0800 [pulsar-web-36-7] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:11:31 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.26.0" 0
2023-02-07T10:11:31,509+0800 [pulsar-web-36-5] WARN org.apache.pulsar.broker.web.ResponseHandlerFilter - Http request Request[GET //172.19.201.155:8080/metrics/]@22480151 async context timeout.
2023-02-07T10:11:31,510+0800 [pulsar-web-36-8] INFO org.eclipse.jetty.server.RequestLog - - - - [07/Feb/2023:10:11:01 +0800] "GET /metrics/ HTTP/1.1" 500 0 "http://172.19.201.155:8080/metrics" "Prometheus/2.26.0" 30001
2023-02-07T10:11:31,512+0800 [prometheus-stats-37-1] WARN org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read policies data, will apply the default backlog quota: namespace=mxsk_data_tenant/cmp_ns
org.apache.pulsar.metadata.api.MetadataStoreException: Failed to get data from /admin/policies/mxsk_data_tenant/cmp_ns
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:91) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.resources.NamespaceResources.getPolicies(NamespaceResources.java:101) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:82) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.getBacklogQuota(PersistentTopic.java:2461) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.getTopicStats(NamespaceStatsAggregator.java:129) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$0(NamespaceStatsAggregator.java:76) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$1(NamespaceStatsAggregator.java:75) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$2(NamespaceStatsAggregator.java:74) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.generate(NamespaceStatsAggregator.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator.generate(PrometheusMetricsGenerator.java:110) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet.lambda$doGet$0(PrometheusMetricsServlet.java:77) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.2.jar:4.14.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:86) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
... 26 more
2023-02-07T10:11:46,508+0800 [pulsar-web-36-6] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:11:46 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.26.0" 0
2023-02-07T10:11:46,508+0800 [pulsar-web-36-5] WARN org.apache.pulsar.broker.web.ResponseHandlerFilter - Http request Request[GET //172.19.201.155:8080/metrics/]@4ffd4ef6 async context timeout.
2023-02-07T10:11:46,509+0800 [pulsar-web-36-3] INFO org.eclipse.jetty.server.RequestLog - - - - [07/Feb/2023:10:11:16 +0800] "GET /metrics/ HTTP/1.1" 500 0 "http://172.19.201.155:8080/metrics" "Prometheus/2.26.0" 30001
2023-02-07T10:11:49,828+0800 [pulsar-backlog-quota-checker-26-1] WARN org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read policies data, will apply the default backlog quota: namespace=mxsk_data_tenant/cmp_ns
org.apache.pulsar.metadata.api.MetadataStoreException: Failed to get data from /admin/policies/mxsk_data_tenant/cmp_ns
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:91) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.resources.NamespaceResources.getPolicies(NamespaceResources.java:101) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:82) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuotaLimitInTime(BacklogQuotaManager.java:105) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.isTimeBacklogExceeded(PersistentTopic.java:2517) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$72(BrokerService.java:1637) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$71(BrokerService.java:1622) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1620) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1631) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.2.jar:4.14.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:86) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
... 21 more
2023-02-07T10:12:01,207+0800 [metadata-store-6-1] WARN org.apache.pulsar.broker.authorization.AuthorizationService - Producer-client with Role - mxsk-cms failed to get permissions for topic - persistent://mxsk_data_tenant/cmp_ns/customer_info_topic-partition-2. null
2023-02-07T10:12:01,207+0800 [metadata-store-6-1] WARN org.apache.pulsar.broker.service.Producer - [mxsk-cms] Get unexpected error while autorizing [persistent://mxsk_data_tenant/cmp_ns/customer_info_topic-partition-2] null
java.util.concurrent.TimeoutException: null
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.authorization.AuthorizationService.canProduce(AuthorizationService.java:230) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.Producer.checkPermissions(Producer.java:654) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$onPoliciesUpdate$106(PersistentTopic.java:2419) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4772) ~[?:?]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.onPoliciesUpdate(PersistentTopic.java:2418) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$84(BrokerService.java:1830) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$85(BrokerService.java:1830) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753) ~[?:?]
at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731) ~[?:?]
at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$86(BrokerService.java:1825) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BrokerService.lambda$handlePoliciesUpdates$87(BrokerService.java:1821) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$get$7(ZKMetadataStore.java:139) ~[org.apache.pulsar-pulsar-metadata-2.9.1.jar:2.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-02-07T10:12:01,207+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.Producer - [mxsk-cms] is not allowed to produce on topic [persistent://mxsk_data_tenant/cmp_ns/customer_info_topic-partition-2] anymore
2023-02-07T10:12:01,207+0800 [metadata-store-6-1] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://mxsk_data_tenant/cmp_ns/customer_info_topic-partition-2}, client=/39.108.17.76:29159, producerName=pulsar-cluster-zk-32-7861416, producerId=9}
2023-02-07T10:12:01,509+0800 [pulsar-web-36-7] INFO org.eclipse.jetty.server.RequestLog - 172.19.201.137 - - [07/Feb/2023:10:12:01 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.26.0" 1
2023-02-07T10:12:01,509+0800 [pulsar-web-36-5] WARN org.apache.pulsar.broker.web.ResponseHandlerFilter - Http request Request[GET //172.19.201.155:8080/metrics/]@1faf0183 async context timeout.
2023-02-07T10:12:01,509+0800 [pulsar-web-36-5] INFO org.eclipse.jetty.server.RequestLog - - - - [07/Feb/2023:10:11:31 +0800] "GET /metrics/ HTTP/1.1" 500 0 "http://172.19.201.155:8080/metrics" "Prometheus/2.26.0" 30000
2023-02-07T10:12:01,513+0800 [prometheus-stats-37-1] WARN org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read policies data, will apply the default backlog quota: namespace=mxsk_data_tenant/cmp_ns
org.apache.pulsar.metadata.api.MetadataStoreException: Failed to get data from /admin/policies/mxsk_data_tenant/cmp_ns
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:91) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.resources.NamespaceResources.getPolicies(NamespaceResources.java:101) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:82) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.getBacklogQuota(PersistentTopic.java:2461) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.getTopicStats(NamespaceStatsAggregator.java:127) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$0(NamespaceStatsAggregator.java:76) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$1(NamespaceStatsAggregator.java:75) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$2(NamespaceStatsAggregator.java:74) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) ~[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.generate(NamespaceStatsAggregator.java:70) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator.generate(PrometheusMetricsGenerator.java:110) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet.lambda$doGet$0(PrometheusMetricsServlet.java:77) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.2.jar:4.14.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.resources.BaseResources.get(BaseResources.java:86) ~[org.apache.pulsar-pulsar-broker-common-2.9.1.ja