Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deployment部署的knowstreaming，接入statefulset部署的kafka集群提示超时 #1227

Open
1 task done
Leif160519 opened this issue May 25, 2024 · 3 comments
Open
1 task done

deployment部署的knowstreaming，接入statefulset部署的kafka集群提示超时 #1227

Leif160519 opened this issue May 25, 2024 · 3 comments

Comments

@Leif160519
Copy link

Leif160519 commented May 25, 2024 •
edited
Loading

  • 我已经在 issues 搜索过相关问题了,并没有重复的。

deployment部署的knowstreaming，接入statefulset部署的kafka集群提示超时

[root@k8s-master-01:~] # kubectl get pod -n knowstreaming 
NAME                                     READY   STATUS    RESTARTS   AGE
elasticsearch-957f899bd-p9fcc            1/1     Running   0          24h
knowstreaming-manager-6747759bc8-6rfg2   1/1     Running   0          23h
knowstreaming-ui-7b8fd498-hpdrt          1/1     Running   0          23h
mysql-7d55ff77f5-45fdb                   1/1     Running   0          24h
[root@k8s-master-01:~7] # kubectl get pod -n midplatform | egrep "kafka|zookeeper"
kafka-0                         1/1     Running   0              22h
kafka-1                         1/1     Running   0              22h
kafka-2                         1/1     Running   0              22h
zookeeper-0                     1/1     Running   0              3d1h
zookeeper-1                     1/1     Running   0              3d1h
zookeeper-2                     1/1     Running   0              3d1h

image

knowstream-manager日志如下：

2024-05-25 15:26:37.995  WARN 12 --- [ adminclient-16] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-16] Error connecting to node kafka-2.kafka-hs:9092 (id: 1010 rack: null)

java.net.UnknownHostException: kafka-2.kafka-hs
	at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:111)
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:513)
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:467)
	at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:172)
	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:985)
	at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:311)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1080)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1321)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264)
	at java.lang.Thread.run(Thread.java:745)

2024-05-25 15:26:38.092  WARN 12 --- [ adminclient-16] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-16] Error connecting to node kafka-0.kafka-hs:9092 (id: 1008 rack: null)

java.net.UnknownHostException: kafka-0.kafka-hs
	at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:111)
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:513)
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:467)
	at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:172)
	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:985)
	at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:311)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1080)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1321)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264)
	at java.lang.Thread.run(Thread.java:745)

2024-05-25 15:26:38.092  WARN 12 --- [ adminclient-16] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-16] Error connecting to node kafka-1.kafka-hs:9092 (id: 1009 rack: null)

java.net.UnknownHostException: kafka-1.kafka-hs
	at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:111)
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:513)
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:467)
	at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:172)
	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:985)
	at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:311)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1080)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1321)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264)
	at java.lang.Thread.run(Thread.java:745)

2024-05-25 15:26:38.093 ERROR 12 --- [p-nio-80-exec-1] c.x.k.s.k.p.k.z.s.impl.KafkaZKDAOImpl    : method=checkBootstrapServersLegal||bootstrapServers=kafka-0.kafka-hs.midpaltform.svc:9092,kafka-1.kafka-hs.midplatform.svc:9092,kafka-2.kafka-hs.midplatform.svc:9092||clientProps={key.deserializer=org.apache.kafka.common.serialization.StringDeserializer, bootstrap.servers=kafka-0.kafka-hs.midpaltform.svc:9092,kafka-1.kafka-hs.midplatform.svc:9092,kafka-2.kafka-hs.midplatform.svc:9092, value.deserializer=org.apache.kafka.common.serialization.StringDeserializer}||msg=get cluster-info failed||errMsg=exception

java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1716621998092, tries=1, nextAllowedTryMs=1716621998193) timed out at 1716621998093 after 1 attempt(s)
	at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
	at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
	at com.xiaojukeji.know.streaming.km.core.service.cluster.impl.ClusterValidateServiceImpl.getDataAndIgnoreCheckBSLegal(ClusterValidateServiceImpl.java:116)
	at com.xiaojukeji.know.streaming.km.core.service.cluster.impl.ClusterValidateServiceImpl.checkKafkaLegal(ClusterValidateServiceImpl.java:72)
	at com.xiaojukeji.know.streaming.km.rest.api.v3.util.UtilsController.validateKafka(UtilsController.java:43)
	at sun.reflect.GeneratedMethodAccessor985.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:764)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:888)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1597)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1716621998092, tries=1, nextAllowedTryMs=1716621998193) timed out at 1716621998093 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes

2024-05-25 15:26:38.094  INFO 12 --- [ adminclient-16] o.a.kafka.common.utils.AppInfoParser     : App info kafka.admin.client for adminclient-16 unregistered
2024-05-25 15:26:38.095  INFO 12 --- [ adminclient-16] o.a.k.c.a.i.AdminMetadataManager         : [AdminClient clientId=adminclient-16] Metadata update failed

org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1716622018202, tries=1, nextAllowedTryMs=-9223372036854775709) timed out at 9223372036854775807 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata

2024-05-25 15:26:38.097  INFO 12 --- [ adminclient-16] org.apache.kafka.common.metrics.Metrics  : Metrics scheduler closed
2024-05-25 15:26:38.097  INFO 12 --- [ adminclient-16] org.apache.kafka.common.metrics.Metrics  : Closing reporter org.apache.kafka.common.metrics.JmxReporter
2024-05-25 15:26:38.097  INFO 12 --- [ adminclient-16] org.apache.kafka.common.metrics.Metrics  : Metrics reporters closed
@Leif160519
Copy link
Author

image
image
image
image
image
image

@Leif160519
Copy link
Author

难道knowstreaming必须要跟kafak部署在同一个命名空间里吗

@ZQKC
Copy link
Collaborator

ZQKC commented Jun 30, 2024

难道knowstreaming必须要跟kafak部署在同一个命名空间里吗

ks所在的机器不识别kafka主机名，所以看看是不是要配置k8s 的kafka集群，网络这块是不是有问题

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ZQKC @Leif160519