Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZRpc + Etcd + Docker Env, After the ZRpc client container slept for 1 hour, An ZRpc client request was initiated again, this request will exception and could not be recovered. The error "context deadline exceeded" appeared every 4 seconds in the log #4420

Open
ysw-jingzhe opened this issue Oct 13, 2024 · 3 comments
Assignees

Comments

@ysw-jingzhe
Copy link

Describe the bug
Zrpc + Etcd + Docker 容器微服务架构, Zrpc Client 端容器在刚启动时 rpc 之间请求正常,可以正常的通过ETCD发现服务端并请求到服务端数据。 但是在该Client 端容器睡眠1小时(1个小时内无任何外部请求) 后,再次触发Rpc客户端请求, 此时Rpc请求出现超时异常,且日志中每隔4秒出现一次 “error context deadline exceeded, key is xxxxxxxx caller=internal/registry.go:240”。 此时检查 Docker 容器正常,容器中网络也正常。

备注:在不使用容器直接在服务器上运行该Client端的Go程序,就不会存在该问题; 在相同容器中使用 kratos 配套google.golang.org/grpc v1.46.2 不会存在该问题;

Zrpc+Etcd+Docker microservice Env, the Zrpc client container requests normally between RPC at the beginning of startup, and can discover the server and request server data through ETCD normally. However, after the client-side container had slept for 1 hour (with no external requests within the hour), the Rpc client request was triggered again. At this time, the Rpc request encountered a timeout exception and appeared every 4 seconds in the log "error context deadline exceeded, key is xxxxxxxx caller=internal/registry.go:240".

Note: If the Go program on the client side is run directly on the server without using a container, this issue will not exist; Using Kratos + “google.golang.org/grpc v1.46.2” in the same container will not result in this issue;

Docker Image: alpine:latest or ubuntu 20.4
Etcd: etcd:v3.5.12
Zrpc: github.com/zeromicro/go-zero v1.7.2
Grpc: google.golang.org/grpc v1.67.0

To Reproduce
Steps to reproduce the behavior, if applicable:

  1. The code is

    // client init
    conn = zrpc.MustNewClient(zrpc.RpcClientConf{
      Etcd: discov.EtcdConf{
         Hosts: ["127.0.0.1:2379"],
         Key: "examination.rpc"
      },
    })
    
    // client every request
    client := greet.NewGreetClient(conn.Conn())
    resp,err := client.Debug(ctx, in, opts...)
    
    
  2. The error is

    2024-10-11T09:33:40.108Z     error  context deadline exceeded, key is examination.rpc   caller=internal/registry.go:240
    2024-10-11T09:33:44.109Z     error  context deadline exceeded, key is examination.rpc   caller=internal/registry.go:240
    2024-10-11T09:33:48.111Z     error  context deadline exceeded, key is examination.rpc   caller=internal/registry.go:240
    2024-10-11T09:33:52.113Z     error  context deadline exceeded, key is examination.rpc   caller=internal/registry.go:240
    2024-10-11T09:33:56.115Z     error  context deadline exceeded, key is examination.rpc   caller=internal/registry.go:240
    
    

Expected behavior
RPC requests are normal and stable

Screenshots

Environments (please complete the following information):

  • OS: Ubuntu
  • go-zero version v1.7.2
  • goctl version v.1.22.6
  • Docker Image: alpine:latest or ubuntu 20.4
  • Etcd: etcd:v3.5.12
  • Grpc: google.golang.org/grpc v1.67.0

More description

@zhoushuguang
Copy link
Collaborator

I'll try to reproduce it.

@zhoushuguang zhoushuguang self-assigned this Oct 14, 2024
@ysw-jingzhe
Copy link
Author

thanks

@ysw-jingzhe
Copy link
Author

Is there any progress? How to be stable? rpc for docker containers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants