Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

owner manager hang in BreakCampaignLoop #58620

Open
tiancaiamao opened this issue Dec 30, 2024 · 1 comment · May be fixed by #58622
Open

owner manager hang in BreakCampaignLoop #58620

tiancaiamao opened this issue Dec 30, 2024 · 1 comment · May be fixed by #58622
Assignees
@tiancaiamao
Labels
affects-8.5 This bug affects the 8.5.x(LTS) versions. component/ddl This issue is related to DDL of TiDB. severity/minor type/bug The issue is confirmed as a bug.

Comments

@tiancaiamao
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tikv exit first, and then tidb fail to exit when I press Ctrl+C

2. What did you expect to see? (Required)

tidb should not hang when exit.

3. What did you see instead (Required)

It hang here:

 0  0x00000000020b972e in runtime.gopark
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:425
 1  0x0000000002094ac5 in runtime.goparkunlock
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:430
 2  0x0000000002094ac5 in runtime.semacquire1
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/runtime/sema.go:178
 3  0x00000000020bb165 in sync.runtime_Semacquire
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/runtime/sema.go:71
 4  0x00000000020d5c68 in sync.(*WaitGroup).Wait
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/sync/waitgroup.go:118
 5  0x0000000003c82ab0 in github.com/pingcap/tidb/pkg/owner.(*ownerManager).BreakCampaignLoop
    at ./pkg/owner/manager.go:349
 6  0x0000000004a6dcc4 in github.com/pingcap/tidb/pkg/ddl.(*ddl).close
    at ./pkg/ddl/ddl.go:1032
 7  0x0000000004a6b6d7 in github.com/pingcap/tidb/pkg/ddl.(*ddl).Stop
    at ./pkg/ddl/ddl.go:765
 8  0x0000000004d0df03 in github.com/pingcap/tidb/pkg/domain.(*Domain).Close
    at ./pkg/domain/domain.go:1248
 9  0x0000000005a461d0 in main.closeDDLOwnerMgrDomainAndStorage
    at ./cmd/tidb-server/main.go:904
10  0x0000000005a462de in main.cleanup
    at ./cmd/tidb-server/main.go:929
11  0x0000000005a421aa in main.main.func1
    at ./cmd/tidb-server/main.go:330
12  0x0000000005a2f0a7 in github.com/pingcap/tidb/pkg/util/signal.SetupSignalHandler.func2
    at ./pkg/util/signal/signal_posix.go:55
13  0x00000000020c2721 in runtime.goexit
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700

ddl close call d.ownerManager.BreakCampaignLoop() and the owner manager hang.
It's waiting for the campaign loop to exit, but the campaign loop never break:

 0  0x00000000020b972e in runtime.gopark
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:425
 1  0x00000000020938a5 in runtime.selectgo
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/runtime/select.go:335
 2  0x0000000002743cd8 in google.golang.org/grpc.(*pickerWrapper).pick
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/picker_wrapper.go:114
 3  0x000000000273e64f in google.golang.org/grpc.(*ClientConn).getTransport
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/clientconn.go:1047
 4  0x00000000027596c5 in google.golang.org/grpc.(*csAttempt).getTransport
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/stream.go:467
 5  0x0000000002758ca5 in google.golang.org/grpc.newClientStreamWithParams.func2
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/stream.go:350
 6  0x000000000275a97a in google.golang.org/grpc.(*clientStream).withRetry
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/stream.go:777
 7  0x0000000002758768 in google.golang.org/grpc.newClientStreamWithParams
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/stream.go:362
 8  0x0000000002757b27 in google.golang.org/grpc.newClientStream.func3
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/stream.go:219
 9  0x00000000027574f9 in google.golang.org/grpc.newClientStream
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/stream.go:254
10  0x0000000002738d77 in google.golang.org/grpc.invoke
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/call.go:66
11  0x0000000002da2a98 in go.etcd.io/etcd/client/v3.(*Client).unaryClientInterceptor.func1
    at /home/genius/go/pkg/mod/go.etcd.io/etcd/client/[email protected]/retry_interceptor.go:58
12  0x0000000002738c45 in google.golang.org/grpc.(*ClientConn).Invoke
    at /home/genius/go/pkg/mod/google.golang.org/[email protected]/call.go:35
13  0x0000000002d2ef65 in go.etcd.io/etcd/api/v3/etcdserverpb.(*leaseClient).LeaseGrant
    at /home/genius/go/pkg/mod/go.etcd.io/etcd/api/[email protected]/etcdserverpb/rpc.pb.go:6812
14  0x0000000002da03d5 in go.etcd.io/etcd/client/v3.(*retryLeaseClient).LeaseGrant
    at /home/genius/go/pkg/mod/go.etcd.io/etcd/client/[email protected]/retry.go:144
15  0x0000000002d99452 in go.etcd.io/etcd/client/v3.(*lessor).Grant
    at /home/genius/go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:216
16  0x0000000003273c73 in go.etcd.io/etcd/client/v3/concurrency.NewSession
    at /home/genius/go/pkg/mod/go.etcd.io/etcd/client/[email protected]/concurrency/session.go:46
17  0x000000000327e565 in github.com/pingcap/tidb/pkg/util.NewSession
    at ./pkg/util/etcd.go:72
18  0x0000000003c83c65 in github.com/pingcap/tidb/pkg/owner.(*ownerManager).refreshSession
    at ./pkg/owner/manager.go:446
19  0x0000000003c82cbb in github.com/pingcap/tidb/pkg/owner.(*ownerManager).campaignLoop
    at ./pkg/owner/manager.go:367
20  0x0000000003c827c8 in github.com/pingcap/tidb/pkg/owner.(*ownerManager).CampaignOwner.gowrap1
    at ./pkg/owner/manager.go:300
21  0x00000000020c2721 in runtime.goexit
    at /home/genius/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700

Similiar to #58418

4. What is your TiDB version? (Required)

master
70393b5
@tiancaiamao tiancaiamao added the type/bug The issue is confirmed as a bug. label Dec 30, 2024
@tiancaiamao tiancaiamao self-assigned this Dec 30, 2024
@tiancaiamao tiancaiamao added affects-8.5 This bug affects the 8.5.x(LTS) versions. severity/minor labels Dec 30, 2024
@tiancaiamao tiancaiamao linked a pull request Dec 30, 2024 that will close this issue
Open
13 tasks
@Defined2014 Defined2014 mentioned this issue Dec 30, 2024
Closed
@jebter jebter added the component/ddl This issue is related to DDL of TiDB. label Dec 31, 2024
@D3Hunter
Copy link
Contributor

it's not related to owner manager itself, Domain should cancel the context before wait other components to exit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tiancaiamao tiancaiamao

Labels
affects-8.5 This bug affects the 8.5.x(LTS) versions. component/ddl This issue is related to DDL of TiDB. severity/minor type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

owner: fix bug of owner manager hang tiancaiamao/tidb
3 participants
@tiancaiamao @D3Hunter @jebter