Should be triggered by some query on information_schema.xxx tables, and that query lead to tidb coprocessor request ...
The trigger conditiion may similiar to this one #57798 (comment)

Seems we need add RefCountOfStmtCtx before use StmtCtx, ref #39368

Thank you for your comments.

In StatementContext, mu is not a pointer but an actual instance, and it is not exported either packages.
There's also no point in the code where mu is replaced with a new Mutex.
Furthermore, everywhere it's used, Lock and Unlock are both called within the same function.

Despite that, I can't figure out why the Unlock call isn't matching. What possible reasons could there be for this?

I think it same as #57799 metioned. We didn't check StatementContext is still using or not before reset it.

I think I'm starting to understand this a bit.

Are you suggesting that StatementContext is being duplicated outside the package while still containing mu?

If that's the case, would it be enough to modify the code (similarly to #57986 ) so that at the beginning of the function, we store the pointer to the mutex in a local variable and then use that?

You mean #57799? I think it's a workaround way. But we need modify a lots functions, such like SetAffectedRows, FoundRows and so on.

Sorry, It's TYPO. (similarly to #57799 ) is collect.

I will try to modify all functions in stmtctx.go .

Sorry, It's TYPO. (similarly to #57799 ) is collect.

I will try to modify all functions in stmtctx.go .

/cc @tiancaiamao

I’ve committed my changes and opened PR #58629.
Does this match what you had in mind?

This is my first contribution, so please let me know if there are any issues or anything I need to fix.
I’ve already taken care of tasks like signing the CLA.

I’d like to gain a deeper understanding of this issue.

As a workaround, I’ve store a pointer of the Mutex to ensure that the target mutex cannot be replaced.

However, I believe the real cause is that StatementContext itself is being overwritten by some operation, like duplication—am I correct in thinking this?
If so, it might be better to fix the usage of StatementContext that occurs outside of the stmtctx package.

That said, I don’t currently have enough knowledge to make that judgment on my own 😢 .

I’d like to gain a deeper understanding of this issue.

As a workaround, I’ve store a pointer of the Mutex to ensure that the target mutex cannot be replaced.

However, I believe the real cause is that StatementContext itself is being overwritten by some operation, like duplication—am I correct in thinking this? If so, it might be better to fix the usage of StatementContext that occurs outside of the stmtctx package.

That said, I don’t currently have enough knowledge to make that judgment on my own 😢 .

I think you understand it correctly. So I think the better way to fix this issue is to add RefCountOfStmtCtx before use StmtCtx. And the ref count will block reset.

I see.
Thank you for your explanation and responding to the pull request.

I see where mu is being replaced.
In StatementContext.Reset() (lines 469–479), a new Mutex object is assigned.

What if we lock the old Mutex before it gets replaced?

        mu := &sc.mu
	mu.Lock()
	cacheMu := &sc.stmtCache.mu
	cacheMu.Lock()
	*sc = StatementContext{
		ctxID:               contextutil.GenContextID(),
		CTEStorageMap:       sc.CTEStorageMap,
		LockTableIDs:        sc.LockTableIDs,
		TableStats:          sc.TableStats,
		MDLRelatedTableIDs:  sc.MDLRelatedTableIDs,
		TblInfo2UnionScan:   sc.TblInfo2UnionScan,
		WarnHandler:         sc.WarnHandler,
		ExtraWarnHandler:    sc.ExtraWarnHandler,
		IndexUsageCollector: sc.IndexUsageCollector,
	}
        mu.Unlock()
        cacheMu.Unlock()

I see where mu is being replaced. In StatementContext.Reset() (lines 469–479), a new Mutex object is assigned.

What if we lock the old Mutex before it gets replaced?

        mu := &sc.mu
	mu.Lock()
	cacheMu := &sc.stmtCache.mu
	cacheMu.Lock()
	*sc = StatementContext{
		ctxID:               contextutil.GenContextID(),
		CTEStorageMap:       sc.CTEStorageMap,
		LockTableIDs:        sc.LockTableIDs,
		TableStats:          sc.TableStats,
		MDLRelatedTableIDs:  sc.MDLRelatedTableIDs,
		TblInfo2UnionScan:   sc.TblInfo2UnionScan,
		WarnHandler:         sc.WarnHandler,
		ExtraWarnHandler:    sc.ExtraWarnHandler,
		IndexUsageCollector: sc.IndexUsageCollector,
	}
        mu.Unlock()
        cacheMu.Unlock()

I think it will block the groutine for a while and influence the SQL execution.

I’m also concerned about the performance impact. Should we go ahead with the workaround in the commit, or is there a more appropriate method? What would be the best approach here?

I have a patch to fix this issue, but I haven't submitted a PR because I haven't found a way to reproduce the panic and can't 100% guarantee the effectiveness of the patch.

diff --git a/pkg/executor/infoschema_reader.go b/pkg/executor/infoschema_reader.go
index 206ebb52fb..e203886e58 100644
--- a/pkg/executor/infoschema_reader.go
+++ b/pkg/executor/infoschema_reader.go
@@ -1839,7 +1839,11 @@ func (e *memtableRetriever) setDataForProcessList(ctx sessionctx.Context) {
                        continue
                }

+               if pi.StmtCtx != nil && pi.RefCountOfStmtCtx != nil && !pi.RefCountOfStmtCtx.TryIncrease() {
+                       continue
+               }
                rows := pi.ToRow(ctx.GetSessionVars().StmtCtx.TimeZone())
+               pi.RefCountOfStmtCtx.Decrease()
                record := types.MakeDatums(rows...)
                records = append(records, record)
        }

I see! So the StatementContext is only reset in InitStatementContext(), and that is also checked by RefCountOfStmtCtx.

In other words, rather than acquiring the mutex lock within Reset(), we can properly handle this by appropriately managing RefCountOfStmtCtx in the code that calls functions like AffectedRows(). Is that correct?

I think so. But we couldn't get RefCountOfStmtCtx in AffectedRows()

Based on what I’ve seen, in my environment, every occurrence of the panic happened during calls originating from setDataForProcessList (infoschema_reader.go:1843).

I’ll try running the patch you provided on my local environment.

If it works, should we skip the previously committed workaround and only introduce the handling of RefCountOfStmtCtx?
Or is it better to keep both changes?

I think it is enough to keep RefCountOfStmtCtx. But I will let other colleagues review this PR and ask for their opinions

I wish you a happy New Year.
Thank you for providing guidance despite it being year-end.

Since last night, I’ve deployed a binary that includes the patch referencing RefCountOfStmtCtx and have started monitoring it.

I’ll keep observing it for a while.

I wish you a happy New Year. Thank you for providing guidance despite it being year-end.

Since last night, I’ve deployed a binary that includes the patch referencing RefCountOfStmtCtx and have started monitoring it.

I’ll keep observing it for a while.

Does panic still exist? If not, you could update your PR and I will ask my colleagues to review it.