Upsert Deadlock Condition Present in Pinot 1.1

[ankitsultana]

We had been seeing frequent deadlocked threads on our Upsert enabled servers, and we were able to trace the root-cause to this commit: #12241.

At a high-level, the issue is that:

1. The commit introduced a new type of lock called "Upsert Lock".
2. There's no precedence defined between the "Segment Locks" and the "Upsert Locks". In some state transitions we may acquire Segment Lock first and then block on the Upsert Lock, and in others the order may be the opposite.
3. This can combine with the Lock Collision issue (see: Avoid collision in SegmentLocks #13632) that was fixed in July 2024, and it could be that the deadlock happens for two segments across two entirely different servers.

This impacts our internal Pinot version and also impacts Pinot 1.1 as far as I know. The Upsert Locks are now removed so I assume this would be fixed now. @Jackie-Jiang : would you be able to confirm this?

[Jackie-Jiang]

#12886 Refactored the message handling logic, and consolidated this lock into the regular segment lock. With #12886 and #13632 this problem should be solved.