TL/UCP: Split single and multithreaded send/receive

[ikryukov]

What

This change introduces two separate code paths for single-threaded and multi-threaded scenarios. During context creation, a specific set of functions and callbacks is selected based on the threading mode, avoiding branching in performance-critical (hot) paths. The single-threaded implementation avoids the use of atomics, which are unnecessary in that context and have been shown to impact performance on ARM systems (3–7% regression on small messages).

Why ?

Recent performance analysis has revealed regressions in TL/UCP, particularly noticeable on ARM platforms. The atomics introduced in #932 addressed correctness in multi-threaded environments but introduced overhead even in single-threaded use cases. This PR mitigates that overhead for single-threaded configurations. A future update will address multi-threaded performance by revising atomic operations to use appropriate memory models for ARM.

OSU Allgather benchmark on AMD x86, 100k iterations for better stability in results on small messages, Optimized - code from this PR.

Size(B)    Master(us)   1.3.2(us)    Optimized(us)  M vs 1.3.2(%)  O vs 1.3.2(%)  O vs M(%)   
------------------------------------------------------------------------------------------
1          94.34        98.67        75.71                 +4.58        +23.27        +19.75
2          83.13        80.98        76.41                 -2.59         +5.65         +8.09
4          84.19        83.67        75.83                 -0.61         +9.37         +9.92
8          645.81       663.58       634.07                +2.75         +4.45         +1.82
16         642.60       667.64       640.89                +3.90         +4.01         +0.27
32         664.89       745.26       667.67               +12.09        +10.41         -0.42

Grace (arm) 8 nodes 1ppn OSU alltoall cuda memory 100k iterations (before diff was 3-7%):

Size      1.3.2(us)    Current(us)    % Difference
----------------------------------------------------
1         9.79         9.83         +0.41%
2         9.78         9.82         +0.41%
4         9.76         9.81         +0.51%
8         9.77         9.82         +0.51%
16        9.76         9.80         +0.41%
32        9.89         9.92         +0.30%

[janjust]

Thanks!

[samnordmann]

LGTM

[janjust]

bot:retest