Several refactors regarding WR indices

[NitzanLavy]

Use CQE QP number

Perftest assumes that all WQEs are created with the QP number as their
wr_id. This creates a situation where WQEs do not have unique IDs. As a
result, in various areas of the code, wr_id field from wc is used as the
index to the relevant QP. The usage of wr_id is replaced with the
actual QP index that also exists in the wc struct.

Use unique WR IDs

While setting send and receive WQEs, assiging to each WQE a unique wr_id
instead of qp index. This will allow the user to detect which work
request got completed.

Perftest: Fix address cycling bug in local address increment logic

The increase_loc_addr() function manages circular buffer addressing.
One of the input argument is rcnt, which represents the number of
post_send/post_receive operations that were called per QP. Based on
rcnt, it internally calculates the local address for the next operation.
When setting up receive WQEs, this function is being called with the
index of the current operation instead of the amount that was already
called. This incorrect indexing caused the last WQE in the buffer to
incorrectly point to the first WQE's address in the QP.
This bug affects the cyclic address calculation used for cache-efficient
data placement. The fix ensures proper address cycling by passing the
correct index to the function.

[sshaulnv]

Hi @NitzanLavy, thanks for the contribution.

did you tested the changes? im getting 0 BW:

./ib_write_bw -q 1 -D 5 localhost
 WARNING: BW peak won't be measured in this run.                                                                                                                                                                                             
---------------------------------------------------------------------------------------                                                                                                                                                      
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON           Lock-free      : OFF
 ibv_wr* API     : ON           Using DDP      : OFF
 TX depth        : 128
 CQ Moderation   : 1
 CQE Poll Batch  : 16
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 1
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x00ac PSN 0x9043ac RKey 0x201800 VAddr 0x007f422106b000
 GID: 254:128:00:00:00:00:00:00:18:112:253:255:254:67:113:64
 remote address: LID 0000 QPN 0x00ab PSN 0x73a028 RKey 0x201500 VAddr 0x007f9738c44000
 GID: 254:128:00:00:00:00:00:00:18:112:253:255:254:67:113:64
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MiB/sec]    BW average[MiB/sec]   MsgRate[Mpps]
 65536      0                0.00               0.00                 0.000000
---------------------------------------------------------------------------------------

[NitzanLavy]

Hi, yes I did test the patches. This is strange. I will take a look.

[NitzanLavy]

Hi @sshaulnv, i fixed some bugs and tested thoroughly the patches.
In my tests I had to add to the command you used flags -x and -c as I am running on EFA. Could you please review the PRs and try running it again?
I divided the content into separate PRs, please see #339 and #338. Closing this one.