Dual port CMAC drops

[ample-alex]

Hello,

We are having trouble with the dual port configuration of Open Nic, as we are hoping to receive 50 Gbps per port with tx disabled on both cmacs. ONIC is configured with dual cmacs and a single QDMA instance on the U55 using the DPDK drivers, where each port is configured as their own physical function sharing a single QDMA instance. Further, we have the vfio-pci driver bound to each individual port.

Currently, we can get 80+ Gbps on either port with next to zero packet drops when we send traffic on only a single port. If we send traffic to both ports simultaneously, we drop almost 20% of packets when the speed is higher than 5 Gbps per port. Are we right to assume that we should be able to receive packets on both ports simultaneously? If so, do any modifications need to be made to the opennic dpdk driver register configuration to reduce the contention between ports?

[cneely-amd]

Hi @ample-alex,

The way you describe your test, makes me wonder what packet lengths that you are using and how bursty is the traffic when this is failing? (It makes me wonder whether somewhere in the design if FIFOs might be undersized and too shallow for coping with the amount of backpressure.) Does the drop rate increase with throughput beyond 5 Gb/s?

Also, can you give an example of how many processor cores you are assigning when you bind your dpdk driver? Are you assuming that each RX core can handle roughly ~5 Gb/s (or so)?
(Presumably though you would have enough because you were seeing 80+ Gbps being received in the single port case?)

Thanks,
--Chris

[ample-alex]

Thanks Chris -

We're sending a mix of packets, with 50% of them being 1.5 KB, and the rest being an even mix between 60 and 1500 bytes. We do have a fair suspicion that the traffic is relatively bursty from our experiments with our single port design. The drop rate does increase with throughput after 5 Gbps.

We've tested with both 4 and 8 cores splitting the queues evenly between each core. 4 is enough for 80 Gbps on the single port, but the drops occur even with 8 cores on dual port. Are there FIFOs that you recommend we look at resizing?

Thanks again for the help.

[cneely-amd]

Without knowing what's going wrong, here are a couple of places within the design that I can suggest maybe probing first with Chipscope might give some insight to monitor if the FIFOs are filling, before increasing the FIFO sizes. Hopefully that could help to understand if these are the problem areas.

1 ) Completion FIFO in QDMA C2H Subsystem: maybe use Chipscope to check if that "cpl_fifo_full" occurs
2 ) At Packet Adapter RX: maybe use Chipscope to check if the "drop" signal gets asserted

Or if that's not possible, maybe just try experimentally increasing the FIFOs in these locations to see if there's any improvement.

Thanks,
--Chris