libvma doesn't support the cascaded epoll usage model

[jiangdong-sun]

Tested on
VMA_VERSION: 8.0.4-0 Release built on May 31 2016 15:34:50
OFED Version: MLNX_OFED_LINUX-3.3-1.0.0.0

I guess all versions of libvma don't support cascaded epoll usage model.

BTW, libkqueue use epoll as cascade model for monitoring EPOLLOUT event, so libkqueue cannot work well with libvma.

It's easy to show the issue. But if you want the test code, I can provide then.

[OphirMunk]

@jiangdong-sun any test code that can demo your issue is appreciated. Please attach

[jiangdong-sun]

The test code is attached here.
t_cascaded_epoll.c.gz

How to verify

• compile the code

• run it to connect any tcp server with/without libvma.so preloaded.

  The test process for your reference as below.
  without libvma.

$ unset LD_PRELOAD
$ ./t_cascaded_epoll host_ip host_port

The EPOLLIN event will be received after the connection succeeded.

But, with libvma.

$ export LD_PRELOAD=/usr/lib64/libvma.so
$ ./t_cascaded_epoll host_ip host_port

The EPOLLIN event will never be triggerred even the connection succeeded

[jiangdong-sun]

@OphirMunk any feedback? Thanks!

[OphirMunk]

@jiangdong-sun Thanks for sharing your code which enables us to recreate your case.
VMA is focused on low latency for RX/TX fast path. Is your cascaded epoll needed for low latency or not? Can the cascaded epoll be used on the "slow path" (for a specific TCP port).
If yes - I would try using libvma.conf file to indicate the specific TCP port that should be handled by the Operating System (OS) and not by VMA.
I would definitely would like to hear if you gave it a try and if it worked for you. You can find details on libvma.conf file here: http://www.mellanox.com/related-docs/prod_acceleration_software/VMA_8_0_4_User_Manual_DOC-00393.pdf (section 4.1)
In addition - I am forwarding your issue to a Mellanox field application engineer that may ask you more about the demand/justification for using cascaded epoll.

[jiangdong-sun]

@OphirMunk Thanks for your reply.
Yes, cascaded epoll is needed for low latency in our environment. Actually, it's exist in libkqueue used by our application. So the slow path doesn't meet our requirement.
Hope you team can provide this feature, and I'd like to answer the questions from your field application engineer. Thanks.