New Fabrics

Session information has not yet been published for this event.


One Line Summary

New network technologies (fabrics, NICs and even APIs) have recently surfaced.


The RDMA stack in Linux has matured significantly over the last 2 years.
Multiple new InfiniBand based NICs have appeared and at the same time a new
non-Verbs based (PSM) hardware device has been added.

2 major device types exist in the tree today

1) Verbs based (IB, RoCE, iWarp) 2) non-Verbs based (PSM)

While the basic high level semantics between the above interfaces are similar.

1) eager send/rcv 2) direct data placement via rendezvous operations 3) support for unconnected operations

… the details between them are vastly different.

Possible questions to be discussed:

1) What is the fundamental purpose of a standardized RDMA stack?

1a) Standardized device management? Connection management? User interface management? 1b) The shortcomings of having an ABI which is directly based on the hardware implementation has been discussed many times before. What is the best way to provide generic access to RDMA functionality of different hardware devices? 1c) Could/Should parts of the stack be removed to make the interface cleaner? Is anyone ever going to implement UC? How many devices support XRC?

2) Without trying to design for hardware which does not exist how can we make
ABIs flexible enough to accommodate the unknown without being overly complex?

2a) Even if the base semantics of RDMA devices are similar it is really good to force them both through a common interface which is potentially sub-optimal for each?

3) How should the Linux kernel support devices which have RDMA hardware which
are not Queue Pair or Connection oriented?

4) What can be learned by the body of work in libfabric or the DRM stack which
supports a wide variety of hardware with a common interface?

5) How do we build in access to special features of new hardware which is
constantly on the bleeding edge of network research?

6) What is the equivalent of setsockopt in rdma?


rdma infiniband omnipath device roce iwarp nic fabrics api


  • Ira Weiny



    Ira Weiny has been working with Linux for over 16 years and has concentrated on Linux’s use in High Performance Computing for the last 10.