rdma

Enumerate and inspect Linux RDMA devices over NETLINK_RDMA.

The shape mirrors rtnetlink: build a connection, spawn it, and drive requests through a cheap cloneable Handle.

use futures_util::stream::TryStreamExt;

let (connection, handle, _) = rdma::new_connection()?;
tokio::spawn(connection);

let mut devices = handle.device().get().execute();
while let Some(device) = devices.try_next().await? {
    println!("{:?} has {} port(s)", device.name, device.port_count);
}

Scope

Read-only, built on netlink-packet-rdma.

This crate	rdma(8) equivalent
handle.device().get()	rdma dev show
handle.port().get()	rdma link show
handle.system().get()	rdma system show
handle.resource().summary()	rdma resource show
handle.resource().queue_pairs(dev)	rdma resource show qp
handle.resource().completion_queues(dev)	rdma resource show cq
handle.resource().memory_regions(dev)	rdma resource show mr
handle.resource().protection_domains(dev)	rdma resource show pd
handle.resource().contexts(dev)	rdma resource show ctx
handle.resource().shared_receive_queues(dev)	rdma resource show srq
handle.resource().cm_ids(dev)	rdma resource show cm_id
handle.stats().get(dev, port)	rdma statistic show link
handle.link().add(name, type, netdev)	rdma link add
handle.link().del(dev)	rdma link delete
handle.device().rename(dev, name)	rdma dev set name
handle.device().set_netns(dev, fd)	rdma dev set netns
handle.device().set_dim(dev, on)	rdma dev set dim
rdma::gids(dev, port)	(no equivalent — sysfs)

Creating software devices

use rdma::LinkType;

handle.link().add("rxe0", LinkType::Rxe, "eth0").execute().await?;   // soft RoCE
handle.link().add("siw0", LinkType::Siw, "eth0").execute().await?;   // soft iWARP

Both need CAP_NET_ADMIN. The kernel loads the backing module on demand via request_module("rdma-link-%s", type).

link().del() only works on drivers that set IBK_ALLOW_USER_UNREG — the software ones. Pointing it at an mlx5 device returns EINVAL rather than unbinding a NIC, which the create_soft_devices example asserts.

device().set_netns() is the one write op that generally will not work. It needs exclusive namespace mode and a driver implementing disassociate_ucontext; see below.

Two kernel-imposed shapes are reflected in the signatures rather than left to fail at runtime:

• Resource dumps take the device index as an argument, not a filter. res_get_common_dumpit returns EINVAL without RDMA_NLDEV_ATTR_DEV_INDEX. The summary is the exception — it covers every device at once.
• stats().get() takes a port, and is not a dump. Port counters come from the doit path, which requires both device and port. The dump path is a different query: it needs RDMA_NLDEV_ATTR_STAT_RES and returns per-QP or per-MR counters. Those two modes are not modelled yet.

Owner PID and kernel module name are only reported to callers with CAP_NET_ADMIN.

Failed dumps are not silent

netlink reports a dump that fails before emitting anything in the NLMSG_DONE payload, not as an NLMSG_ERROR. netlink-proto drops Done by default, so asking a bogus device index for its queue pairs yields Ok([]) — an empty success — rather than an error.

The connection constructors here enable set_forward_done(true) and turn a non-zero code into Error::DumpFailed:

handle.resource().queue_pairs(9999)  ->  Err(DumpFailed { code: -22 })

Device, Port and SystemInfo are typed views over the raw attributes. They are lossless: every attribute the kernel sent is still in .attributes, in order, including ones with no named field. A property test enforces that.

new_monitor_connection() subscribes to RDMA_NL_GROUP_NOTIFY for register/unregister and netdev attach/detach events, the equivalent of rdma monitor.

GIDs come from sysfs, not netlink

RDMA netlink does not expose GID tables. The uapi header says the table is "supposed to be exported ... once it will be exposed through the netlink", but as of Linux 6.18 no RDMA_NLDEV_ATTR_*GID* attribute exists. libibverbs reads sysfs for the same reason, so rdma::gids() is not working around a missing binding — sysfs is the interface.

for gid in rdma::gids("mlx5_0", 1)? {
    println!("{:3} {} {:?} {:?}", gid.index, gid.addr, gid.gid_type, gid.netdev);
}

Unpopulated slots are skipped, so the returned indices are not contiguous: a port advertises 255 entries and typically populates two. Note these are blocking reads — tiny, but blocking.

Testing

cargo test                       # views, no hardware needed
cargo run --example show_devices # against the running kernel

The view tests include property tests (via bolero) over arbitrary attribute lists, checking that no attribute is dropped and that the named fields agree with a manual scan of the list.

show_devices on a host with two ConnectX NICs:

system: netns_mode=Some(1) copy_on_fork=Some(1)

mlx5_0 (index 0, 1 port(s))
    fw          32.48.1000
    node type   IbCa
    node guid   0x58a2e103000431a8
    port 1     state Some(Down), netdev enp2s0f0np0
        gid 0   fe80::5aa2:e1ff:fe04:31a8   RoceV1 via enp2s0f0np0
        gid 1   fe80::5aa2:e1ff:fe04:31a8   RoceV2 via enp2s0f0np0

Verified against rdma link show and sysfs.

For hardware-free coverage, create_soft_devices builds one of each kind, inspects it, and tears it down. It needs CAP_NET_ADMIN:

cargo run --example create_soft_devices -- eth0

== rxe_test (rxe) over eth0
   index 4 protocol Some("roce") node_type Some(IbCa)
   port 1 state Some(Active) netdev eth0
   resources: pd 1  cq 1  qp 1  cm_id 0  mr 0  ctx 0  srq 0

== siw_test (siw) over eth0
   index 5 protocol Some("iw") node_type Some(Rnic)
   port 1 state Some(Active) netdev eth0

refused to delete mlx5_0, as expected: ... Invalid argument (os error 22)

RDMA devices cannot be namespace-isolated in practice

Worth knowing before designing a test harness around this crate.

rdma system set netns exclusive fails with EBUSY on any real machine: rdma_dev_init_net registers every non-init netns in rdma_nets, whether or not it uses RDMA, and rdma_compatdev_set refuses while that set is non-empty. The supported route is the boot-time parameter ib_core.netns_mode=0 (perms 0444, so it cannot be set at runtime).

Even then, rxe and siw can never be isolated, for two independent reasons:

1. ib_alloc_device() hardcodes &init_net; only ib_alloc_device_with_net() honours the caller. rxe and siw use the former — mlx5 is the only driver in the tree using the latter.
2. ib_device_set_netns_put returns EOPNOTSUPP unless the driver implements disassociate_ucontext. rxe and siw do not; mlx5 does.

So a soft device created inside a container is visible from the host, and vice versa. This is fine for testing this crate — you get a real device to enumerate — but do not expect isolation from it.