How to use the RDMA on Azure

Question

I purchased 1 H16r Virtual Machine on Azure, and I tried the setting method of RDMA on "About H-series and compute-intensive A-series VMs"：

azure config mode arm
azure vm extension set <resource-group> <vm-name> RDMAUpdateForLinux Microsoft.OSTCExtensions 0.1

As azure official introduced: "This interface allows RDMA-capable instances to communicate with each other over an InfiniBand network, operating at FDR rates for H16r and H16mr virtual machines, and QDR rates for A8 and A9 virtual machines."

So the H16r flavor is right using the Infiniband tech, but I still cannot find IB interface inside the H16 VM after the rdma driver is installed by azure-cli!

Answer 1

It's not much of an "RDMA Capable" device if you can't do RDMA!

On other H series (H[bc]*r*), you have an ib0 interface and Mellanox exposed in the bus through an SR-IOV interface, but with the H16r, you don't:

NR_14-22:58:03 ~ $ ifconfig ib0  
  ib0: error fetching interface
  information: Device not found  

NR_14-22:58:43 ~ $ lsmod | grep rdma 

NR_14-22:58:57 ~ $ lspci | grep Mell

Microsoft states here

On these CentOS-based HPC images for non-SR-IOV enabled VMs, kernel updates are disabled in the yum configuration file. This is because the NetworkDirect Linux RDMA drivers are distributed as an RPM package, and driver updates might not work if the kernel is updated.

... but there is no such "NetworkDirect" Linux driver.