Connect QEMU-KVM VMs using vhost-user-client and ovs-dpdk

Question

My goal is to connect two QEMU-KVM VMs on an overlay network. Each VM is running on a separate physical host and must have a static IP on the network 10.0.0.0/24. To achieve this goal, I want to use an OVS bridge with DPDK. I want to use the vhost-user-client protocol to connect the OVS bridge with the VMs.

My physical setup is the following: two physical machines equipped with a Mellanox ConnectX6-DX, and connected back-to-back (no physical switch). What I want to achieve is this:

+------------------+            +------------------+
| HOST_1           |            | HOST_2           |
|                  |            |                  |
|  +------------+  |            |  +------------+  |
|  | VM_1       |  |            |  | VM_2       |  |
|  |            |  |            |  |            |  |
|  | +--------+ |  |            |  | +--------+ |  |
|  | | ens_2  | |  |            |  | | ens_2  | |  |
|  | |10.0.0.1| |  |            |  | |10.0.0.2| |  |
|  +-+---+----+-+  |            |  +-+---+----+-+  |
|        |         |            |        |         | 
|  vhost-client-1  |            |  vhost-client-1  |
|        |         |            |        |         |
|  +-----+------+  |            |  +-----+------+  |
|  |  bridge    |  |            |  |  bridge    |  |
|  |    br0     |  |            |  |    br0     |  |
|  |192.168.57.1|  |            |  |192.168.57.2|  |
|  +-----+------+  |            |  +-----+------+  |
|        |         |            |        |         |
|    +---+---      |            |    +---+---+     |
|    | dpdk0 |     |            |    | dpdk0 |     |
+----+---+--+------+            +----+---+---+-----+
         |                               |
         +-------------------------------+

I successfully created the OVS bridge (here, br0) and the DPDK port (here, dpdk0). On each physical machine, I am able to ping the bridge on the other machine. Then, I created a vhost-user-client port and attached it to the bridge. On each guest, I assigned a static IP according to the above picture, and the ens2 interface is up.

However, at this point I am not able to ping VM2 from VM1 or vice-versa. It seems like no traffic is exchanged through the vhost-client port at all. Ping fails with the Destination Host Unreachable message.

Some useful information:

ovs-vsctl show

Bridge br0
        datapath_type: netdev
        Port br0
            Interface br0
                type: internal
        Port dpdk0
            Interface dpdk0
                type: dpdk
                options: {dpdk-devargs="0000:01:00.0"}
        Port vhost-client-1
            Interface vhost-client-1
                type: dpdkvhostuserclient
                options: {vhost-server-path="/tmp/vhost-client-1"}
    ovs_version: "2.16.1"

ovs-vsctl -- --columns=name,ofport list Interface

name                : br0
ofport              : 65534

name                : dpdk0
ofport              : 6

name                : vhost-client-1
ofport              : 2

ovs-ofctl dump-flows br0

cookie=0x0, duration=104.689s, table=0, n_packets=0, n_bytes=0, in_port="vhost-client-1" actions=output:dpdk0
cookie=0x0, duration=99.573s, table=0, n_packets=4, n_bytes=924, in_port=dpdk0 actions=output:"vhost-client-1"

ovs-ofctl show br0

OFPT_FEATURES_REPLY (xid=0x2): dpid:0000b8cef64def2e
n_tables:254, n_buffers:0
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
 2(vhost-client-1): addr:00:00:00:00:00:00
     config:     0
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 6(dpdk0): addr:b8:ce:f6:4d:ef:2e
     config:     0
     state:      0
     current:    AUTO_NEG
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br0): addr:b8:ce:f6:4d:ef:2e
     config:     0
     state:      0
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

Libvirt XML configuration (relevant parts)

<domain type='kvm'>
  <name>ubuntu-server</name>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <interface type='vhostuser'>
      <mac address='52:54:00:16:a5:76'/>
      <source type='unix' path='/tmp/vhost-client-1' mode='server'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
  </devices>
</domain>

Which configuration option am I missing? I have followed several guides but still I am unable to route any traffic between my VMs.

I suspect that the problem is related to the LINK_DOWN status of the vhost-client-1 port as reported by the ovs-ofctl show command. I've tried to set that status as UP with the command ovs-ofctl mod-port br0 vhost-client-1 up. Even though the command did not fail, nothing changed.

Any thoughts?

Answer 1

Eventually, I managed to solve my problem. Vipin's answer was useful, but did not solve the issue. The configuration option I was missing was the numa option within the cpu element.

I post the working configuration file just in case it is useful for other people. The first part is about memory backing (under the domain element):

  <memory unit='KiB'>[VM memory size]</memory>
  <currentMemory unit='KiB'>[VM memory size]</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB'/>
    </hugepages>
    <locked/>
    <source type='file'/>
    <access mode='shared'/>
    <allocation mode='immediate'/>
    <discard/>
  </memoryBacking>

But we also needed the numa configuration, even if our machine had just one processor:

<cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>qemu64</model>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='lahf_lm'/>
    <feature policy='disable' name='svm'/>
    <numa>
      <cell id='0' cpus='0-1' memory='[VM memory size]' unit='KiB' memAccess='shared'/>
    </numa>
</cpu>

Answer 2

suspect that the problem is related to the LINK_DOWN status of the vhost-client-1 port

[Answer] Yes, the cause for packet not reaching VM is OVS sees the interface down. Even with the presence of a rule to forward traffic from dpdk0 to vhost-client-1 the packets will be dropped.

The primary cause of link not up can be due to the qemu configuration memory backed with hugepages not being used. Vhost-USER port created by DPDK=OVS resides on the huge page memory area and access to the same is required.

A similar DPDK-QEMU StackOverflow address the use of viritio-client (via cmdline). So please adapt the settings shared into the link to virsh xml

template:

<memory unit='KiB'>[VM memory size]</memory> 
<currentMemory unit='KiB'>[VM memory size]</currentMemory>
<memoryBacking> <hugepages/> </memoryBacking>

[EDIT-1] Based on the information shared from @ellere Vipin's answer was useful, but did not solve the issue. The configuration option I was missing was the numa option within the cpu element.

depending whether the memory used is from NUMA-SOCKET please add appropriate tag