I have an Arch Linux host which runs virtualised router.
When using a LXC guest as router, everything is fine. I get 100MBits Up/Down and almost no CPU usage at all.
However, when I use libvirt gest (pfSense FreeBSD) as a router, whenever there is heavy network traffic going through the guest, the CPU usage goes unreasonably high (up to 100%) but the worst thing is that the network throughput is halved! I get 45-49Mbits max.
Host doesn’t support PCI pass through, so this is my config for the libvirtd VM:
Nic1 (wan)
Network source: Direct ‘eth0’
Source mode: passthrough
Device model: virtio
Nic2 (lan)
Bridge name: br0
Device model: virtio
I tried e1000 instead but it changes absolutely nothing.
Host CPU: AMD A4-5000 Kabini
Guest CPU: default or Opteron_G3
This has been so since over a year now, since I started using KVM. If I do not solve this problem, I will have to dump libvirt because such performance is unacceptable.
It is pretty hard to diagnose these sort of problems with such limited information. Definitely don't use e1000 or any other NIC model - virtio-net will offer the best performance of any virtualized NIC. Make sure the host has /dev/vhost-net available as that accelerates guest NIC traffic in host kernel space.
If you want to use a guest as a high performance network routing appliance though, there's quite a few ways to tune it the VM in general. Pinning the guest vCPUs to specific host physical CPUs, and keeping other guests off these CPUs ensures the guest won't get its cache trashed by being pre-empted by other processes. Next up, use huge pages for the guest RAM to massively increase the TLB cache hit rate for guest memory access. If the host has multiple NUMA nodes, makes sure the guest CPU and guest RAM (hugepages) are fixed to come from the same host NUMA node. Similarly ensure IRQ handling for the host NIC used by the guest has affinity set to match the pCPUs used by the guest.