nvidia 1060 gpu passhtrough with libvirt

Question

I'm currently trying to setup a passthrough of my nvidia GPU to a windows guest. I'm actually using arch

here is what I have done so far and the issues I'm facing :

enabled iommu

dmesg|grep -i iommu
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=0b58f681-aba7-4bdd-983a-c3fb9e69201e rw cryptdevice=/dev/nvme0n1p2:cryptroot cryptkey=rootfs:/crypto/root.key intel_iommu=on iommu=pt nouveau.blacklist=1 quiet
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=0b58f681-aba7-4bdd-983a-c3fb9e69201e rw cryptdevice=/dev/nvme0n1p2:cryptroot cryptkey=rootfs:/crypto/root.key intel_iommu=on iommu=pt nouveau.blacklist=1 quiet
[    0.000000] DMAR: IOMMU enabled
[    0.039812] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.608811] iommu: Adding device 0000:00:00.0 to group 0
[    0.608819] iommu: Adding device 0000:00:01.0 to group 1
[    0.608824] iommu: Adding device 0000:00:02.0 to group 2
[    0.608829] iommu: Adding device 0000:00:04.0 to group 3
[    0.608838] iommu: Adding device 0000:00:14.0 to group 4
[    0.608842] iommu: Adding device 0000:00:14.2 to group 4
[    0.608850] iommu: Adding device 0000:00:15.0 to group 5
[    0.608854] iommu: Adding device 0000:00:15.1 to group 5
[    0.608860] iommu: Adding device 0000:00:16.0 to group 6
[    0.608865] iommu: Adding device 0000:00:17.0 to group 7
[    0.608876] iommu: Adding device 0000:00:1c.0 to group 8
[    0.608886] iommu: Adding device 0000:00:1c.4 to group 9
[    0.608897] iommu: Adding device 0000:00:1c.5 to group 10
[    0.608907] iommu: Adding device 0000:00:1d.0 to group 11
[    0.608919] iommu: Adding device 0000:00:1f.0 to group 12
[    0.608924] iommu: Adding device 0000:00:1f.2 to group 12
[    0.608928] iommu: Adding device 0000:00:1f.3 to group 12
[    0.608933] iommu: Adding device 0000:00:1f.4 to group 12
[    0.608936] iommu: Adding device 0000:01:00.0 to group 1
[    0.608947] iommu: Adding device 0000:3b:00.0 to group 13
[    0.608957] iommu: Adding device 0000:3c:00.0 to group 14
[    0.608963] iommu: Adding device 0000:3d:00.0 to group 15

iommu group of the gpu

IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106M [GeForce GTX 1060 Mobile 3GB] [10de:1c20] (rev a1)

enabled vfio

lspci -k 
01:00.0 VGA compatible controller: NVIDIA Corporation GP106M [GeForce GTX 1060 Mobile 3GB] (rev a1)
    Kernel driver in use: vfio-pci
    Kernel modules: nouveau

libvirt VM configuration

<domain type='kvm'>
  <name>windows10-gaming</name>
  <uuid>826da537-6982-40e3-a51c-f308d2c76c85</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.11'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/ovmf/ovmf_code_x64.bin</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/windows10-gaming_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <kvm>
      <hidden state='off'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough' check='none'/>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/guy/virtu/windows10-gaming.qcow2'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:96:d8:8e'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='vga' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

But when the guest boot I get this error

2018-01-11T19:57:45.154191Z qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:01:00.0

Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=

I have found that I should be able to dump this romfile buy reading the adequate file

[root@pc-guy guy]# cat /sys/bus/pci/devices/0000\:00\:01.0/0000\:01\:00.0/rom 
cat: '/sys/bus/pci/devices/0000:00:01.0/0000:01:00.0/rom': Erreur d'entrée/sortie

"input/output error" (in english)

since this solution didn't worked I tried to read it from memory, I should be able to dump it, but I need his address location in memory. I should find it with grep 'Video ROM' /proc/iomem but this command return nothing

I'm now stucked and I'm looking for a solution to make this gpu passthrough work, I don't understand what is wrong and every suggestion is welcome

Answer 1

In your boot entries, you should have vfio-pci.ids=10de:1c20 in your options. I saw your pci-id value 10de:1c20 from your dmesg output. Verify this by running lcpci -nn and also add the audio device id that goes along with the graphics card.

Also, your iommu groups may not be unique to each device unfortunately. Could you run find /sys/kernel/iommu_groups/ -type l and see if your gpu is on its own unique iommu?

I have ran into this issue as well, but was luckily solved with a BIOS motherboard update. If you have the latest update BIOS, then you will have to recompile the kernel with ACS patch.

There is a really nice guide written by GrayBoltWolf here. I followed his guide and helped me really well. In his guide, he did a video tutorial in ubuntu, but I successfully did this with a gtx 1060 running Arch Linux.

Hope this helps!

Answer 2

As far as I see this you trying to passhtrough something on notebook? It will be hard, maybe even impossible. You can take bios from https://www.techpowerup.com/vgabios/189965/189965 (maybe you will need to shrink some of starting bytes from it, check https://github.com/Matoking/NVIDIA-vBIOS-VFIO-Patcher/blob/master/nvidia_vbios_vfio_patcher.py for reference). But even if you will boot your wm, most likely nvidia control panel will show that no monitors connected to your adapter.

Passhtrough on notebooks are bad idea in general, due to way the screen shared between integrated gpu and discreete one, in most cases discrete gpu will not be able to use HDMI port or notebook screen if it passed to VM. Just forget it for now.