1. SR-IOV and KVM virtual machines under GNU\/Linux Debian (Jessie) Intel X520 10Gbps cards Yoann Juet @ University of Nantes, France Information Technology Services Version 1.2 (12 Jun 2015)
\n<\/p>\n
2. 2\/19 Our goal \u2022 Virtualize high-performance servers, firewalls requiring: – Low network latency and jitter – Low processor impact (I\/O) – High throughput (10Gbps or more) \u2022 Solution: Single Root \u2013 IO Virtualization (SR-IOV) – A single PCI card is showed up as multiple virtual PCI cards – Exposes n virtual interfaces from a single physical interface > Shared bandwidth 17. 17\/19 Debian: PCI passthrough with libvirt Host machine \u2022 In each guest XML file, specify the source pool, vlan id as well as (if required) the interface mac address <\/p>\n OR 1. SR-IOV and KVM virtual machines under GNU\/Linux Debian (Jessie) Intel X520 10Gbps cards Yoann Juet @ University of Nantes, France Information Technology Services Version<\/p>\n
\n 3. 3\/19 Prerequisites \u2022 Virtualization Technology for Directed I\/O: Intel VT-d or AMD-Vi – Must be supported by both the CPU and the chipset – Guest machines gain direct memory access (DMA) to PCI(e) devices, such as Ethernet cards \u2022 PCI-SIG Single Root I\/O Virtualization: SR-IOV – Must be supported by both the Ethernet cards and the BIOS – Guest machines are able to achieve ~ bare metal performance
\n 4. 4\/19 Technical environment \u2022 Dell PowerEdge R720xd – Intel Xeon CPU E5-2660 – Quad Broadcom BCM5720 1000Base-T interfaces > Logical names eth2 to eth5 – Dual Intel X520 SFP+ 10Gbps interfaces > SR-IOV compatible card > Logical names eth0 and eth1 – Operating System Debian 8 (code name “Jessie”) > Installed on both hosts and guests machines
\n 5. 5\/19 BIOS Host machine \u2022 Ensure Intel VT-d feature is enabled – System BIOS > Processor Settings > Virtualization Technology
\n 6. 6\/19 BIOS Host machine \u2022 Ensure SR-IOV BIOS option is enabled – Device Settings > [Select NIC] > Device Level Configuration > Virtualization mode = SR-IOV
\n 7. 7\/19 BIOS Host machine \u2022 Ensure SR-IOV BIOS option is enabled – Device Settings > [Select NIC] > NIC Configuration > PCI Virtual Functions Advertised = 64
\n 8. 8\/19 Debian: Starting with SR-IOV Host machine \u2022 Some Kernel requirements: CONFIG_PCI_IOV={y|m} CONFIG_PCI_STUB={y|m} CONFIG_VFIO_IOMMU_TYPE1={y|m} CONFIG_VFIO={y|m} CONFIG_VFIO_PCI={y|m} CONFIG_INTEL_IOMMU_DEFAULT_ON={y|m} \u2022 On Jessie default kernel, CONFIG_INTEL_IOMMU_DEFAULT_ON is not set require a grub special configuration\u2192
\n 9. 9\/19 Debian: Starting with SR-IOV Host machine \u2022 Edit file \/etc\/default\/grub and update the following parameter GRUB_CMDLINE_LINUX=”intel_iommu=on” \u2022 Execute the command update-grub and finaly reboot
\n 10. 10\/19 Debian: Starting with SR-IOV Host machine \u2022 Check for SR-IOV hardware support on NICs: # lspci -v \u2026 42:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01) Subsystem: Intel Corporation 10GbE 2P X520 Adapter … Capabilities: [160] Single Root I\/O Virtualization (SR-IOV) Kernel driver in use: ixgbe 42:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01) Subsystem: Intel Corporation 10GbE 2P X520 Adapter … Capabilities: [160] Single Root I\/O Virtualization (SR-IOV) Kernel driver in use: ixgbe eth0 eth1
\n 11. 11\/19 Debian: Starting with SR-IOV Host machine \u2022 Check for Intel’s VT-d IOMMU support: # dmesg | egrep -i \u201cDMA|IOMMU\u201d \u2026 Kernel command line: BOOT_IMAGE=\/vmlinuz-3.16.0-4-amd64 root=UUID=821747a0-fe42-473c-9273-391feb7f82cf ro intel_iommu=on quiet Intel-IOMMU: enabled … dmar: IOMMU 0: reg_base_addr d5000000 ver 1:0 cap d2078c106f0466 ecap f020de dmar: IOMMU 1: reg_base_addr df900000 ver 1:0 cap d2078c106f0466 ecap f020de … IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 – 0xffffff] PCI-DMA: Intel(R) Virtualization Technology for Directed I\/O \u2026 https:\/\/www.kernel.org\/doc\/Documentation\/vfio.txt
\n 12. 12\/19 Debian: Starting with SR-IOV Host machine \u2022 Activate SR-IOV on both 10Gbps interfaces with 8 VFs (64 max. allowed) per PF # echo 8 > \/sys\/bus\/pci\/devices\/0000:42:00.0\/sriov_numvfs # echo 8 > \/sys\/bus\/pci\/devices\/0000:42:00.1\/sriov_numvfs USB IDs for eth0 and eth1
\n 13. 13\/19 Debian: Starting with SR-IOV Host machine \u2022 Check for new virtual PCIe devices (Virtual Functions): # lspci … 42:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01) 42:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01) 42:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 42:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 42:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 42:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 42:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 42:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 42:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 42:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 42:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 42:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 42:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 42:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 42:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 42:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 42:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 42:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 8 VFs on the second PF (eth1) 8 VFs on the first PF (eth0)
\n 14. 14\/19 Debian: Starting with SR-IOV Host machine \u2022 Each VF behaves like a traditional network interface – below, logical names eth6 eth21\u2192 # ip link show 6: eth0:
\n 15. 15\/19 Debian: Starting with SR-IOV Host machine 9: eth6:
\n 16. 16\/19 Debian: PCI passthrough with libvirt Host machine \u2022 Assign two pools of PCIe devices to passthrough ; no need to worry about VF PCI IDs… Allocation of ressources is dynamic. <\/p>\n\r\n# vi \/etc\/libvirt\/qemu\/networks\/pf-eth0.xml \r\n<\/pre>\n
\r\n<network> \r\n <name>pf-eth0<\/name> \r\n <forward mode='hostdev' managed='yes'> \r\n <driver name='vfio'\/> \r\n <pf dev='eth0'\/> \r\n <\/forward> \r\n<\/network> \r\n<\/pre>\n
\r\n# virsh net-define \/etc\/libvirt\/qemu\/networks\/pf-eth0.xml \r\n# virsh net-start pf-eth0 \r\n# virsh net-autostart pf-eth0 \r\n# modprobe vfio \r\n# vi \/etc\/libvirt\/qemu\/networks\/pf-eth1.xml \r\n<\/pre>\n
\r\n<network> \r\n <name>pf-eth1<\/name> \r\n <forward mode='hostdev' managed='yes'>\r\n <driver name='vfio'\/> \r\n <pf dev='eth1'\/> \r\n <\/forward> \r\n<\/network> \r\n<\/pre>\n
\r\n# virsh net-define \/etc\/libvirt\/qemu\/networks\/pf-eth1.xml \r\n# virsh net-start pf-eth1 \r\n# virsh net-autostart pf-eth1 \r\n# virsh net-list\r\n<\/pre>\n
# vi \/etc\/libvirt\/qemu\/myguest.xml<\/pre>\n
<interface type='network'> \r\n <source network='pf-eth<0|1>'\/> \r\n <vlan> \r\n <tag id='<vlan_id>'\/> \r\n <\/vlan>\r\n<\/interface><\/pre>\n
\r\n# virsh define myguest.xml \r\n# virsh autostart myguest \r\n# virsh start myguest \r\n# vi \/etc\/libvirt\/qemu\/myguest.xml\r\n<\/pre>\n
\r\n<interface type='network'> \r\n <mac address='<mac-address>'\/> \r\n <source network='pf-eth<0|1>'\/> \r\n <vlan> \r\n <tag id='<vlan_id>'\/> \r\n <\/vlan> \r\n<\/interface> \r\n<\/pre>\n
\r\n# virsh define myguest.xml \r\n# virsh autostart myguest \r\n# virsh start myguest\r\n<\/pre>\n
\n 18. 18\/19 Debian: Starting Guest machine \u2022 No prerequisite, nor specific configuration on the guest linux machine \u2022 \u201ca pure\u201d Debian 8 (kernel 3.16.x) works perfectly \u2022 Virtual interfaces are using the driver ixgbevf
\n 19. 19\/19 University of Nantes \u2013 IT Services Questions Yoann (dot) Juet (at) univ\u2013nantes.fr<\/p>\n","protected":false},"excerpt":{"rendered":"