Notes:

  • The following configuration was employed on XenServer 6.0.2, but should work on XenServer 5.x as well
  • The solution has not been rigorously tested and is not supported by Citrix.
  • This article does not go into detail on how to use the Linux command line. Please refer to Linux documentation for further information if necessary.

Background:

As I started working with SR-IOV on XenServer I found some interesting nuances. The point of this blog is to hopefully shed some light on the topic to keep others from pulling their hair out! 🙂 So, here we go!

Digging Deeper:

The first step is to enable the Intel driver to implement SR-IOV. The procedure varies slightly depending on the version of XenServer and the card in use (1Gb vs. 10Gb). For XenServer 5.6 there’s a KB article to follow. In XenServer 6.0 the iommu setting is already enabled, but you still need to configure the driver. If you’re using 1Gb card, the driver will be igb. If you’re using a 10Gb card, the driver will be ixgbe. In /etc/modprobe.d there are two files which are named igb and ixgbe. You’ll want to modify one or both of them, depending on the cards you’re using. At a minimum you’ll want to uncomment the options line (remove the “#”). A couple of our forum posts I’ve seen suggest that you should set your driver options to:

options igb max_vfs="7,7,7,7"

…which doesn’t always make sense. I’ve seen a number of forum posts saying that users don’t see all the virtual functions they expect to see, so let’s shed some light on that. First, let’s look at that options line and break it down. It was a bit tough, but I actually found documentation on Dell’s site that details how it should be configured:

For igb: http://support.ap.dell.com/support/edocs/network/intelpro/71920/en/ug/igb.htm
For ixgbe: http://support.ap.dell.com/support/edocs/network/intelpro/71920/en/ug/ixgbe.htm

The ixgbe doc explains the max_vfs option the best:

The parameters for the driver are referenced by position. So, if you have a dual port 82599-based adapter and you want N virtual functions per port, you must specify a number for each port with each parameter separated by a comma.

The max value for the max_vfs options are:

  • igb: 7
  • ixgbe: 63

So, looking at the options line above with four 7’s, it says to enable SR-IOV on the first 4 1Gb ports it finds. In my lab situation, I have two on-board 82576 NICs, and a quad-port 82576 NIC card, so I have 6 1Gb NIC ports. The options line above only enables SR-IOV on 4 of the six ports. To enable it on all 6 ports I need to change the value to “7,7,7,7,7,7”. But after doing that, I only had SR-IOV enabled on 4 of my 6 ports. So then I’ve got two questions, which four are enabled, and why aren’t the other two enabled? To answer the questions we need to get a better understanding of how the options positions apply to the individual ports. Before enabling, here’s the grep’d output of lspci on my example system:

lspci | grep Ethernet
06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0a:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0a:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

For those that are not aware, the numbers at the beginning of the line equate to :.. A quad-port NIC typically appears on the bus as two devices on subsequent buses. So the above output would have my on-board NICs be the two functions on bus 0a, and my quad-port NIC is the two devices and 4 ports on buses 06 and 07.

To identify how the parameter values applied to each port, I enabled a single port at a time to determine the result (i.e. — “7,0,0,0,0,0”, then “0,7,0,0,0,0”, and so on). What I found is that XenServer seems to enumerate from high to low with respect to the bus number (0a, 07, 06), and low to high with respect to the function number (0,1). So the options value applies (at least for me with the igb driver) in this order:

  • 0a:00.0 on-board port 0
  • 0a:00.1 on-board port 1
  • 07:00.0 quad-card port 1
  • 07:00.1 quad-card port 2
  • 06:00.0 quad-card port 3
  • 06:00.1 quad-card port 4

Here’s an example output of the grep’d output of lspci with an options value of “0,0,7,0,0,0”:

lspci | grep Ethernet
06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0a:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0a:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

Another thing I found is that enabling SR-IOV on the on-board ports DID NOT result in virtual functions being presented. So what gives? A quick look at dmesg solved that:

igb: 0000:0a:00.0: igb_validate_option: max_vfs - SR-IOV VF devices set to 7
igb 0000:0a:00.0: not enough MMIO resources for SR-IOV

igb: 0000:0a:00.1: igb_validate_option: max_vfs - SR-IOV VF devices set to 7
igb 0000:0a:00.1: not enough MMIO resources for SR-IOV

So that explains it. There aren’t enough Memory-Mapped I/O resources to enable it. My understanding is that a BIOS update would be needed to allow for SR-IOV to operate with the on-board NICs. In my case, I’m using the latest BIOS, so I can’t use the on-board NICs with SR-IOV.

So with SR-IOV enabled on all the ports that will allow it (on my system, “0,0,7,7,7,7”), here’s the grep’d output of lspci:

lspci | grep Ethernet
06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
06:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0a:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0a:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

Now imagine the output if you were using the ixgbe (10Gb) driver with an options value of “0,0,63,63,63,63”! 🙂

We’ve now got our virtual functions enabled and can pass them to our VMs on XenServer. But how do you determine which virtual functions apply to each physical function (port)? If you look closely at the outputs above, you might notice that ports that have a function of 0 yield even-numbered virtual functions, and ports that have a function of 1 yield odd-numbered virtual functions, both on the same bus as the physical function. So if we massage the output above, here’s the breakdown:

06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
06:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

06:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
06:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
06:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

0a:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

0a:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

OK, we now have SR-IOV enabled on the NIC ports, and we have some idea of what to use to assign the virtual function to a VM, per this Citrix KB article. But we’re still not quite sure which ethN interface maps to the physical functions (ports) of the cards. We can find the information we need from dmesg, but there’s a twist. If we grep dmesg like this:

dmesg | grep -E igb.+MAC
igb 0000:0a:00.0: eth0: MAC: 00:a0:d1:e9:9a:f0
igb 0000:0a:00.1: eth1: MAC: 00:a0:d1:e9:9a:f1
igb 0000:07:00.0: eth2: MAC: 00:1b:21:66:d8:f0
igb 0000:07:00.1: eth3: MAC: 00:1b:21:66:d8:f1
igb 0000:06:00.0: eth4: MAC: 00:1b:21:66:d8:f2
igb 0000:06:00.1: eth5: MAC: 00:1b:21:66:d8:f3

Voilá! PCI info, coupled with the interface and even the MAC! Just what we needed, right? Yes and no! That’s the information we need, but the information is not accurate. Further down in dmesg you find the rest of the information:

dmesg | grep -E \(igb.+MAC\|eth.+udevd\)
igb 0000:0a:00.0: eth0: MAC: 00:a0:d1:e9:9a:f0
igb 0000:0a:00.1: eth1: MAC: 00:a0:d1:e9:9a:f1
igb 0000:07:00.0: eth2: MAC: 00:1b:21:66:d8:f0
igb 0000:07:00.1: eth3: MAC: 00:1b:21:66:d8:f1
igb 0000:06:00.0: eth4: MAC: 00:1b:21:66:d8:f2
igb 0000:06:00.1: eth5: MAC: 00:1b:21:66:d8:f3
eth4 renamed to eth4_rename by udevd [5465]
eth2 renamed to eth4 by udevd [5491]
eth3 renamed to eth3_rename by udevd [5490]
eth4_rename renamed to eth2 by udevd [5465]
eth5 renamed to eth3 by udevd [5564]
eth3_rename renamed to eth5 by udevd [5490]

There appear to be some udev rules that change the way the interface names are mapped to the PCI functions. The net of the changes on my system are that eth2 and eth4 are swapped, and eth3 and eth5 are swapped. OK, we can work with that, but there’s got to be a better way of telling us what the running config is rather than doing it in our heads or writing it down. Here’s one way:

ls -l /sys/class/net/eth*/device | cut -c 39-
/sys/class/net/eth0/device -> ../../../devices/pci0000:00/0000:00:01.0/0000:0a:00.0
/sys/class/net/eth1/device -> ../../../devices/pci0000:00/0000:00:01.0/0000:0a:00.1
/sys/class/net/eth2/device -> ../../../devices/pci0000:00/0000:00:09.0/0000:04:00.0/0000:05:04.0/0000:06:00.0
/sys/class/net/eth3/device -> ../../../devices/pci0000:00/0000:00:09.0/0000:04:00.0/0000:05:04.0/0000:06:00.1
/sys/class/net/eth4/device -> ../../../devices/pci0000:00/0000:00:09.0/0000:04:00.0/0000:05:02.0/0000:07:00.0
/sys/class/net/eth5/device -> ../../../devices/pci0000:00/0000:00:09.0/0000:04:00.0/0000:05:02.0/0000:07:00.1

This is just a long directory listing with permissions, size and modification dates removed. Note that this output maps the ethN device to the PCI function listed at the very end of the lines.

There’s one last thing we need to deal with. We need to know which ethN device maps to the physical port on the back of the box. I’ve found the easiest way to do this is to tail-follow the messages file while you plug the port you want to use into a switch:

tail -fn 0 /var/log/messages | grep igb
Aug  3 11:29:09 schizo kernel: igb: eth4 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Aug  3 11:29:14 schizo kernel: igb: eth4 NIC Link is Down

After issuing the tail command above, I plugged the port into the switch, waited for the link light to illuminate, then unplugged it from the switch. So in my case, if I want to pass a virtual function from physical function eth4 to a VM I’d need to pass one of the following:

07:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
07:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

vfns:

Now, finally, wouldn’t it be nice to have an output that would do all that heavy lifting for you? I looked around a bit, but couldn’t quickly find a command or script to implement, so I wrote a python script to do the work we need. Here’s the output of the script, and the script itself follows the output.

vfns Output
eth0
     address:  00:a0:d1:e9:9a:f0
     pci address:  0a:00.0
eth1
     address:  00:a0:d1:e9:9a:f1
     pci address:  0a:00.1
eth2
     address:  00:1b:21:66:d8:f2
     pci address:  06:00.0
     virtfn0:  06:10.0
     virtfn1:  06:10.2
     virtfn2:  06:10.4
     virtfn3:  06:10.6
     virtfn4:  06:11.0
     virtfn5:  06:11.2
     virtfn6:  06:11.4
eth3
     address:  00:1b:21:66:d8:f3
     pci address:  06:00.1
     virtfn0:  06:10.1
     virtfn1:  06:10.3
     virtfn2:  06:10.5
     virtfn3:  06:10.7
     virtfn4:  06:11.1
     virtfn5:  06:11.3
     virtfn6:  06:11.5
eth4
     address:  00:1b:21:66:d8:f0
     pci address:  07:00.0
     virtfn0:  07:10.0
     virtfn1:  07:10.2
     virtfn2:  07:10.4
     virtfn3:  07:10.6
     virtfn4:  07:11.0
     virtfn5:  07:11.2
     virtfn6:  07:11.4
eth5
     address:  00:1b:21:66:d8:f1
     pci address:  07:00.1
     virtfn0:  07:10.1
     virtfn1:  07:10.3
     virtfn2:  07:10.5
     virtfn3:  07:10.7
     virtfn4:  07:11.1
     virtfn5:  07:11.3
     virtfn6:  07:11.5
vfns Source
#!/usr/bin/python
# Copyright (C) 2012 Steve Jordahl
#
# This program is free software; you can redistribute it and/or modify 
# it under the terms of the GNU Lesser General Public License as published 
# by the Free Software Foundation; version 2.1 only.
#
# This program is distributed in the hope that it will be useful, 
# but WITHOUT ANY WARRANTY; without even the implied warranty of 
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 
# GNU Lesser General Public License for more details.
#
# vfns: list SR-IOV virtual functions

import os

info = {}

def catFile(filename):
        readfile = open(filename)
        return readfile.read().strip()

for dev in os.listdir('/sys/class/net'):
        if dev.startswith('eth'):
                info[dev] = {}
                info[dev]['address'] = catFile('/sys/class/net/' + dev + '/address')

for dev in info.keys():
        devLink = os.readlink('/sys/class/net/' + dev + '/device')
        info[dev]['pci address'] = devLink[-7:]
        os.chdir('/sys/class/net/' + dev)
        for devInfo in os.listdir(devLink):
                if devInfo.startswith('virtfn'):
                        info[dev][devInfo] = os.readlink(os.path.join(devLink, devInfo))[-7:]

for dev in sorted(info.keys()):
        print dev
        for detail in sorted(info[dev].keys()):
                print "     " + detail + ":  " + info[dev][detail]

I hope this information helps clear the fog around using SR-IOV with XenServer. Please share your experience with the use of it.