7 Replies Latest reply on Feb 11, 2014 8:16 AM by slk2k

    Unable to PXE boot guest using SR-IOV VFs

    slk2k

      Hello,

       

      Config-centric info:

      Using HP BL460C-G8 and 560FLB (82599 based) NIC.  Host and guest OS is RHEL6.4 (latest - 2.6.32-358.2.1.el6.x86_64). Guest is defined SRIOV interfaces, but when booting, the bios doesn't appear to see any NICs  and hence does not pxe boot (seen by examining the BIOS (seabios) output in the console).  BTW, we are using libvirt and KVM.

       

      I believe I need to create an optional ROM to load into libvirt, but when trying to build for ipxe, it fails.  gpxe which ships with RH (latest - gpxe-roms-qemu-0.9.7-6.9.el6.noarch) has a option ROM (e1000-0x100e.rom), but it doesn't help.

       

      Thanks!

      Shawn

      ========================================================================

       

      # lspci | grep -i virtual

      00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 05)

      01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02)

      04:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      04:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      04:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      04:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      04:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      04:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      04:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

      04:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

       

      # lspci -n -s 04:10.0

      04:10.0 0200: 8086:10ed (rev 01)

       

      # lspci -n -s 04:10.1

      04:10.1 0200: 8086:10ed (rev 01)

       

      Followed directions here:

      http://ipxe.org/howto/romburning

       

      This is the failed build output:

      arch/i386/scripts/i386.lds:1: undefined symbol `obj_808610ed' referenced in expression

       

      XML definition for VM:

      <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

        <!--

              01-s00c09h0

        -->

        <name>01-s00c09h0</name>

        <uuid>55ca8e0c-adf9-11e2-b630-e4115b950168</uuid>

        <memory>29360128</memory>

        <currentMemory>29360128</currentMemory>

        <vcpu>7</vcpu>

        <cputune>

          <vcpupin vcpu='0' cpuset='1,17'/>

          <vcpupin vcpu='1' cpuset='2,18'/>

          <vcpupin vcpu='2' cpuset='3,19'/>

          <vcpupin vcpu='3' cpuset='4,20'/>

          <vcpupin vcpu='4' cpuset='5,21'/>

          <vcpupin vcpu='5' cpuset='6,22'/>

          <vcpupin vcpu='6' cpuset='7,23'/>

        </cputune>

        <os>

          <type arch='x86_64' machine='rhel6.4.0'>hvm</type>

          <boot dev='hd'/>

          <boot dev='network'/>

          <bootmenu enable='yes'/>

          <bios useserial='yes'/>

        </os>

        <features>

          <acpi/>

          <apic/>

          <pae/>

        </features>

        <clock offset='utc'/>

        <on_poweroff>destroy</on_poweroff>

        <on_reboot>restart</on_reboot>

        <on_crash>restart</on_crash>

        <devices>

          <emulator>/usr/libexec/qemu-kvm</emulator>

          <disk type='file' device='disk'>

            <driver name='qemu' type='raw' cache='none' io='threads'/>

            <source file='/var/lib/libvirt/images/01-s00c09h0_1.img'/>

            <target dev='vda' bus='virtio'/>

            <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>

          </disk>

          <interface type='hostdev' managed='yes'>

            <source>

               <address type='pci' domain='0x00' bus='0x04' slot='0x10' function='0x00'/>

            </source>

            <mac address='c2:a1:d9:81:00:40'/>

            <vlan>

               <tag id='910'/>

            </vlan>

            <rom bar='on' file='/usr/share/gpxe/e1000-0x100e.rom'/>

          </interface>

          <interface type='hostdev' managed='yes'>

            <source>

               <address type='pci' domain='0x00' bus='0x04' slot='0x10' function='0x01'/>

            </source>

            <mac address='c2:a1:d9:81:00:41'/>

            <vlan>

               <tag id='911'/>

            </vlan>

            <rom bar='on' file='/usr/share/gpxe/e1000-0x100e.rom'/>

          </interface>

          <serial type='pty'>

            <target port='0'/>

          </serial>

          <console type='pty'>

            <target type='serial' port='0'/>

          </console>

          <input type='mouse' bus='ps2'/>

          <graphics type='vnc' port='-1' autoport='yes'/>

          <video>

            <model type='cirrus' vram='9216' heads='1'/>

            <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

          </video>

          <watchdog model='i6300esb' action='reset'>

            <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>

          </watchdog>

          <memballoon model='virtio'>

            <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

          </memballoon>

          <!-- KVM virtio channels -->

          <channel type='pipe'>

            <source path='/var/lib/libvirt/qemu/channels/FROM-01-s00c09h0'/>

            <target type='virtio' name='virtio2host'/>

          </channel>

          <channel type='pipe'>

            <source path='/var/lib/libvirt/qemu/channels/HGC-01-s00c09h0'/>

            <target type='virtio' name='virtio_host_guest_check'/>

          </channel>

        </devices>

        <qemu:commandline>

          <qemu:arg value='-bios'/>

          <qemu:arg value='/usr/share/qemu-kvm/bios.bin'/>

        </qemu:commandline>

        <qemu:commandline>

          <qemu:arg value='-option-rom'/>

          <qemu:arg value='/usr/share/qemu-kvm/sgabios.bin'/>

        </qemu:commandline>

      </domain>

       

      Intel NIC Driver on Host:

      #modinfo ixgbe

      filename:       /lib/modules/2.6.32-358.2.1.el6.x86_64/extra/ixgbe/ixgbe.ko

      version:        3.13.10

      license:        GPL

      description:    Intel(R) 10 Gigabit PCI Express Network Driver

      author:         Intel Corporation, <linux.nics@intel.com>

       

      Modprobe config file:

      # cat /etc/modprobe.d/ixgbe.conf

      blacklist ixgbevf

      options ixgbe max_vfs=50,50

       

      VF Networking Info:

      2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1508 qdisc mq state UP qlen 10000

          link/ether e4:11:5b:94:f4:30 brd ff:ff:ff:ff:ff:ff

          vf 0 MAC c2:a1:d9:81:00:40, vlan 910

      3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1508 qdisc mq state UP qlen 10000

          link/ether e4:11:5b:94:f4:31 brd ff:ff:ff:ff:ff:ff

          vf 0 MAC c2:a1:d9:81:00:41, vlan 911

       

      Console Output:

      SeaBIOS (version seabios-0.6.1.2-26.el6)

      Press F12 for boot menu.


      Google, Inc.
      Serial Graphics Adapter 07/26/11
      SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ (mockbuild@hs20-bc2-3.build.redhat.com) Tue Jul 26 15:05:08 UTC 2011
      Term: 129x67
      8 0
      Booting  from  Hard  Disk....

      Boot  failed:  not  a  bootable  disk

       

      Booting  from  Floppy....

      Boot  failed:  could  not  read  the  boot  disk

       

      No  bootable  device.

        • 1. Re: Unable to PXE boot guest using SR-IOV VFs
          Patrick_Kutch

          Thanx for posting your question.

           

          Always love to see interesting SR-IOV questions!

           

          I will go and double check this with my experts, however I am pretty sure you cannot perform any kind of network boot over a SR-IOV interface.  The network boot (PXE, iSCSI Boot & FCoE Boot) all use the option ROM for the booting.  A SR-IOV VF does not have access to the option ROM.

           

          Again, I will double check this and get back to you, however I think what you need to do is have an emulated device assigned to the VM that can do the PXE boot and then once the VM OS is running, switch over to SR-IOV.

           

          I'll post back more information when I get it.

           

          thanx,

           

          Patrick

          • 2. Re: Unable to PXE boot guest using SR-IOV VFs
            slk2k

            I absolutely agree with you that an option ROM is needed.  Does Intel offer such a thinking for their devices??  According to libvirt documentation, there is a way to load a option ROM for the device for sriov interfaces. That is the "<rom bar='on file="/path/to/rom'/> i listed before.  I think al I need is a way to build one.

             

            from libvirt:

            rom

            The rom element is used to change how a PCI device's ROM is presented to the guest. The optional bar attribute can be set to "on" or "off", and determines whether or not the device's ROM will be visible in the guest's memory map. (In PCI documentation, the "rombar" setting controls the presence of the Base Address Register for the ROM). If no rom bar is specified, the qemu default will be used (older versions of qemu used a default of "off", while newer qemus have a default of "on"). Since 0.9.7 (QEMU and KVM only). The optional file attribute is used to point to a binary file to be presented to the guest as the device's ROM BIOS. This can be useful, for example, to provide a PXE boot ROM for a virtual function of an sr-iov capable ethernet device (which has no boot ROMs for the VFs). Since 0.9.10 (QEMU and KVM only).

            • 3. Re: Unable to PXE boot guest using SR-IOV VFs
              Patrick_Kutch

              Intel does have option ROMS to allow the devices to boot via PXE, iSCSI Boot or FCoE Boot.  However as noted before, this is not avaialble via a SR-IOV VF.

               

              While libvirt has a mechansim by which you may be able to roll your own for a VF provided you have access to the option ROM binary.

               

              Intel does not provide the option ROM other than attached to the NICs, nor have we done any kind of testing of what libvirt advertises is possible.  While I doubt this was the answer you were looking for, I've confirmed my original thesis that you can have an emulated device attached to your VM for PXE boot, then use the VF once the OS is up and running.

               

              thanx,

               

              Patrick

              1 of 1 people found this helpful
              • 4. Re: Unable to PXE boot guest using SR-IOV VFs
                slk2k

                Hi Patrick,

                 

                You are right.  This is NOT the answer I was looking for.  While Intel does not support this kind of configuration, I would be most welcome if Intel could provide (without any warranties) the option rom source so I could build my own.

                 

                Meanwhile regarding your counter proposal, I do not follow how one would accomplish that unless you are proposing some kind of hot-swap on the ethernet device.  Can you elaborate more??

                 

                Thanks!!

                Shawn

                • 5. Re: Unable to PXE boot guest using SR-IOV VFs
                  Patrick_Kutch

                  Would love to accomidate your wish, however those pesky lawers would probably fire me, even if I knew where to get those ROMs.

                   

                  It is not uncommon to assign more than one Ethernet interface to a Virtual Machine.  You can assign an emulated device that support PXE through the hypervisor that would allow you to get your OS up and running.  Then when it is up and running you could, via a script or something disable the emulated device so that only your VF was available for use.

                   

                  - Patrick

                  • 6. Re: Unable to PXE boot guest using SR-IOV VFs
                    devicenull

                    This is actually pretty easy to do.  You're on the right track with your <rom> entry to libvirt.

                     

                    So, we're going to compile iPXE to act as the boot firmware.  Follow the docs at iPXE - open source boot firmware [howto:romburning] to generate a rom file for your network card (You can compile the rom easily at ROM-o-matic.eu | Generate iPXE images | open source network boot firmware , choose 'Advanced', then 'ROM binary (flashable) image (.rom)').  I'm using an i350 chipset, so my PCI ID was 8086 device code 1520.  This leaves  you with a file 80861520.rom

                     

                    Modify your libvirt xml to include this line in your <interface>:

                     

                    <rom bar='on' file='/root/80861520.rom'/>

                     

                    Destroy and start the VM, and you should see iPXE attempting to network boot.  Out of the box, iPXE works just like the standard rom firmware.  It has a *ton* of additional features though, so take a look at the documentation!

                    • 7. Re: Unable to PXE boot guest using SR-IOV VFs
                      slk2k

                      A follow-up to the thread in case anyone is looking....

                       

                      An option rom is exactly what is needed here, but is unavailable via ipxe project (which what the rom-o-matic uses).

                      Until it gets pulled in, there is no support.

                       

                      Thanks,

                      Shawn