Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4810 Discussions

a problem with 82599 using SR-IOV in linux

idata
Employee
5,549 Views

Hi,

Now I am doing test about sr-iov. I can setup vfs using 82576 and pass it to VMs with xen 3.2 and kernel 18 in CENTOS 5.4. However, when I change it to 82599, no VF appears any more. If I use "dmesg" to see what happens, it displays "not enough MMIO resource for SR-IOV" . First ,we think that it may be caused by the OS, but we have tried several OS including Centos 5.4(i386 and X64),5.6, REDHAT 5.5,5.6 and the problem remains.Now we really don't know what's the problem.Thanks very much!

If anyone happen to know what to do with it, I hope you can give me a hand.

0 Kudos
44 Replies
Patrick_K_Intel1
Employee
2,471 Views

Thanx for using Intel Ethernet.

While I am no longer the TME responsible for virtualization, I'll jump in and see if I can lend a hand.

How many VF's did you try to instantiate with the 82599?

The usual reason for this message is that the BIOS did not properly allocate sufficient MMIO space for all of the potentila VFs. You might try updating your BIOS.

If you look at the following blog:

/community/wired/blog/2011/01/19/come-and-get-it-sr-iov-primer-document-updated http://communities.intel.com/community/wired/blog/2011/01/19/come-and-get-it-sr-iov-primer-document-updated

It will take you to my SR-IOV Primer and you can get the details in there of what and how the BIOS is supposed to calculate the MMIO space.

You can also try creating less VF's (max_vfs=5) say for the driver parameter, and see what happens.

Looking forward to seeing your response.

- Patrick

0 Kudos
idata
Employee
2,472 Views

Hi,

Thanks !

I just create 4 vfs. Even if I only just create 1, it fails. And the probelm is just the same as https://bugzilla.redhat.com/show_bug.cgi?id=523341 https://bugzilla.redhat.com/show_bug.cgi?id=523341. I have already get the vfs of 82576( gigabit ethernet ) , however when I just do the same things for 82599 ( gigabit ethernet ), it fails. I will try to update the BIOS. And I will read the document you recommended carefully!

Thanks a lot!

liyu

0 Kudos
idata
Employee
2,472 Views

Hi, Liyu

I can confirm that 82599 SR-IOV works in the system that also have 82576. So, both work with SR-IOV (SL6.0/kvm in my case, guest is also SL60 or Windows). The system is Supermicro X8TD.

cheers

Artem.

0 Kudos
idata
Employee
2,472 Views

Hi, Artem

we have traced the path of resource allcating,then we find the first time the pcibios allocate resource failed for the bus which containing the 82599,after this,we can see no proper resource availabe in the bus. In this case, we let the linux kernel do the work that bios should do. However, when we get the BIOS to allocate resources, it still get a problem of resource allocation failed (we have update the BIOS to the latest virsion).

Might it be that the kernel get a bug? We would go on reading the pci part of linux kernel. Could it be a solution ?

cheers

Liyu

0 Kudos
idata
Employee
2,472 Views

Hi Liyu,

Ok, I recalled some details and looked again at my system.

First, I had to turn two options in BIOS (on my supermicro board): one was for VT-d, the other was to explicitely enable SRIOV. Check it out in PCI section of you BIOS, and probably it's a good idea to walk through BIOS setting s with BIOS manual from the manufacturer.

The other things is : run

lspci -vv -s $pci_id

and check if it references SR-IOV capabilities, like my X520

Capabilities: [160] Single Root I/O Virtualization (SR-IOV)IOVCap: Migration-, Interrupt Message Number: 000IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+IOVSta: Migration-Initial VFs: 64, Total VFs: 64, Number of VFs: 7, Function Dependency Link: 00VF offset: 128, stride: 2, Device ID: 10edSupported Page Size: 00000553, System Page Size: 00000001Region 0: Memory at 00000000fa800000 (64-bit, non-prefetchable)Region 3: Memory at 00000000fa900000 (64-bit, non-prefetchable)VF Migration: offset: 00000000, BIR: 0

So it's important that "Region" memories were not at 0

Another things is that MSI-X is enabled (also checked in the output of lspci), and yet another thing to look for is ARI capabiliti and hierarhy support.

Also, just try a different PCI-e slot, and try to get other cards out and have X520 alone.

cheers

Artem.

0 Kudos
idata
Employee
2,472 Views

Artem,

 

I have a similar setup (Centos 6.0, Supermicro MB, 82576 NIC) but only the Linux guests work properly with the 82576 VF. The Windows ones (both 2008R2 and 7 x64) see an Ethernet Controller but the intel driver does not detect them. I am using KVM.

 

Is there anything in your setup that configures the NICs in a particular way?

 

Is there any particular configuration required in the VMs xml file?

 

This is the output of lspci:

 

sudo lspci -vv -s 04:00.1

 

04:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

 

Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter

 

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+

 

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-   Latency: 0, Cache Line Size: 64 bytes

 

Interrupt: pin B routed to IRQ 17

 

Region 0: Memory at fdfe0000 (32-bit, non-prefetchable) [size=128K]

 

Region 2: I/O ports at d800 [size=32]

 

Region 3: Memory at fdfdc000 (32-bit, non-prefetchable) [size=16K]

 

Expansion ROM at fe000000 [disabled] [size=4M]

 

Capabilities: [40] Power Management version 3

 

Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)

 

Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-

 

Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

 

Address: 0000000000000000 Data: 0000

 

Masking: 00000000 Pending: 00000000

 

Capabilities: [70] MSI-X: Enable+ Count=10 Masked-

 

Vector table: BAR=3 offset=00000000

 

PBA: BAR=3 offset=00002000

 

Capabilities: [a0] Express (v2) Endpoint, MSI 00

 

DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us<p>  ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+

 

DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+

 

RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-

 

MaxPayload 128 bytes, MaxReadReq 512 bytes

 

DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-

 

LnkCap: Port # 0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <4us, L1 <64us<p>  ClockPM- Surprise- LLActRep- BwNot-

 

LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+

 

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

 

LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

 

DevCap2: Completion Timeout: Range ABCD, TimeoutDis+

 

DevCtl2: Completion Timeout: 16ms to 55ms, TimeoutDis-

 

LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB

 

Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-

 

Compliance De-emphasis: -6dB

 

LnkSta2: Current De-emphasis Level: -6dB

 

Capabilities: [100] Advanced Error Reporting

 

UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

 

UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

 

UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-

 

CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+

 

CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+

 

AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-

 

Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-a1-04-f2

 

Capabilities: [150] Alternative Routing-ID Interpretation (ARI)

 

ARICap: MFVC- ACS-, Next Function: 0

 

ARICtl: MFVC- ACS-, Function Group: 0

 

Capabilities: [160] Single Root I/O Virtualization (SR-IOV)

 

IOVCap: Migration-, Interrupt Message Number: 000

 

IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy-

 

IOVSta: Migration-

 

Initial VFs: 8, Total VFs: 8, Number of VFs: 7, Function Dependency Link: 01

 

VF offset: 128, stride: 2, Device ID: 10ca

 

Supported Page Size: 00000553, System Page Size: 00000001

 

Region 0: Memory at 00000000fd340000 (64-bit, non-prefetchable)

 

Region 3: Memory at 00000000fd360000 (64-bit, non-prefetchable)

 

VF Migration: offset: 00000000, BIR: 0

 

Kernel driver in use: igb

 

Kernel modules: igb

 

Any suggestions?

 

Thank you very much for your help

 

Message was edited by: chbarg (changed font size)

0 Kudos
Patrick_K_Intel1
Employee
2,473 Views

Windows 7 is a client OS, as such there are no drivers available for the VFs.

Due to resource contstaings, Intel hs not yet developed a VF driver for the 82576 VF's for Windows Server 2008R2 - though there are drivers for Windows Server 2008.

0 Kudos
idata
Employee
2,473 Views

Patrick,

Thank you for the reply.

I will be waiting for the drivers for 2008 R2 and Win 7.

Thank you very much for your help !!!

0 Kudos
idata
Employee
2,473 Views

I can see VF of 82599 now after I reduce the space proccessed by the VGA in BIOS

0 Kudos
idata
Employee
2,473 Views

Hi,

I met the same problem as liyu, but I use Fedora 15!!

Fedora 15 support 82599 SR-IOV function?

liyu, could you kindly tell me how to "reduce the space processed by the VGA in BIOS"

Thanks!

Mark

0 Kudos
idata
Employee
2,473 Views

Hi,

My OS is Fedora 15, BIOS is version 2.00.1201 AMI, ASUS server.

I have enabled the Vd-T and Vd-X. It seems no other setting in BIOS about SR-IOV.

Mark

0 Kudos
Mark_H_Intel
Employee
2,473 Views

Hi Mark,

I did some resarch on the Fedora Project site. Fedora 15 uses a renamed 3.1 kernel aka 2.6.41.x kernel. See https://fedoraproject.org/wiki/Kernel https://fedoraproject.org/wiki/Kernel for details. That kernel has support for SR-IOV, so your OS should work. I am not sure what drivers are included in the distribution package, but you can always get the latest PF and VF drivers from http://downloadcenter.intel.com http://downloadcenter.intel.com.

Unfortunately, the lack of any BIOS settings for SR-IOV makes me think that your server hardware (or the BIOS) does not support SR-IOV. You need BIOS and hardware support on the platform. You should probably check with ASUS about SR-IOV support.

Mark H

0 Kudos
Patrick_K_Intel1
Employee
2,473 Views

Sorry for the delay - I was in Antarctica and just got back.

Mark H. is correct, sounds like your server simply does not have a SR-IOV enabled BIOS. Please check with your server vendor and best of luck.

Patrick

0 Kudos
idata
Employee
2,473 Views

Hello,

I just downloaded the Intel PROWinx64.exe V17 dated 03/23/2012 that should include support for VF under Win 2008 R2 from:

http://downloadcenter.intel.com/detail_desc.aspx?agr=Y&DwnldID=18725 http://downloadcenter.intel.com/detail_desc.aspx?agr=Y&DwnldID=18725

From inside guest VM I can see the NIC with missing drivers. Win 2008 R2 reports the correct hardware ID:

PCI\VEN_8086&DEV_10CA&SUBSYS_A03C8086&REV_01

 

PCI\VEN_8086&DEV_10CA&SUBSYS_A03C8086

 

PCI\VEN_8086&DEV_10CA&CC_020000

 

PCI\VEN_8086&DEV_10CA&CC_0200

Are these the correct hardware IDs for the 82576 VF?

Linux guests work perfectly using the VF so I assume that the MB and Bios correctly supports SR-IOV.

Thank you very much for your help

PS:

MB = SuperMicro H8SCM BIOS V 2.0

CPU = Opteron 4180 or Opteron 4226

OS = Centos 6.2 x64 with all updates

Guest OS = Win 2008 R2

0 Kudos
Patrick_K_Intel1
Employee
2,473 Views

The Windows Server 2008 R2 VF support for the Intel 82576 Ethernet controller is still under development. Looks like it will be available maybe around August or so.

I wish we had drivers for everything available, unfortunatly engineering resources are limited so we have to prioritize some drivers over others.

0 Kudos
idata
Employee
2,473 Views

Thnak you.

I will try again around August/September.

Is there a way to subscribe to notices that a new version of the NIC drivers is available?

Thanks again for your help

0 Kudos
Mark_H_Intel
Employee
2,473 Views

You can get an RSS feed from the Intel Download Center although getting the feed is not obvious. Here are steps for signing up for an RSS feed for any product:

  1. Go to http://downloadcenter.intel.com/Default.aspx Download Center

     

  2. Use theFind downloads by product name feature to select the product you want.

     

  3. The link for the RSS feed is in the right-hand column.

     

http://feeds.downloadcenter.intel.com/rss/?p=3024&lang=eng http://feeds.downloadcenter.intel.com/rss/?p=3024&lang=eng is the English language link to the RSS feed for downloads associated with the Intel® 82576 Gigabit Ethernet Controller.

The RSS feed will show updated files everytime a new software version is released, but the feed will not tell you what is new. Adding new VF driver support is a big change that would be reflected in the What's New section of the release notes (readme.txt).

You can also see the software change highlights listed at http://www.intel.com/support/network/sb/CS-006333.htm What is new in the latest Intel® Ethernet Software release?.

I should also mention that if you require a VF driver for Windows Server 2008 R2 on a gigabit adapter now, the VF driver is available for the http://www.intel.com/content/www/us/en/network-adapters/gigabit-network-adapters/ethernet-server-adapter-i350.html Intel® Ethernet I350 Server Adapter Family. I am not sure if buying new adapters makes sense for you, but I wanted you to know that the VF driver was available so you can consider all your options.

Mark H

0 Kudos
idata
Employee
2,473 Views

Mark,

Thank you very much for the information.

I subscribed to the RSS and I am evaluating purchasing an I350T2 for my home server.

I am currently running Centos 6.2 & KVM with the open source drivers.

Is there a big advantage in installing the Intel drivers for Linux on the host?

Thanks again for your help

0 Kudos
Patrick_K_Intel1
Employee
2,473 Views

The kernel drivers are made by Intel as well. Same base source code as you find on Source Forge.

There is not a big difference between the two.

0 Kudos
idata
Employee
2,341 Views

Hi ,

Another question about 82599 chip...

There are two 10G ports in 82599, each one has one PF and maximum 63 VFs. I know in one port, the PF and VF can communicate with each other by internal switch in 82588, how about the VF between different port? Can they communicate each other by the internal switch in 82599?

Thanks!

Mark

 

0 Kudos
Reply