Skip navigation

In an earlier post IPv6 support was explained.  I got a question off line about wanting to know about TCP/IPv6 and UDP/IPv6 offloads of the TCP/UDP Checksums. At first glance, its easy to say that IPv6 doesn't have header checksums. But as IPv6 payload, the TCP and UDP headers and there checksums are still valid.  So its a great albeit subtle question.

 

Short answer: After the Intel® 82544 Gigabit Ethernet Controller, Yes.  Before it:  No.  The 82544 hasn't been offered for a while, making the really short short answer to be “Yes”.

 

But there is a lot of fun to be had giving the Long answer:  Danger HTML table!!

 

Here is each as described by the Open Source Development manual for each HW family from  e1000.sf.net or from the datasheet that was current at time of publishing this article.

 

Here is the TCP/UDP Checksum when working with IPv6 details:

 

Packet Type

82540

82541

82545/82546

82571/2
82573

82574

ESB2 MAC

82575

82576

82598

82599

IPv6 + TCP packets

Yes

Yes

Yes

Yes

Yes

Yes

IPv6 with next header options: Hop by hop

Yes

Yes

Yes

Yes

Yes

Yes

IPv6 with next header options: Destination Options

Yes

Yes

Yes

Yes

Yes

Yes

IPv6 with next header options:Security Options (AH/ESP)

No

No

No

Yes

No

Yes

 

Thanks for using Intel® networking products.

Stumbled upon a great article:  10Gbit Ethernet: Killing Another Bottleneck? that details some of the bottlenecks having to do with Ethernet performance (both throughput and CPU utilization) in a virtualized environment.

 

 

I thought it was a great article not only because it refers to and uses some graphic from a paper I wrote two years ago on Virtual Machine Device Queue (VMDq) (available here), but that it also shows the tremendous performance benefits that can be realized using VMware* NetQueue and Intel VMDq technologies.

 

 

While the article incorrectly identifies Intel VMDq technology as a SR-IOV solution (I’ve written the author of the article explaining what the difference is, hopefully the article will soon be updated), it does show the benefits of Intel’s 10GbE solution in a virtualized environment.

 

 

I encourage you to go and read this article and take a look at the results they find.  After you have taken a look at it, you can then go look at a small video I put together last year that essentially shows the same results that the article found.  The video is here.

 

 

 

If you updated your wired Ethernet driver In Windows XP* recently, you might have encountered a bug that kept your Intel® PRO/1000 PL Network Connection from connecting to your network.  You might also know this connection as the Intel® 82573L Gigabit Ethernet Controller.

Here is what you can do to fix your Intel® PRO/1000 PL Network Connection that stopped working after upgrading your driver.

 

1. Download the updated software CD, E1EUpdtCD_211237.zip

 

2. Uninstall Intel® Network Connections from “Add or Remove Programs” in the Windows control panel

 

3. Right click on the folder and then click Extract All.

 

4.  In the Extraction Wizard, use the Browse button to choose the location where you want to extract the CD files. The Desktop is a convenient place for finding the files after extraction.

 

5. After extraction is finished, you can find the E1EUpdtCD_211237 folder in the location you chose in the previous step.

 

6. Double-click the E1EUpdtCD_211237 folder.

 

7. Double-click Autorun.exe. The CD splash screen opens.

 

8. Click Install drivers and software

 

9. Follow the installation wizard prompts.

 

10.  Installation is complete. You may delete the E1EUpdtCD_211237 folder.

 

This fix will be included as part of the upcoming version 15.2 software release planned for May.

 

The zip file is huge, about 162 MB. If you want to download a smaller file, you can try an older version of the driver that did not have this issue. You can download version 14.6 of PROXP.exe. The E1EUpdtCD contains later bug fixes that were not in version 14.6, but version 14.6 will probably work OK for most people.

 

We are confident that the update will fix the loss of connectivity with the Intel® PRO/1000 PL Network Connection. Nevertheless, if the update does not work for you, feel free to post details about what you did and about the issue you see.

 

NOTE: This E1E Update CD is only recommended for use with the Intel® PRO/1000 PL Network Connection in Windows XP*.

So Intel has this new super-cool SR-IOV technology that allows you to directly assign multiple VMs to an Intel NIC based on the Intel® 82576 GbE Controller.  But how, you may be wondering do you get it to work?  I’ve written a document that explains this, and am going to make a quick blog out of it.  I hope you find it useful, if you do, then my name is Patrick, if you don’t, then my name is Paul.:-)

 

It should be noted that there is a bug in the version of the Xen kernel that shipped with Red Hat* 5.4 Xen that causes the SR-IOV performance to be poor.  The upcoming Red Hat 5.5 release will have the fix in it.

1  Introduction

This document is intended to help those wishing to setup and configure SR-IOV on Red Hat 5.4 Xen with the Intel® 82576 GbE controller.

Included within these instructions are step-by-step procedures for installing Red Hat 5.4 Xen along with configuring the installation for SR-IOV usage and how to assign a SR-IOV Virtual Function to a Virtual Machine.

The server where Red Hat 5.4 Xen is to be installed must SR-IOV.  There are a number of platform requirements for SR-IOV that include BIOS support and I/O re-mapping capability such as that available as part of the Intel® Virtualization Technology for Directed I/O (Intel® VT-d).

In addition a SR-IOV capable Ethernet controller is required.  The system used for the purposes of this paper has an Intel® 82576 GbE controller as the LOM, thus meeting the requirement for a SR-IOV capable Ethernet controller.

 

2.1 Server Used

At the time of the writing of this document there are very few commercially available servers that support SR-IOV.  The system used for this paper is a Supermicro X8DTN+ with the UEFI BIOS which supports SR-IOV.

In addition to the on-board Intel® 82576 GbE Controller, an Intel® 82572EI GbE Controller has been added to the server to be used for non SR-IOV traffic.

2.2 BIOS Configuration

There will be some sort of BIOS configuration required to enable SR-IOV for the server.  This section details the configuration required for the Supermicro X8DTN+.

2.2.1 Enable Intel® VT-d

This configuration is found under the ChipsetàNorth BridgeàTylersburg IOH ConfigurationàIntel VT for Directed I/O Configuration menu.

bios_setup_vtd1.JPG

Figure 1 BIOS SR-IOV Configuration


2.2.2 Enable Intel® Virtualization Technology (VT-x)

In addition to enabling Intel® VT-d, the hypervisor itself also requires Intel VT-x to be enabled.  This is done through the BIOS option of:

Advanced->CPU Configuration->Intel Virtualization

bios_enablt_vtx1.JPG

Figure 2 Enable Intel VT-x

The first part of the installation is just like normal installation of a Red Hat distribution; simply put in the CD/DVD and boot up the computer.

3.1 Installation/Subscription Number

When prompted for the Installation Number dialog box, enter the appropriate Installation Number:

Installation_Number.jpg

Figure 3 Installation Number

It is vital that a valid Installation/Subscription number be provided at this point.  If this is not done then the option to select Virtualization support (Xen) will not be available.

Continue with installation.

3.2 Selecting Virtualization Support

If a valid Installation/Subscription number was entered (see section 3.1) then a dialog should appear as follows:

select virtualization support.jpg

Figure 4 Select Virtualization Support

If so, make sure Virtualization is selected and continue with installation as normal.

If, however the screen appears as:

install_without_number.jpg

Figure 5 Virtualization Option Not Available

This is an indication that a valid Installation/Subscription number was not entered as described in Section 3.1.

At this point Red Hat Xen 5.4 should be installed and running.  It will look just like ‘normal’ Red Hat Linux.  One difference will be the ability to launch the Virtual Machine Manager from the ApplicationàSystem Tools menu.

Additionally information about the Xen installation can be obtained from a command prompt by entering the command:

#xm info

4.1 Checking to see if any SR-IOV VF’s are Present

Now that Xen in installed and running, you may want to see if any SR-IOV Virtual Functions exist on the system (they won’t).  To verify this, enter the following command:

#lspci | grep Eth

The resulting output should look very similar to the following:

01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

04:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)

This shows three Intel® Ethernet controllers, including the dual port Intel® 82576 GbE controller; however no Virtual Functions appear yet.

4.2 Modifying modprobe.conf

The modprobe.conf file, located in the /etc/ directory needs to be updated with some options for the Intel igb driver.  The additions to the file are as follows:

options igb max_vfs=7

blacklist igbvf

options loop max_loop=128

Make these modifications and save the file.

The resulting file I ended up with is as follows:

alias eth0 e1000e

alias eth1 igb

alias eth2 igb

alias scsi_hostadapter ahci

options igb max_vfs=7

blacklist igbvf

options loop max_loop=128

 

4.2.1 Explanation of Settings

·         options igb max_vfs=7

This option tells the igb driver to allocate sufficient SR-IOV resources for 7 Virtual Functions per port.

·         blacklist igbvf

The blacklist line makes it so modules ignore any igbvf devices that show up in the hypervisor PCIe space

·         options loop max_loop=128

By default Linux only has 7 loop devices.  SR-IOV requires many more, this option increases the number to 128

4.3 Modifying grub.conf

The grub.conf file presents options on how to load the kernel when booting.  This file is located in the /boot/grub/ directory.

This file needs to be updated to indicate that SR-IOV capability is desired.  This is accomplished by adding a single parameter to one of the lines in the grub.conf file.  And 2 additional parameters to another line.

Specifically “pci_pt_e820_access=on” needs to be added to the “module /vmlinuz-*” line.  While the kernel /xen.* line needs “iommu=1” and “msi=1” added.

My preference is to make 2 boot options, leaving the original one and making an alternate with SR-IOV enabled. To do this simply copy and paste the existing configuration to a new entry, rename it and then set the default to be the SR-IOV entry.  The resulting grub.conf file is as follows:

# grub.conf generated by anaconda

#

# Note that you do not have to rerun grub after making changes to this file

# NOTICE:  You have a /boot partition.  This means that

#          all kernel and initrd paths are relative to /boot/, eg.

#          root (hd0,1)

#          kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00

#          initrd /initrd-version.img

#boot=/dev/sda

default=1

timeout=30

splashimage=(hd0,1)/grub/splash.xpm.gz

hiddenmenu

title Red Hat Enterprise Linux Server (2.6.18-164.el5xen)

       root (hd0,1)

       kernel /xen.gz-2.6.18-164.el5

       module /vmlinuz-2.6.18-164.el5xen ro root=/dev/VolGroup00/LogVol00 rhgb quiet

       module /initrd-2.6.18-164.el5xen.img

 

title Red Hat Xen (2.6.18-164.el5xen)

       root (hd0,1)

       kernel /xen.gz-2.6.18-164.el5 iommu=1 msi=1

       module /vmlinuz-2.6.18-164.el5xen ro root=/dev/VolGroup00/LogVol00 rhgb quiet pci_pt_e820_access=on

       module /initrd-2.6.18-164.el5xen.img

After doing this, the “Red Hat Xen (2.6.18-164el15xen)” option is default.  When the system boots next time, if you press a key at grub startup you will see something like Figure 6.

4.3.1 Explanation of Settings

·         default=1

o    This indicates that of the two boot options now in the grub.conf file, the default will be the newly created Red Hat Xen (2.6.18-164.el5xen) option.

·         iommu=1

This option tells the kernel to enable SR-IOV

·         msi=1

This option tells the kernel to enable MSI-X interrupts

·         pci_pt_e820_access=on

This option enables direct assignment of a PCIe device to a Virtual Machine

boot options.jpg

Figure 6 Grub Menu, with SR-IOV Option

4.4 Checking Again for SR-IOV VF’s

After making the changes outlined in Sections 4.2 and 4.3 reboot the server.  Now try once again to see if there are SR-IOV Virtual Functions available for assignment to a Virtual Machine.

To verify this, enter the following command:

#lspci | grep Eth

The resulting output should look very similar to the following:

01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

01:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

01:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

04:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)

 

Note that as before, the two Intel® 82576 GbE controllers are still there, along with fourteen Virtual Functions.  Recall in Section 4.2 the options set 7 VF’s per port; with two ports that presents 14 Virtual Functions for assignment to a Virtual Machine.

If for some reason you do not see something similar, double check to make sure that the configuration steps in Sections 4.2 and 4.3 are correct – a simple typo will prevent things from working properly.

This section will describe a mechanism for adding a SR-IOV VF to a Virtual Machine.  There are several ways to accomplish this - the explanation presented here will utilize the Virtual Machine Manager GUI provided with the Red Hat 5.4 Xen.

The steps outlined here will begin with the creation of the Virtual Machine.  If you wish to add a Virtual Function to an existing Virtual Machine, skip to step 5.2.

5.1 Creating a Virtual Machine

Run the Virtual Machine Manager application via the Applications->System Tools menu.

5.1.1 Step 1 – Start Create Wizard

Select the localhost in the window, and then the ‘New’ button.

Create_New_VM.jpg

Figure 7 Virtual Machine Manager

Follow the steps within the wizard to create a new virtual machine until you come to the point where the wizard prompts for information about the network.

5.1.2 Step 2 – Network Configuration

When the Virtual Machine creation wizard prompts for information about the Network to be used, simply go with the default of ‘Virtual Network’

Create_New_VM_VM_Install_Network.jpg

Figure 8 Network Setup

Finish with the wizard.  When the Virtual Machine begins to boot up for the 1st time, power it down using the Virtual Machine Manager.

5.2 Assigning the SR-IOV Virtual Function

Select the powered down Virtual Machine from the Virtual Machine Manager. 

5.2.1 Step 1 Hardware Configuration Tab

Then select the hardware tab.

If an emulated NIC already configured, it can be removed using this panel.

REmove_default_nic.jpg

Figure 9 Removing Default NIC

  Step 2 – Add Hardware

Select the Add Hardware button at the bottom of the window.  The Adding Virtual Hardware window will appear:

Add_Virtual_Hardware_Select_Physical.jpg

Figure 10 Adding Virtual Hardware

Select “Physical Host Device” from the dropdown list, press the “Forward” Button”

5.2.2 Step 3 – Select 82576 Virtual Function

From the Device drop down list, scroll down until the 82576 Virtual Function devices appear and select one.

select_physical_device.jpg

Figure 11 Select a Virtual Function

After selecting the Virtual Function, finish the wizard and boot up the Virtual Machine.  The new hardware will be discovered and depending upon the guest Operating System the driver will either installed automatically or you will be prompted for a driver CD.

Install the Intel® 82576 Virtual Function Driver and enjoy the SR-IOV functionality of the Intel® 82576 GbE Controller.

5.2.3 Virtual Function Number and Physical Port Relationship

Care should be taken when selecting the Virtual Function to be assigned to a Virtual Machine.  Each VF has a PCIe BUS:Device:Function ID’s.  As seen in Figure 11 the ID’s for the dual port Intel® 82576 GbE controller are numbered 01:10:n and 01:11:n. 

If the last number (n) is an even number, then the Virtual Function is on Physical Function (port) 0.  If the number (n) is odd, then the Virtual Function is on Physical Function (port) 1.

Note: Care should be taken not to assign the same Virtual Function number to more than one Virtual Machine.  The GUI will allow this to occur, however when the Virtual Machines are run, the first one loaded will function properly, however any subsequently loaded Virtual Machines that are assigned the same Virtual Function will immediately fail with an error as shown in Figure 12.

add_vf_already assigned.jpg

Figure 12 Attempt to Use the Same VF Twice

Filter Blog