Increase Your BMC Network Performance by Over 800%!

Your friendly neighborhood manageability nerd here announcing my latest whitepaper.

The NC-SI (Network Controller Sideband Interface) is a connection between a Network Controller and a BMC.  This DMTF defined interface is designed to provide 100Mbps full duplex connectivity to a BMC.

Over a year ago I began researching why I was having customers were complaining to me that the NC-SI performance under some circumstances was horrible, in most reports only a few megabits per second.

After literally hundreds of tests and countless hours of research and tweaking of different things, I began to understand what the problem was and how to go about providing a solution.
Figure 1 Network Performance Streaming to BMC

What I found was that the NC-SI interface worked exactly as it was supposed to, however the BMC itself needs to take into account that it is using NC-SI and that NC-SI does not have as much buffer space as the BMC itself has.

I tested with not only Intel Network Controllers, but also another well-known manufacturer of Network Controllers.  The results were pretty much identical, very poor performance until I modified the BMC to be a bit more intelligent.

Making changes detailed in the whitepaper, average performance when streaming data to the BMC went from less than 8Mbps to nearly 65Mbps.  The performance would be even better, however a BMC is not a super-powerful processor and it can’t yet handle 100MBps data rates.

All my research, conclusions and actual sample data is included within the whitepaper.   I also provided an overview of what NC-SI is for the uninitiated.

I hope you find it of use.  If you do please comment so we know if folks actually read these docs .

The paper is available here:  https://www-ssl.intel.com/content/www/us/en/ethernet-controllers/nc-si-overview-and-performance-notes.html

It has been nearly 3 years since I wrote my first blog, and it was on setting up SR-IOV in Red Hat
Xen (for reference, that blog is here).

It has been a longand interesting road for SR-IOV since then, I’ve written several documents and
made a few videos on the topic, available here).

One of the lastmajor virtualization Operating Systems has now added support for SR-IOV, VMware
now provides the capability to utilize SR-IOV.

My talented co-workers have written a very nice step-by-step guide (with lots and lots
of pictures).  This very nice paper is located here.

We hope you enjoy the latest addition to our growing collection of documents for virtualization
technologies available within Intel® Ethernet Devices.

Enjoy!

- Patrick

OK, maybe not ALL you ever wanted to know, but all we've shared with you up until now, all in one place!

It has been nearly 2½ years since I posted my first ever Blog, it was about

Setting up Red Hat 5.4 Xen* for SR-IOV using the Intel® 82576 GbE.  Since that time I’ve managed to post a blog now and then, usually pointing you to a new paper or video I’ve published.

The very smart fellow who is now responsible for the Intel Ethernet Virtualization technology has continued to write great documents on how to configure things such as SR-IOV in different environments.

So we have managed to accumulate a fairly nice list of documents and videos that talk about SR-IOV and one usage of it, Flexible Port Partitioning.

To make it easier for you, I’ve decided to compile a list of all these documents, videos and blogs in one place as a nice reference.  So without any more fanfare from me, here are the various docs that we have written:

SR-IOV Background, Introduction

·       SR-IOV Primer [Document]

o   http://www.intel.com/content/www/us/en/pci-express/pci-sig-sr-iov-primer-sr-iov-technology-paper.html

·       SR-IOV Explanation [Video]

o   http://www.youtube.com/watch?v=hRHsk8Nycdg

SR-IOV Configuration

·       Setting up Red Hat 5.4 Xen for SR-IOV using the Intel 82576 GbE [Blog]

o   http://communities.intel.com/community/wired/blog/2010/03/01/setting-up-red-hat-54-xen-for-sr-iov-using-the-intel-82576-gbe

·       Using Intel® Ethernet and the PCISIG* Single Root I/O Virtualization(SR-IOV) and Sharing Specification on Red Hat* Enterprise Linux* Technical Brief [Document]

o   http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ethernet-x520-sr-iov-red-hat-tech-brief.html

·       How to Configure Intel® X520 Ethernet Server Adapter Based Virtual Functions on SuSE*Enterprise Linux Server* using Xen* [Document]

o   http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ethernet-x520-suse-linux-xen-tech-brief.html

• How to Configure Intel® X520 Ethernet Server Adapter Based Virtual Functions on Citrix*XenServer 6.0* [Document]
  • http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-x520-citrix-xenserver-tech-brief.html

Intel Flexible Port Partitioning

·       Intel Flexible Port Partitioning using SR-IOV Demonstration

o   http://www.youtube.com/watch?v=bOMB9RsQfo4 [Video]

·       An Introduction to Intel Flexible Port Partitioning Using SR-IOV Technology Technical Brief [Document]

o   http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/10-gbe-ethernet-flexible-port-partitioning-brief.html

·       Configure QoS with Intel® Flexible Port Partitioning [Document]

o   http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/config-qos-with-flexible-port-partitioning.html

SR-IOV is not just for virtualization anymore!

In my blog posting last week, I pointed you to a video  I made discussing and demonstrating Intel Flexible Port Partitioning (FPP).  If you haven’t watched the video yet, I think it is worth a view (though I am admittedly biased ).

FPP is a new way to look at using SR-IOV to partition up a single, discrete Intel Ethernet connection into multiple Ethernet devices within an open source Operating System.   The quickest and easiest way to understand what I’m talking about is to watch the video.

We have also finished and published a whitepaper that discusses FPP in more details.  It discussed what it is, how it works and how it is useful for both Intel 10 Gigabit 1 Gigabit controllers.

The whitepaper: An Introduction to Intel Flexible Port Partitioning Using SR-IOV Technology Technical Brief is intended to be the 1^st part of a 2 part series.  We are already working in the 2^nd part, which will detail the usage of the different tools used to configure FPP and some additional usage models of FPP.

Not sure what SR-IOV is?  Click on the link to see a video explanation.  I also have the PCI-SIG SR-IOV Primer you might want to check out.

We hope you find the paper of use.  Enjoy!

My time as the virtualization guy in the Intel LAN group has all but ended.  My last major task will be over in just one more week.  I will be co-teaching a course at IDF in San Francisco next Thursday (September 15^th)  entitled Using Industry Standards to Get the Most Out of 10 Gigabit Ethernet in Linux* Virtualization and Cloud Environments.

The bulk of this session will detail new usage models for SR-IOV, including solutions for migration of VM’s using SR-IOV.  I will also be discussing Flexible Port Partitioning, which is where we use SR-IOV Virtual Functions in the standard Linux kernel to provide a highly flexible mechanism to carve up your Ethernet connection.

My partner in crime (Brian Johnson) and I gave a version of this presentation at IDF in Beijing earlier in the year, it was very well received.  We have been working very hard to improve it since then – so we are hoping it is even better and even easier to follow.

So if you are going to be at IDF, please join us.

We will also be doing a live demonstration of Flexible Port Partitioning in the Intel Booth - #903 in the Data Center Zone.  I will be in the booth most of the time to answer questions.

I will be manning a booth at the Red Hat Summit next week in Boston.  In this booth, I will be doing a live demonstration of  Intel® SR-IOV in a Linux* OS to effectively and efficiently utilize a 10Gb Ethernet connection for kernel processes such as iSCSI, Backup, NFS, etc., while also providing direct access to the Intel® 10Gb Ethernet device from a Virtual Machine.

I will further be discussing how using an Intel SR-IOV capable Ethernet device gives you instant QoS by the inherent nature of our round-robin scheduling.  Then I’ll be showing you how our controller and drivers allow you to enforce even more QoS by applying transmit rate limiting to individual SR-IOV Virtual Functions.

So come around and find me in Booth #412.  I’ll be the terribly handsome guy talking all about SR-IOV features being used in both a virtualized and non-virtualized environment.  And I do mean terribly handsome.  My esteemed co-worker, Greg, will also be there with me answering questions. Greg is the primary author of the Open Source SR-IOV drivers for the Intel® Ethernet controllers.

Not sure what and SR-IOV is?  Click on the link to see a video explanation.  I also have the PCI-SIG SR-IOV Primer you might want to check out.

I will soon have some whitepapers and additional blogs on this topic.  So stay tuned!

Steve Worley of the System x Performance and Analysis Benchmarking group at IBM just released a paper that we here at the Intel Ethernet Virtualization team are pretty excited about.

This paper, entitled Effect of SR-IOV Support in Red Hat KVM on Network Performance in Virtualized Environments discusses how Steve used the SPECvirt_sc2010 benchmark to test performance when using SR-IOV and when not using SR-IOV.

They ran a variety of workloads; here is a quote from the document detailing the workloads:

The benchmark uses several workloads representing applications that are commonly consolidated into a virtualized environment. Scaling is achieved by running additional sets of virtual machines, called “tiles,” until overall throughput reaches a peak. Each tile consists of six different virtual machines:

• Infraserver – Serves file downloads directory for web workload, runs web backend simulator

• Webserver – Runs Web workload

• Mailserver – Runs Internet Message Access Protocol (IMAP) workload

• Appserver – Runs application server workload

• Dbserver – Runs database as backend to application server workload

• Idleserver – Runs a poll workload

Because each tile represents six VMs, a system running 10 tiles would consist of 60 VMs.

The article itself does not indicate which SR-IOV Ethernet controller is used for the test, however the detailed configuration document that it links to does.  The Intel® 82576 Gigabit Ethernet Controller was used for these tests.

Results

The results of the tests are very interesting.  It shows that SR-IOV definitely reduces CPU overhead and that by doing so, it can increase QoS due to the fact that the CPU utilization is reduced.

The following chart is a graphical representation of the data published within the paper by IBM:

SR-IOV Data Results

Note that there is no data for 13 Tiles in non-SR-IOV mode – this is because the CPU utilization was so high that the test run could not be completed.  The details of each of the tests can be found here.

So What Does This Mean

Good question.  It does not mean that the SR-IOV enabled network path is a replacement for the Emulated path for all workloads.  In fact, if you dig into the details of the test, IBM seems to have picked and chosen specific workloads to use SR-IOV while leaving the others on the Emulated path.

The current ecosystem for SR-IOV today has its limitations – one of the major ones is that live migration is not currently supported (though we are working with the industry on this right now).  For this reason alone, SR-IOV may not be suitable for all virtualized workloads.

However, as these test results show, there are definitely some cases for some customers where the performance benefits gained by using SR-IOV outweigh the current limitation associated with it.

For more information, I suggest you carefully read the document by Steve Worley entitled Effect of SR-IOV Support in Red Hat KVM on Network Performance in Virtualized Environments, and take a close look at the SPECvirt_sc2010 Result information.

BTW… If you take a look at the other test results posted on the SPECvirt_sc2010 site, all of the systems are using Intel® Ethernet Controllers for their primary benchmark network;

• Intel® Ethernet Server Adapter X520 10GbE dual-port with SR-IOV support
  
• HP* NC365T 4-port Server Ethernet Adapter that uses Intel® Ethernet 82580 Controller.
  
• Intel® Gigabit ET Adapter that uses the Intel® Ethernet 82576 Controllers with SR-IOV.
  

http://www.spec.org/virt_sc2010/results/specvirt_sc2010_perf.html

Curious about what SR-IOV is, and how it works?  We have a blog on that!

From time to time I get questions about which Intel® Ethernet controllers support iSCSI Boot for VMware* ESX. In the not-too-distant-past this was a very easy question to answer because, while there are a number of Intel® Ethernet devices that support iSCSI Boot, VMware did not have support for iSCSI Boot until VMware ESX 4.1, making it a simple answer.

However as of ESX 4.1, VMware does support iSCSI Boot on a growing number of Intel® Ethernet controllers and it was becoming more and more difficult for me to simply rattle off the Intel® Ethernet controllers that supported iSCSI Boot.  So I went up to VMware’s HCL page here and did a search for “Intel iSCSI Boot” under the “IO Devices” tab.  The result of which was a big goose-egg – no controllers found.

After some research, I found out that VMware does not refer to this technology the way we at Intel do.  We call it iSCSI Boot, the VMware HCL refers to it as IBFT (iSCSI Boot Firmware Table).

So, a refined search came up with a much better result.  Over 80 Intel® Ethernet devices show up that support iSCSI Boot (IBFT).

So if you want to know what Intel® Ethernet Controllers support iSCSI Boot for VMware ESX, simply click here to see the comprehensive list.