Skip navigation
dougb

Port Density: Dual vs Quad

Posted by dougb Dec 18, 2009

      Ethernet ports are force multipliers. With a single computer you can do neat things, but with a network, one plus one is more than two.  When you get to hundreds of computers you get amazing things, and when you have millions of computers you get magic.  The magic wouldn’t happen if every system had just one port, the number of clients would require an equal number servers.  But we don't have that ratio.  So there are two ways to get the number of clients to balance the smaller number of servers.  One, big pipes. Two, tons of ports. Today the topic is making intelligent purchasing decisions for dual or quad port solutions to maximize your servers.

     On the surface it’s a pretty easy decision.  If you need more ports, buy a card with more ports!  But that isn't always the right answer.  With the current generation of quad port adapters, a switch is required to get the port density right.  This sounds innocuous enough, but there is a hidden cost.  Even something that is as fast as a PCI Express switch will still add latency.  The more devices between the end silicon and memory complex will slow things down.  At its root, performance is all about speed.  You may have enough bandwidth to send the packets, but you can run out of time to send them all.  With small packets, 1.44 million packets a second at 1 Gigabit means you only have a few fleeting moments to get the packet from host memory out onto the wire.  Any delay may cause packets to be missed.  This is where using more dual port cards can help out.  By eliminating one intermediate device, that latency can be saved.  Make sure you check your motherboard's documentation since there may be "hidden" switches on the motherboard that are increasing the slot density.  Putting a quad into a switched port on a motherboard, especially one off of the IOH or ICH will really be worse case for latency.  If you are latency sensitive, try to put duals into those switched slots.

     Another thing to factor is the redundancy factor.  With multiple cards you have a buffer to single points of failure.  On a quad port adapter, if the switch goes out all the other ports on the devices go away.  Intel uses highly reliable switch suppliers and the Mean Time Between Failures (MTBF) is roughly the same on dual and quad port adapter, but if your business criteria require more distributed ports for up time, dual is the way.    Actually, the dual ports have better MTBF due to having less parts but both the dual and quad ports have respectable MTBF numbers.

     The last issue is electro-magnetic interference.  EMI is part of using electronics.  All those electrons zipping around cause a small field that can potentially cause negative side effects in some devices.  When using a quad port card, the density of components is higher, which can change the EMI profile of a machine when compared to just using a dual port card.  Just like with the MTBF, the difference between a dual and a quad can be very small, but if it’s a critical focus of your design, you need to go in eyes open. All our cards are certified to the same Class A or Class B standards but in a particular user setup, there may be incremental value in using 2 cards that may broaden any transmission spectrum and reduce the overall impact to a particular frequency. This setup can also reduce the combined cable effects since 4 ports in a single slot leaves the cables near one another and the multi slot implementation will have cable separation by default.

     One thing you don't have to factor in is software support.  Our drivers support both dual and quad port adapters the same.  The driver for a dual port adapter using the Intel® 82571 Gigabit Controller is the same as for a quad port adapter using the 82571.  So you can qualify one driver and know it will work on either the dual or quad version of the same adapter family.  This can save qualification time, which is money.  This will also allow you qualify and build out with duals today and add-in quad ports for higher densities without having to qualify a new driver.

     Now don't think that I'm talking bad about our quad port adapters.  This article is just providing you a full discussion of the dual versus quad decision.  Quad ports provide great feature sets with port density that maximizes your system investment.  When using 1U machines, for example, the only way to get the density needed for today’s datacenters is via a quad port adapter.  Quads also don't take up as much space internally as two boards, so it can change the air flow characterizations.  Quad ports are big sellers, and it’s easy to see why!

Let’s wrap it up:

1)      Dual vs. Quad isn’t just about port density

2)      Intel makes both dual and quad adapters

3)      Thanks for your interest in Intel® Ethernet products

     Receive Side Scaling(RSS) is a technology that uses a very complex hash function to force a routing of packets to separate cores.  Instead of just making one CPU do all the work, it makes all the cores able to help with the work.  It requires MSI-X in order to route the interrupts to the core doing the work thus maximizing the work offload, but can work with just MSI.  Multiple queues also help make RSS work better, but can still provide some help when used with just one queue.

     Windows 2008* supports RSS, but defaults to just a handful of queues.  With the 10 Gigabit PCI Express product line we have added a ton of queues, and this handful just isn't enough especially when you consider the number of cores that can ship in even a very modest system.  I know you probably think I get paid by the word, but pictures work best here.

 

Here is what just having the driver installed would look like.  I used a couple of systems for the pictures, so don't be worried if your machine doesn't exactly match my sample pics.

RSSNormal.jpg

     As you can see, the O/S limits you out at 4 queues in this dialogue box.  After the picture I selected 4 queues for my test.  But your Intel® 82599 10 Gigabit Ethernet Controller supports more than that!  So does your machine!  How do you get that horsepower down to the road?

 

 

     Here is what the four cores being loaded looks like.  These use Intel® Hyper-threading Technology, which is really an extra half core.  For our workloads, we can't use the Hyper-threaded core, just the base core, so you'll see an every other CPU on the output.

RSSCPUonly4.jpg

     Here is how to get all the cores you paid for.  Install the Intel® PROSet GUI that comes with the driver media.  If you run the autorun, it will offer to install it.  Our webpacks have it there as well.  Notice things have changed quiet a bit.

snap.jpg

     After you set the number of queues, hit the okay button and follow the heed of the reboot message.  The GUI changes some of the O/S parameters too, so if you don't reboot, you won't see the goodness!

 

 

     Here is what it looks like after the reboot:

RSSAfterReboot.jpg

     Very nicely spread out.  (Again remember that Hyper-threading can't do our network loads so they end up low).

 

RSS is a great technology that allows you to get the maximum spreading of CPU loading on your Windows boxes.  With great power comes great responsibility, so I’ve heard, and RSS is no exception.  Increasing the number of RSS queues above the default value can actually degrade performance if the application in use isn't tuned for it (i.e. CPU lock contention).  Some performance testing software for example is horrible at 16 queues. Microsoft spent a lot of time tuning the HTTP Web Server for high numbers of RSS queues so we definitely recommend this for HTTP web servers, but everyone else's mileage may vary.  Use with caution!  I'll do another posting soon about the ins and out of RSS but that's a whole 'nother post.

 

 

Big review!

1)  To get more than the 4 queue RSS spread, you will need to install PROSet and configure the queues using it

2)  Remember to reboot or you won't see the goodness

3)  Thanks for using Intel® Ethernet products.

     I'm pleased to announce the public posting of the drivers for a bunch of key operating systems for the Embedded market segment (i.e. Intel® 82574 and 82567 Gigabit Ethernet connections).  We are hosting the files up on the Embedded Design Center, a key portal for development with Intel® processors and products in the Embedded market.  We have been providing the files and installer manually to customers in need, but we finally got the ship release authorization to make them all public.  A multiple year program it finally provides key operating system support for several Microsoft* Embedded operating systems.

 

     In the download is a file called DriverSelectionGuide.txt which outlines the devices ID supported by which driver.  The other file is the actual installer for the webpack.  After you accept the license, it will put the files into a folder (normally called C:\IntelEmbedded7.0) with all the sub folders holding the goods.  After that its broken up by family type, PRO100, PRO1000 and PROXGB.  Now I bet you saying what is 10 Gigabit doing in Embedded?  Microsoft Windows XP* (the normal version) is a regular request and while we can't support it on our normal product releases, it is placed here for the ease of install for our customers. If you are interested in XPE* for 10G, drop me a line via the private e-mail function in my profile, and we can talk about options.  The plumbing in XP can't handle the burden of 10G, but for some that is an acceptable limitation and we don't like saying no to our customers.  Please read the DriverSelectionGuide for some important 10 Gigabit information.

    

     We also have some older Embedded software, also known as CE5* in the webpack.  CE5 is not available for all parts, notably for the more modern 1 Gigabit parts.  If you require CE5 for the newer parts, please contact your Intel field sales representative so we can build a business case for possible future development.  You probably are also wondering why there are Server2003* and Server2008* directories.  Those are not Embedded!  Au contaire!  It is often joked that Embedded is just Latin for "Everything else" (it's really panton alius) We often have customers that want to use desktop boards in server type functionality.  Most in the network attached storage world, they need the extra connections that a server operating system can support.  As a time to market tradeoff, not all our desktop parts have server operating system support.  This closes that gap.  These files and the XP ones for 10 Gigabit do not come with the digital signing files, called CAT files, and will install with warnings.  For an Embedded design that is okay since an end-user will never see the warning.  We have no plans for getting the certifications for the extra files.

 

     In the coming clean department, there is one oversight in the DriverSelectionGuide.txt file.  We have support for the 1501 device ID for CE6 and XPE in the drivers and install packages, but missed getting it into the selection guide.  So if you need the 1501 device ID for CE6 or XPE, rest assured it's in there.  We are working on fixing the DriverSelectionGuide, but had to go to press with what we had on hand.

Here is the driver link in text for copy paste purposes:  http://edc.intel.com/Download.aspx?id=2795&returnurl=/Software/Downloads/default.aspx

 

Thanks to the Embedded Design Center for allowing us to host the files on their site!

 

Time for the usual review:

1)  CE6 and XPE files are now publicly available

2)  1501 device is supported.

3)  Thanks for using Intel(R) Ethernet products!

Filter Blog