Few months back I saw an press release on Reuters from Fusion IO and HP claiming to hit 1 Million IOPS with a combination of Five 320GB ioDrives Duos and Six 160GB IO drives in an HP Proliant DL785 G5 which is a 4 Socket server with each socket having 4 cores, that makes a total of 16 cores in the server. I went saying wow that is amazing, a million IOPS is something any DBA running a high performance Database would like to get hands on. But when I did a quick search on the Internet for on how affordable the solution would be, I was horrified to see the cost which was clsoe enough to buy me couple of Mercedes E class sedan, all though the performance was stellar the cost and 2KB chunk size made me say which application does a 2KB read/write anyways, the default windows allocation is 4KB.
As time went by I got busy with other work till our Nand Storage Group told us that they are coming up with a product concept based on PCIe to show a real 1 Million IOPS with 4KB block sizes which application in real world uses. This triggered the thought on what takes to achieve a 1 Million IOPS using generically available off-the shelf components. I hit my lab desk to figure out what it takes.
Basically getting a Million IOPS depends on Three things:
1. Blazing fast Storage drives.
2. Server hardware with enough PCIe slots and good processors.
3. Host Bus Adapters capable of handling the significant number of IOPS
Setup:
Intel Solid State Drives was my choice, there has been a lot discussed and written about the performance of Intel SSD's and that was easy choice make. I selected Intel X25-M 160GB MLC drives made using 34nm process. These drives are rated for 35K Random 4KB read IOPS and seemed like a perfect fit for my testing.
Then I started searching for the right Dual Socket server, this Intel® Server Systems SR2625URLX with 5 PCIe 2.0 x8 provided enough slots to connect HBA's. The server was configured with Two Intel Xeon W5580 running at 3.2Ghz and 12GB of memory.
Search for the HBA was ended when LSI showed their 9210-8i series (Code named as Falcon) which has been rated to perform 300K IOPS. These are entry level HBA's which can be configured to hook up up to Eight drives to eight Internal ports.
Finally I had to house the SSD's some where in a nice looking container, and a container was necessary to provide power connectivity to the drives. I zeored in on Super Micro 2U SuperChassis 216 SAS/SATA HD BAY, this came with Dual power supply and without any board inside it, but it provided me an option to simply plug in the drives to the panel and not worry about getting them powered. The other interesting thing about this Chassis is that, it comes with Six individual connectors on the back plane so all each connector handles only Four drives, this is very different from active back planes which routes the signal across all the drives connected to them, this allowed me to just connect 4 drives per port on the HBA. I also had to get a 4 slot disk enclosure ( Just some unnamed brand from local shop) in total I had capability to connect 28 drives.
With all the hardware in place, I went ahead and installed Windows 2008 enterprise server edition and Iometer (Open source tool to test IO performance). 2 HBA's were populated fully utilizing all 8 ports on them while other 3 HBA's were just populated with 4 ports only. The drives were left without a partition on them. Iometer was configured with two manager processes with 19 worker threads 11 on one Manager and 8 on the other. The 4KB Random reads were selected with Sector alignment set to 4KB. The IOmeter was set to fetch last update on the result screen.
Result:
Once the test started with 24 drives, and felt I was short of few thousands to reach 1M IOPS so I had to find the 4 bay enclosure to connect another 4 more SSD's taking the total number of SSD's to 28. There was a Million sustained IOPS from the server with an average of 0.88 ms latency and 80-85% of CPU utilization. Please see below pics for more pictorial representation of the setup.
Conclusion:
Recently we demonstrated this setup at Intel Developer Forum 2009 at San Francisco, this grabbed attention of many visitors due to the fact that this is something an IT organization can achieve realistically without spending a lot of initial investment, the good thing about this setup is that the availability of parts and equipments in open market. As Intel we wanted to get this thought started that High Performance storage without robbing a ton of money from your IT department's budget. Once a storage admin gets the idea on what is possible the industry will take more innovative approach to expand and tryout new setups using of the shelf components.
Next Steps:
I would be spending sometime to get this setup running with a RAID config and possibly use a real world application to drive the storage. This needs a lot of CPU resources and I have in mind one upcoming Platfrom from Intel which will let me do this. 
. I come up with followup experiments.
-Bhaskar Gowda.
The Most Energy Efficient Investment That Government Won’t Help You Make
Posted by Bryce Olson Sep 21, 2009
Fall IDF: Is Italian Pasta the Actual Inspiration for Server Virtualization?
Posted by Iddo Kadim Sep 17, 2009I love food! Since I was a kid I’ve loved noodles, especially Italian pasta. I used to think that Spaghetti was the general name for Italian noodles. Learning how to twist Spaghetti on a fork gave a great sense of achievement and joy.
Many years later my wife and I travelled to Rome. Naturally – both of us really love food, we spent a lot of time seeking out restaurants and checking out new food. One of the wonderful dishes we had was Pappardelle (on the left) with Duck Ragu. It was my 1st encounter with Pappardelle –a very wide form of pasta. You get only a few Pappardelle on your plate but it’s still the same amount of pasta. I found it not as practical to twist the Pappardelle around my fork, so I cut them up into smaller pieces to eat them.
I’ve thought about it recently when I looked at the back of a virtualized server. Looks similar, no? J
A typical virtualized server has 8-10 1Gigabit Ethernet (1GbE) ports, and 2 Fibre Channel ports. This makes for a lot of cabling, and many add-in cards. It translates to a lot of cost, power, and complexity (and thus reliability risk) for an IT shop. As a result, there’s a lot of buzz around high-speed networks, specifically 10GbE. That technology presents the opportunity to consolidate all these 1GbE ports to a significantly smaller number of higher bandwidth, i.e. 10GbE ports. It makes for a much tidier server.
Kind of like substituting Pappardelle for SpaghettiJ
In that case, iSCSI or FCoE (Fibre Channel over Ethernet) could be used for the SAN connection, still using the same high-speed ports. Standards like Data Center Bridging (DCB) could add a lossless character to the 10GbE link to make it friendlier to FCoE.
Few new solutions though come without new challenges. The common way for VMs to share I/O devices in a today’s environment in through mediation of the hypervisor, using emulation or para-virtualization. That reduces the effective I/O bandwidth. It also becomes a fairly significant overhead to the server in its own right, reducing the available server capacity for application processing, and it adds latency. With the growing trend in IT to treat virtualization as a default deployment mode for any application, these issues become quite limiting.
We at Intel have thought that the best way to overcome these issues is by using “direct assignment”. Using the Intel® VT-d technology (launched in the Xeon 5500 platform), a VM can be assigned a dedicated I/O device. This nearly eliminates the overheard related to the hypervisor mediation I mentioned above. A side benefit is that it increases the VM to VM isolation and security. But assigning an individual I/O device to one VM is not very scalable…
This is where the PCI-SIG’s SR-IOV (Single Root I/O Virtualization) standard comes into play. This standard allows a single I/O device to present itself as multiple virtual devices. With SR-IOV, each virtual device can be assigned to a VM, adding scalability to the direct assignment model, effectively allowing the physical I/O to be shared yet with greater security and reliability.
Another challenge with the direct assignment model is related to live migration. Hypervisors have typically assumed the SW mediated IOV model. As a result, hypervisors need to be modified to adapt their live migration solutions to direct assignment.
These technologies span many different components of the server platform. Intel® VT-d is necessary, so Xeon 5500 must be used (or later platforms). SR-IOV capable I/O devices – NICs or Storage controllers, are required. BIOS must be modified, as well as hypervisor software. This is pretty heavy lifting.
So you can only imagine how excited I am to be able to showcase 4 different SR-IOV demos at IDF next week! The demos involve 2 server vendors, 3 VMM vendors – 3 different vendors implementing 3 different hypervisor architectures, and 3 different IHVs representing 2 different I/O technologies. We show the performance improvements, as well as VM live-migration. It works!
Come and see it (Booths 517, 707, 709, and 711 in the IDF showcase)!
Are you hearing this clamor? Nope, this is not London Calling! But your employees calling for more performance, your customers calling for faster response time, your boss for more savings.
Have you been waiting to upgrade until your existing servers clash, I mean, crash? This economy has led to a lot of indecision, but when it comes to upgrading your servers, the benefits are pretty big not matter the size of your company.
Good news, the new Intel® Xeon® processor 5500 series-based servers will deliver just that and more.
And for more info, check out this new brochure:
• Save money. By spending money now, you can save in the long run. The latest Intel Xeon processor-based servers deliver more performance than previous generations. Small businesses can consolidate three older servers to one new server and still have room to grow (1). And make sure to take advantage of government and manufacturer server incentives. All of that adds up to a return on your refresh investment in about a year. This tool can help your calculate your ROI: www.intel.com/go/xeonestimator
• Be more competitive. You want to be ready when things rebound and rely on competitive IT equipment. The additional performance and improved reliability offered by updated servers means a more productive staff and faster response times for your customers.
• Avoid hidden costs. The other thing to consider with older servers is the expenses that you don’t expect, like maintenance and downtime. You know - one day is fine, next day is black. To get your boss off your back and your business running smoothly, newer equipment now is a great idea.
So, if fast ROI, savings, increased performance, improved productivity, new warranty sound like music to your ears, talk to your IT solutions provider (http://premierlocator.intel.com) about going with an Intel Xeon processor-based server.
Almost as good as the lyrics from The Clash 
[1] Source: Intel Xeon Server Refresh Savings Estimator, Jul 09
Submarines, Stealth Fighters and Evolving Needs of Information Security
Posted by Chris P_Intel Sep 9, 2009
Networking Equipment Webinar: Intel and Sun discuss Xeon 5500 blade servers
Posted by gwagnon Sep 2, 2009
Intel Intelligent Power Node Manager is a new technology that is available with the Xeon 5500 Series Platforms released earlier this year. Many of you have asked me questions via Twitter (@Toadster) about "How can I use Node Manager?" - so I wanted to present some simple use cases to simplify the explanation of Node Manager and how you can best use the technology in your own enterprise.
First of all, let's explain the growth problem at hand. As servers shrink in size, the density of each server 'footprint' is growing from a power perspective... a few years ago, a single 42U rack could hold about 21 servers (estimating 2U servers) - and usually hosting one or two apps/servers per physical server, depending on if you had single or dual-socket servers. In modern datacenters, that same 42U rack can hold 42 servers (1U each) with 2P per server - so you have an immediate density increase of 2X the # of servers, and 2-4X the number of sockets - which can equate to 16X the number processor threads per rack... one good thing is that Intel has been developing newer technologies to keep the TDP of each CPU roughly the same over the same time period between processor updates... where you used to have 2 or 4 cores, you now have 8 to 16 cores at the same thermal envelope!
Knowing how much power your platform uses is a key factor in populating racks and rows in your datacenter. Prior to Node Manager technology, most Datacenter Managers would base rack population on 'nameplate' power - or the (W) rating on your power supply. That's the 'max' power utilized by the platform, and what the PSU is rated for (worst case). See the image below...
As you can see - using Intel Intelligent Power Node Manager technology, you can view your system's power utilization in real-time using Intel Datacenter Manager and the administrator can implement the power caps to ensure your server rack stays within your required power limits. By utilizing the 'actual' power limits instead of nameplate power, you can increase your rack density thereby increasing your ROI, and decrease your TCO! Lets face it - everyone loves saving money!
Many of us are familiar with this next scenario... it's summertime, and the power company is announcing that the power grid is under strain. Personal homes start having their A/C cut-off to save the power grid from brown-outs... now your enterprise can help reduce those risks as well!
Over the next few weeks, I hope to post more blogs/videos:
1. Single Node Power Monitoring & Management
2. Group/Rack Power Monitoring & Management
3. Thermal Monitoring & Management
Please provide some feedback, and post your questions and ideas for upcoming blogs!
Intel has just launched the Intel® Ethernet Server Adapter X520 family. These NICs are Intel’s first 10 Gigabit adapter products that support “pluggable” optics. This additional configuration option gives IT users a great deal more flexibility in how they deploy 10 Gigabit in their servers and datacenters.
The X520 Family of adapters support bailed optics that allow the removal or addition of different kinds of optics or support with no optics at all. For previous 10 Gigabit products, if you wanted 10 Gigabit SR Fiber connectivity, you had to purchase a 10 Gigabit SR adapter. But with the pluggable X520 adapter family, you can support SR, LR, or simply an SFP+ direct attach cable via the same card by simply removing / exchanging the optics.
With the X520 you can still buy an SR or LR fiber configured adapter, but you can also switch back and forth after purchase by ordering only the new optics that you want to support (not a whole new adapter). In the case of the Direct Attach adapter that supports an SFP+ cage, but comes without optics inserted, you can still use Twin-Ax copper cables to run in the rack less than 7m length runs of 10 Gigabit, but you can also upgrade the Direct Attach adapter later with SR or LR optics as the needs for the particular adapter may change. You can also mix and match optics modules in a dual-port adapter, meaning you could have an LR module in one port and an SR module in the other. You could also throw a Twinax cable into the mix.
The Intel® Ethernet optics modules for the X520 family of adapters also support both 1 Gigabit and 10 Gigabit speeds to help with backward compatibility – an industry first.
Finally, while this new pluggable capability of Intel 10 Gigabit adapters adds a bit more usage flexibility from an IT perspective, the performance capabilities and advanced features for the datacenter I’ve discussed over the past 18 months are also supported. The X520 is based on the Intel® 82599 10 Gigabit Ethernet Controller, so the end result is a flexible product that can help unleash server IO performance whether FCoE, iSCSI, Virtualization, Security, or just raw IO performance. Regardless of your 10 Gigabit needs, the X520 probably has what your Server environment needs.
--
Ben Hacker