The Server Room Blog

Green Storage

curt.e.bruns@intel.com — Wed, 18 Nov 2009 20:52:02 GMT

It’s not just about energy-sipping systems—it’s also about your storage footprint

Most of us are familiar with the concept of green IT: increasing energy efficiency across the enterprise to trim costs and optimize resources. While you hear a lot about servers helping to reduce energy usage, not as much is said about storage. Intel and the storage industry are working together to provide green storage solutions, too.

For the storage community, every system has to be cost-effective as well as performance-driven, which means energy efficiency is a key consideration. It starts at the processor level, where the Intel® Xeon® processor 5500 series is extending the boundaries of energy efficient performance.

Many storage system providers have picked up on the Intel Xeon processor 5500 series since it was introduced last March. For example, the HP StorageWorks XP10000* Disk Array and 3000 Enterprise* Virtual Array are based on the new processors. Schooner Information Technology appliances leverage quad-core Intel Xeon 5500 processors and half a terabyte of Intel® X25-E flash memory. The bottom line for the Schooner appliances is an 80 percent decrease in power and cooling requirements versus ordinary servers.

But green storage isn’t just about power consumption at the processor or system level. An equally important green strategy is to reduce the overall storage footprint, and a number of technologies are available to help IT organizations implement this strategy.

Virtualization is driving huge data center energy savings by greatly reducing the number of physical machines in the data center. As Bob Fine, director of product marketing at Compellent, pointed out at the 2009 Storage Networking World conference last spring, many large enterprises realize that they’re approaching a cap. “They can only get a certain amount of power in their data centers and see virtualization as a way to reduce their power requirements,” says Fine. “Instead of building new data centers, they can stay in the ones they have, saving millions of dollars in the process.”

Many IT managers tell Intel that storage can be a big gating factor when it comes to scaling virtual environments. The Intel Xeon processor 5500 series uses Intel® HT Technology within each processor core, doubling the number of threads that can be processed at the same time. This option permits more efficient workloads and enables storage servers to virtualize more applications. Intel HT Technology is also more energy efficient than traditional threaded processing.

Compellent and Hitachi Data Systems (HDS), both users of the Intel Xeon processors, recommend reducing the storage footprint in other ways as well. “Limit the amount of content you need to store by using technologies like data deduplication,” advises Asim Zaheer, vice president of product and competitive marketing at HDS. “Also, don’t have wasted capacity or wasted systems—that’s where tiered storage and virtualization come into play.”

Compellent’s Fine sees tiered storage as especially important when using expensive disk resources like solid-state drives (SSD). By limiting SSD to the top tier, a company could save on drive costs and increase storage efficiency. “Only the active data would sit on SSD, and all the inactive data would go onto a tier-three SATA drive,” says Fine. “Since SSD drives are about 10 times the cost of Fibre Channel, it’s very important to gain those kinds of efficiencies.”

Isilon Systems, another user of Intel processors, has a pay-as-you-grow model for its clustered storage products that makes it easier to avoid over-provisioning and wasting power. If a customer needs to add more performance, Isilon can provide nodes with Intel processors and memory, but no storage. If the customer requires capacity only, Isilon sells nodes with just disks. In addition, Isilon uses ColdWatt power supplies, which it says are about 30 percent more efficient than traditional power supplies.

As Intel works with the storage industry to deliver more energy-efficient and high-performance storage solutions, we’d like to know what IT organizations are doing to implement green storage technologies in the data center. If you work in IT and have fresh perspectives to make your organization more efficient, you’re invited to share your ideas here.

The One Million IOPS game

bhaskar.d.gowda@intel.com — Sun, 27 Sep 2009 03:47:12 GMT

Few months back I saw an press release on Reuters from Fusion IO and HP claiming to hit 1 Million IOPS with a combination of Five 320GB ioDrives Duos and Six 160GB IO drives in an HP Proliant DL785 G5 which is a 4 Socket server with each socket having 4 cores, that makes a total of 16 cores in the server. I went saying wow that is amazing, a million IOPS is something any DBA running a high performance Database would like to get hands on. But when I did a quick search on the Internet for on how affordable the solution would be, I was horrified to see the cost which was clsoe enough to buy me couple of Mercedes E class sedan, all though the performance was stellar the cost and 2KB chunk size made me say which application does a 2KB read/write anyways, the default windows allocation is 4KB.

As time went by I got busy with other work till our Nand Storage Group told us that they are coming up with a product concept based on PCIe to show a real 1 Million IOPS with 4KB block sizes which application in real world uses. This triggered the thought on what takes to achieve a 1 Million IOPS using generically available off-the shelf components. I hit my lab desk to figure out what it takes.

Basically getting a Million IOPS depends on Three things:

1. Blazing fast Storage drives.
2. Server hardware with enough PCIe slots and good processors.
3. Host Bus Adapters capable of handling the significant number of IOPS

Setup:

Intel Solid State Drives was my choice, there has been a lot discussed and written about the performance of Intel SSD's and that was easy choice make. I selected Intel X25-M 160GB MLC drives made using 34nm process. These drives are rated for 35K Random 4KB read IOPS and seemed like a perfect fit for my testing.

Then I started searching for the right Dual Socket server, this Intel® Server Systems SR2625URLX with 5 PCIe 2.0 x8 provided enough slots to connect HBA's. The server was configured with Two Intel Xeon W5580 running at 3.2Ghz and 12GB of memory.

Search for the HBA was ended when LSI showed their 9210-8i series (Code named as Falcon) which has been rated to perform 300K IOPS. These are entry level HBA's which can be configured to hook up up to Eight drives to eight Internal ports.

Finally I had to house the SSD's some where in a nice looking container, and a container was necessary to provide power connectivity to the drives. I zeored in on Super Micro 2U SuperChassis 216 SAS/SATA HD BAY, this came with Dual power supply and without any board inside it, but it provided me an option to simply plug in the drives to the panel and not worry about getting them powered. The other interesting thing about this Chassis is that, it comes with Six individual connectors on the back plane so all each connector handles only Four drives, this is very different from active back planes which routes the signal across all the drives connected to them, this allowed me to just connect 4 drives per port on the HBA. I also had to get a 4 slot disk enclosure ( Just some unnamed brand from local shop) in total I had capability to connect 28 drives.

With all the hardware in place, I went ahead and installed Windows 2008 enterprise server edition and Iometer (Open source tool to test IO performance). 2 HBA's were populated fully utilizing all 8 ports on them while other 3 HBA's were just populated with 4 ports only. The drives were left without a partition on them. Iometer was configured with two manager processes with 19 worker threads 11 on one Manager and 8 on the other. The 4KB Random reads were selected with Sector alignment set to 4KB. The IOmeter was set to fetch last update on the result screen.

Result:

Once the test started with 24 drives, and felt I was short of few thousands to reach 1M IOPS so I had to find the 4 bay enclosure to connect another 4 more SSD's taking the total number of SSD's to 28. There was a Million sustained IOPS from the server with an average of 0.88 ms latency and 80-85% of CPU utilization. Please see below pics for more pictorial representation of the setup.

Conclusion:

Recently we demonstrated this setup at Intel Developer Forum 2009 at San Francisco, this grabbed attention of many visitors due to the fact that this is something an IT organization can achieve realistically without spending a lot of initial investment, the good thing about this setup is that the availability of parts and equipments in open market. As Intel we wanted to get this thought started that High Performance storage without robbing a ton of money from your IT department's budget. Once a storage admin gets the idea on what is possible the industry will take more innovative approach to expand and tryout new setups using of the shelf components.

Next Steps:

I would be spending sometime to get this setup running with a RAID config and possibly use a real world application to drive the storage. This needs a lot of CPU resources and I have in mind one upcoming Platfrom from Intel which will let me do this. . I come up with followup experiments.

-Bhaskar Gowda.

SSD - Risky Business?

javed@lodhis.com — Tue, 02 Dec 2008 18:02:01 GMT

Now before we kick off on this topic, first let us decide what risk factor are we talking about - Performance, Stability, Durability, and Price?

After coming across Mario Apicella's (Storage Advisor) article, I am not sure I do agree to all of it since my experience over SSDs is a bit different but you see this varies from situation to situation. I do agree that SSD costs a lot more than the typical mechanical drives at the moment but then when you go for an SSD, price is not your primary concern due to one very simple reason - you are aiming for a better and far more stable solution. You see, Apicella has benchmarked with SATA SSD drives but did not provide any feedback over SAS based SSDs. Now as you may know that SAS drives perform better as compared to IDE, SATA or SATA II since SATA is half duplex while SAS drives are based on Serial Attached SCSI where we all know that SCSI drives perform RW in a duplex mode, outclassing SATA/II in performance. I hope you are getting my point. See! You just cannot say that SSD drives are a risky trade because of a fact that the price difference between SSD and the conventional mechanical drives is significant but my point is that, not everyone would order an SSD. People who ask for an SSD in their solution are actually aiming at a far more stable and LOW-RISK solution and price to them does matter, to the extent where they would compromise stability over price.

Now I wouldn’t say that by plugging in a SSD, you would get phenomenal increase in IO but it does perform quite decently as compared to the other members of storage medium family. Yet again, an SSD is not just of one type and depending on the type of SSD, performance varies.

One interesting point that I see a lot of people reviewing SSD miss out on is, Virtualization. You see, plugging in a 2.5” SATA or SAS based SSD drive in an Intel Blade Server will give you a much better performance as compared to mechanical SATA or SAS but when we talk of virtualized storage especially coming to Intel’s Modular Servers, how well will these drives perform? See, it’s not a discussion we can wrap up in a couple of paragraphs and jump onto conclusions. In continuation of this blog, I will post one of my researches over SSD technology and the performance benchmarks considering the price factor in mind but for now, I would say that SSD is expensive but then the performance gain and stability is better than the mechanical drives which a lot of customers actually aim for. Adding more to it, price of SSD drives is dropping faster than those of mechanical drives; and as such the days of the mechanical hard drive are numbered. We are constantly hearing about the fruition of solid state memory technologies (such as MRAM which has been theorized since the 1970's) that provide more density and reliability, lower power, and faster write times. Mechanical hard drives are now once again bottle-necked by technology since perpendicular heads have become main-stream as have the corresponding aerial density increase that accompanied perpendicular heads.

These are a few simple facts and I think it would not be fair to say that solid state upgrade is a risky business at all. Perhaps the subject should have been, “Solid State Upgrades - Cost or Peace of Mind?"

Javed Lodhi

"Live From" Oracle OpenWorld08--Intel Innovation Zone

william.h.lea@intel.com — Wed, 24 Sep 2008 04:24:49 GMT

"Live From" Oracle Open World and the Intel Innovation Zone...first impression...this is a big event. The Moscone Center here in San Francisco is rocking and Intel has some really interesting and cool demos inside the Innovation Zone. Check out this one where Intel is announcing a new Solid State Drive and demos it at the show:

Check back for more demos and show updates...

NAND solid state disks, the server wave…

anton.roug@intel.com — Mon, 17 Mar 2008 19:40:48 GMT

So what's Intel been doing with NAND based Solid State Disk (SSDs) since my blog on our next generation broadband video streaming demo (http://communities.intel.com/openport/blogs/server/2007/11/14/). Two things: 1) we're close to launching Intel's SATA based SSD products and 2) we've been engaging you to get more details on your usage models and value propositions. In the last few months, there have been a number of announcements for SSDs in server and enterprise storage applications (e.g. EMC: http://www.emc.com/about/news/press/us/2008/011408-1.htm) including a number of small startups offering solutions targeted for server deployments. Based on my discussions with you and looking at what's going on in the industry, here's my view of the value of SSDs in servers and how that maps to server usage models.

As a person who focuses typically on the end-users, SSDs are interesting because they weren't designed to specifically solve an end-user server problem. As I said in my previous blog "because we could", SSD largely exist "because they can". They are what Clayton Christensen would call a disruptive technology. As SSDs are considered for server based applications, I look at how SSDs as a technology can provide greater value when replacing server hard drives (HDDs) or server memory and then build possible usage models from there.

When comparing to HDD usage in servers, I start with the following:

Performance: SSDs can have much better random access performance as measured by higher IOPS, higher throughput and lower read/write latency. SSDs are typically achieving a least 10 times the number of IOPs as HDDs, at least 2-3 times better random access read rate and on the order of 10 times less read and write latency than HDDs. For random access performance, most SSDs blow the highest performing 15K RPM hard drives away.

Power: SSDs use lower power especially when compared to a disk is that is active (i.e. spinning). Given that for most server based applications, the hard disk is always active, this is especially significant. My general observation is that SSDs typically use less than 1/5th of the power of an active HDD. Here they look to be a key technology for making data centers more power efficient.

Cost: When comparing cost per Gigabyte, SSDs are higher priced. Given this, SSDs today are largely being considered for applications where storage IO is the bottleneck - where many hard drives can be replaced with just a few SSDs.

SSDs can be compared to DDR memory with the same three value vectors:

Performance: Unlike the SSD to HDD comparison, memory has higher throughput and lower latency than an SSD. When comparing SSDs to memory for server usages, the primary consideration looks to be latency. SSD reads and writes are on the order of 100s of microseconds. On the other hand, memory based reads and writes are typically less than 100 nanoseconds. Even so, for some applications (e.g. video on demand streaming) 100s of microseconds of latency looks to be acceptable.

Power: Like HDDs, when comparing active power usage, SSDs draw much less power than DDR memory as measure by watts per gigabyte. How much is dependent on how the application uses memory. But generally, SSDs looked to consume 1/10th of the power.

Cost: Unlike HDDs, when comparing cost per gigabyte, SSDs are significantly lower priced than DDR memory. Generally, I start with NAND based SSDs as being half the price of DDR based memory. Depending on the size of the SSD and the technology (whether Single Level Cell (SLC) or Multi Level Cell (MLC)) the difference can be much more.

One final vector to look at is the reliability of SSDs when compared to hard disk drives and memory. Going with just the MTBF numbers being published, SSDs look to be better than HDDs and just as reliable as memory. One area that generates confusion is how the write cycle limitations of NAND technology affect the life-time (as measured by MTBF) of SSDs for server applications. Getting into details on this is a good subject for a future blog. But based on discussions with you, I haven't encountered a server application where the write cycle limitation is the deciding factor in a deployment for SLC SSDs (at least for how we expect Intel's SSDs to perform). For many server applications, it's not the deciding factor for MLC SSDs either.

Using these value vectors, here are my generalizations for the SSD value for enterprise and portal applications:

Use SSDs for the server boot device. When compared to HDDs, SSDs enable faster boot (typically 30%), consume lower power, and are more reliable.
Use SSDs for high throughput, high IOP, low latency application storage. If storage IO is the application bottleneck, replacing with SSDs shifts the bottleneck back to CPU utilization. Example applications include video streaming, search query, and OLTP.
Use SSDs for building a high performance storage tier. Many applications have hot and cold (or long tail) data. By creating a storage tier, the solution cost of a deployment can be reduced significantly. Example applications include using SSDs for improving performance in a NAS or SAN (e.g. what EMC calls Tier 0) or to creating a high performance direct attached storage (DAS) solution (e.g an SSD optimized server).
Consider SSDs as a lower cost alternative to placing application data in memory. Many applications create memory based databases to achieve low latency access times. These applications create custom data structures, use RamDisks or rely on caching through the OS (e.g. SWAP). For many IO bound applications, memory is typically being used as a buffer for disk data. The lower latency and higher throughput of SSDs promise to require less memory for buffering while maintaining the quality of service objectives of the application.

Bottom line for servers today, SSDs look to be cost effective for applications where storage IO throughput and low latency are key. They move the application bottleneck from IO to back to CPU utilization. Get back to me on whether you agree and what additional usage models you're finding.

Creating a broadband video streaming server with 10GE and solid state drives (SSDs)…because we could!

anton.roug@intel.com — Wed, 14 Nov 2007 21:05:00 GMT

I'm excited about our server room blogs as a way for us to get feedback from you quickly. Would love to get your comments on a technology concept demo we did over the last 6 months.

I have been looking at the Internet video phenomenon over the last year. One interesting usage model is making most of what we see on TV today into video on-demand (wiki has a good description http://en.wikipedia.org/wiki/Video_on_demand) either as over the web (e.g. Google YouTube) or provided by a service provider via IPTV (e.g. AT&T Uverse). As Intel, we of course want to understand how we can optimize the on-demand video workload on Intel server technology.

On-demand video deployments today are engineered largely around three resources:

Server: typically a 2 socket, dual core processors per socket, 8G DRAM, rack mount server (workload is writing new videos to disk, reading requested videos from disk, formatting the video packets, transmitting video to client)

WAN: using GE ports, some configurations pushing to exceed 10GE

Storage: as a JBOD, in the past SCSI, moving to SAS and SATA Hard Disk Drives (HDDs)

Understanding this, we challenged ourselves to create a next generation configuration using our leading technology.

Here's what we ended up with:

Server: Fit into a 2U form factor with an integrated JBOD (http://www.intel.com/design/servers/storage/ssr212mc2/)

WAN: Replace GE with the Intel Dual 10GE NIC, target to achieve 20Gbps throughput (http://www.intel.com/network/connectivity/products/10GbE_XF_SR_server_adapter.htm)

Storage: Replace HDDs in the JBOD with prototypes of the Intel enterprise solid state disk drives (SSD)

We worked with Kasenna (http://www.kasenna.com/) to pull the technology together into a prototype demonstration. Actually, they did most of the work as experts in doing high throughput on demand video streaming. Kasenna in the test achieved about 16Gb/s streaming throughput. In IPTV terms, approximately 4000 simultaneous standard definition (3.75Mb/s MPEG2) streams. The demonstration largely focused on the HDD versus SSD engineering.

If you're not familiar with SSD technology, wiki has a good overview (http://en.wikipedia.org/wiki/Solid_state_drive). Intel also discussed our NAND based solid state drive technology at fall IDF (http://www.intel.com/pressroom/kits/events/idffall_2007/webcasts.htm). Pat Gelsinger introduces the technology about 40 minutes into his Tick-Tock - Powerful, Efficient, and Predictable presentation. Knut Grimsrud gives a good overview of the NAND technology in his Challenges and Opportunities for Non-Volatile Memory in Platforms presentation.

I won't post the gory details of the configurations today. If you're interested, send me email. Simple net, it took approximately 60 15K RPM HDDs to achieve the same throughput as 12 Intel prototype SSDs. Two major takeaways:

1. Intel solid state drives look to be ideal for high throughput workloads like on-demand video that require random access from disk. Kasenna achieved about 5 times the throughput on each solid state disk drive over the hard disk drives.

2. In this case, the Intel SSD configuration lowered the peak power for the configurations (disks, server, NICs, memory) to about 1/3 of the HDD configuration.

The demo also raised a number of other thoughts on whether the higher performance of the SSDs could reduce the amount of memory required in the server. No conclusions on this yet.

This was my first step in understanding the advantage of solid state drive technology for a server application. My conclusion, Intel NAND based solid state drive technology looks to be a promising technology for achieving higher throughput and lower power when compared to a hard disk. I'll be posting more examples in the future on where SSDs looked to be a good fit for applications. I would be interested in hearing your feedback on this concept demo and about server applications where you see SSDs as having high value.