The Data Stack

Go Ahead and Virtualize IBM InfoSphere* Information Server

tim.allen@intel.com — Thu, 23 May 2013 15:42:13 GMT

If you want to virtualize IBM InfoSphere* but have put it off, I have some new information that might give you the bump you need to move forward: How does 98 percent throughput sound?

In a nutshell: extensive testing shows that InfoSphere can perform very well when virtualized. How well? We achieved between 90 and 98 percent of the throughput that we typically see in a physical environment, with an overhead tax of only about 10 percent on I/O-intensive workloads.

You can get all of the configuration and results details in the white paper about InfoSphere virtualization here, but I’ll hit the highlights for you in this post.

While virtualizing InfoSphere Information Server is cool, here’s the icing on the cake: virtualization with VMware vSphere* is now supported as an IBM PureApplication* pattern. “Patterns” are a growing ecosystem of software stacks for IBM PureSystems* running on Intel® Xeon® processors. Hence, adding the leading VMware vSphere 5.1 stack indeed broadens the use of these rich appliance systems versus other industry systems. PureSystems are innovative, quick-to-uptime, and run industry-standard software stacks.

Testing InfoSphere Virtualization

IBM, VMware, and Intel teamed up recently to see how IBM InfoSphere performs when it’s virtualized. Specifically, we looked at the runtime performance of:

IBM InfoSphere DataStage* 8.7
Running on VMware vSphere 5.0
On a server powered by the Intel Xeon processor E7 processor family

The tests found that InfoSphere DataStage scaled smoothly as we cranked up the number of virtual CPUs (vCPUs), while clocking throughput at up to 98 percent of that found in a physical environment.

More good news: we saw only a slight performance difference when using VMFS (Virtual Machine File System) versus RDM (Raw Device Mapping) data stores. This means that you can reap the benefits of VMFS for storage provisioning with virtualized workloads without concern over performance.

Now let’s get down into some of the details so you can put these results in context.

InfoSphere Virtualization Test Configuration

Server

We used an IBM* System x3850 X5 with a network-attached IBM System Storage* DS5300:

Four socket system
40 physical cores
Intel Xeon E7-8870 processors
Configured as “optimized for performance”

To get a good comparison between physical and virtual environments, we controlled RAM and processor availability to the native environment so that we could match the virtual environment as closely as possible. We enabled all of the virtualization-related options, except for Intel® Hyper-Threading Technology (Intel® HT Technology), which we disabled to simplify the comparison.

IBM InfoSphere

We installed all InfoSphere DataStage components on one physical server in the native environment and on one virtual machine in the virtualized environment:

IBM WebSphere* 8.1 Application Server (WAS)
XMeta repository
DataStage engine

We found that DataStage engine running in a single virtual machine had higher throughput. The specific reasons for this result weren’t clear, but we’re hopeful that further testing will provide more detail.

ETL Workload Virtualization Test Results

As expected, throughput in both environments increased as the number of processor cores increased, with performance in the two environments varying only between 2 and 10 percent. The overhead for the virtual environment was only 10 percent for all tested configurations.

Host Server Memory-Management Test Results

We also wanted to see what would happen with DataStage performance when overcommitting the host server. We measured a 34 percent drop in throughput when the system was 100 percent committed, and throughput continued to drop as the host processors were further committed.

Working with VMware engineers, we determined that this drop was because the eight vCPUs were not mapping neatly onto the 10-core, Non-Uniform Memory Access (NUMA)–node design of the Intel Xeon processor E7 family’s microarchitecture.

To work around this issue, you could use a five vCPU configuration instead of an eight vCPU configuration. Five vCPUs would map well to the 10-core NUMA nodes. Another workaround would be to turn off NUMA scheduling in BIOS or VMware vSphere, which would allow all of the CPU cores to be used, though you would see a lag in memory performance. The lesson? Understand NUMA configuration to optimize VM performance.

We repeated this over-commitment test with the VMware vSphere “reserve CPU” option for the DataStage guest set to maximum, and the result showed minimal performance impact. However, this move can potentially impact the performance of the other non-reserved virtual machines running on the same host.

If your system is overcommitted and you’re not seeing the DataStage runtime performance you want, the best option would be to increase the host system capacity or to move the VM to another host.

Go Ahead and Virtualize InfoSphere

Our tests showed InfoSphere Information Server runtime performed very well in a virtualized environment hosted on a platform powered by the Intel Xeon processor E7 family and using VMware vSphere 5.0. So if you’ve wanted to virtualize your InfoSphere applications but were concerned about performance, now might be the time.

Take a look at the white paper to see the test configuration, procedure, and results. And follow me on Twitter, @TimIntel, to get more useful InfoSphere and DB2* tidbits. You can also get the latest news and technology updates at the joint Intel and IBM DB2 website.

Server Storage Caching Considerations

webadmin@intel.com — Wed, 22 May 2013 14:00:16 GMT

Caching is storage. Here’s how Wikipedia defines caching:

“… .a cache (pron.: /ˈ k æ ʃ / KASH) is a component that transparently stores data so that future requests for that data can be served faster.”

With the data explosion and consumers creating content and wanting to access it immediately, caching is becoming more and more important.

My colleague, Susan Bobholz is a Marketing Director in Intel’s Data Center Software Division and talks about the considerations for server storage caching.

Right now, one of the hottest storage topics is storage caching. It seems hardly a week goes by without some type of caching software showing up in the press. I thought I’d spend a bit of time talking about this trend and provide some things to think about when choosing caching software.

In many datacenters, the hottest, most frequently accessed data is stored on 15K serial attached SCSI (SAS) hard drives. But those hard drives can become a bottleneck because they are mechanical devices with moving parts. They simply can’t move fastest enough to keep up with some application demands. One solution to resolve this is to replace all those 15K SAS hard drives with Solid State Drives (SSDs) but this can be an expensive undertaking. The beauty of storage caching is that it protects your investment in hard drives, because your application performance is improved without replacing all those hard drives with SSDs.

To be as simple as possible, storage caching allows the hottest, most important data to be stored in a SSD instead of hard drives, allowing that data to be accessed significantly faster. Often caching is implemented as a server application, but sometimes it’s actually part of the firmware on a RAID HBA

So you’ve decided you want to implement storage caching. There are several caching options out there. Other than cost, what are some key questions to consider about when deciding which to use?

Where does your cache physically reside?

As mentioned above, some caching solutions are integrated into a RAID HBA. This means that the Cache SSD must be attached to the RAID HBA itself and only data on hard drives connected to that RAID HBA can be accelerated.

Other caching solutions allow the Cache SSD to be anywhere inside the server itself. This provides additional flexibility as the data being cached can be anywhere on the server - behind a RAID HBA, behind a SAS HBA or even attached to the chipset SATA ports.

In addition to being able to have the Cache SSD inside the server, some caching solutions allow the Cache SSD to be outside the server, in a SAN or NAS. This is important in virtualized servers as this allows virtual machine migration to occur automatically. The cache remains active while the virtual machine moves from host to host.

Consider where you want the cache SSD to connect to your server when choosing a caching solution.

Which OSes are supported?

Think about what OSes exist in your datacenter. Windows? Linux? Virtualized OSes such as VMware ESX or Xen? Think about whether it’s important to have a common caching solution from one vendor across all these environments. Not all caching solutions support all these OSes.

Being able to choose what goes into the cache

This may sound unimportant, but imagine having an SLA with a customer that requires you to deliver the lowest latency to the data associated with that application. What if you could guarantee that specific data was always in the cache, ready to be accessed? Several caching solutions available today offer proprietary ways to pin data into the cache. This is becoming so important that standards bodies such as ANSI T10 are looking into ways to standardize ways to determine whether data should be kept into a cache at all times.

Read caching or write caching?

Look at the applications you want to accelerate. Do they mostly read data from hard drives or do they mostly write data? Or is it a mix? Some caching solutions are better are accelerate reads, others are better at writes. Choose a caching solution that meets the needs of the applications you want to accelerate.

Caching algorithms aren’t all the same

We all learned about Least Recently Used caching in school. Just as the name implies when the cache is full but new data needs to be added to the cache, the data that has been sitting in the cache the longest without being use will be evicted to make room for the new data. This can be an effective algorithm and is very common. But some caching solutions add intelligence to the caching algorithm and are able to decide to keep specific most popular/active data in the cache longer, protecting it from being evicted by more recent, but less popular data. This reduces the probability that important data is evicted from the cache, improving overall application performance.

So, these are just some of the areas to consider when choosing a storage caching solution. What is important to you when you choose a caching solution? Let me know!

Full disclosure: Intel has its own caching solution: Intel® Cache Acceleration Software that works with Intel® Datacenter SSDs. We think it’s pretty cool. A 30 day trial is available on intel.com.

Susan Bobholz is a Marketing Director in Intel’s Datacenter Software Division. She’s been with Intel for 20 years, doing everything from software development to initiative management to product marketing, focused on storage technologies and products. Prior to joining Intel, Susan developed software at Siemens Medical Labs and firmware for Motorola cell phones. She graduated from the University of Wisconsin with a BS in Electrical and Computer Engineering. She holds 3 patents.

At TDWI: Finding a middle way between Hadoop and relational data warehousing

tim.allen@intel.com — Fri, 17 May 2013 22:23:37 GMT

If you’re thinking big data analytics will solve all your BI challenges, you may be looking at it wrong…

I realized this when I was in Chicago the first week of May, attending The Data Warehousing Institute (TDWI) conference, where the theme was “Preparing for the Practical Realities of Big Data.”

There are high expectations around big data at the moment. Many people in marketing, product development, and analytics teams can’t wait to get their hands on big data intelligence to better understand and target audiences.

However, TDWI isn’t their show. At TDWI, the focus instead is on traditional database administrators, data analysts, and data scientists, and it’s a very technical conference firmly based on OLAP and OLTP analytics and hands-on issues of data warehousing.

With this in mind, I attended the keynote address by Ken Rudin, director of Analytics at Facebook—a leader in cool, cutting-edge big data processing and analysis addressing a conference of (what some would consider) old-school DBAs. The message from Rudin, who has held senior leadership positions at Zynga, Salesforce.com, and Oracle, was fascinating: Don’t get caught in the tyranny of either/or when it comes to data analysis—businesses need both traditional relational database processing and Hadoop*-based big data analysis.

@krudin said that Facebook started out relying almost solely on Hadoop and big data when the social media giant first launched, but now is increasingly incorporating OLAP and OLTP processes into its analytics. Hadoop is best at exploring huge data sets—putting all the data into one system to discover patterns. Traditional relational data analytics is best at business, looking at focused data to derive metrics and more actionable, granular analysis. Both are valuable technologies: which one is best depends on what kind of impact you are looking for.

So the question is not, how do you get from old-school to cutting edge as soon as possible. Rather, ask which technology is right to generate impact out of data.

This was a conclusion that appealed to many at TDWI, as several people I spoke to registered a bit of skepticism about the value of Hadoop as an engine for analytics. For them, the main attraction of Hadoop is its potential to act as a backend data storage mechanism, where unstructured data can be warehoused.

Then the question becomes: What’s the best way to integrate data stored in Hadoop into a traditional OLAP or OLTP infrastructure for processing? The answer is just around the corner.

At the SAP booth, I presented a discussion of the newly released joint solution from SAP and Intel that has optimized Intel^® Distribution of Apache Hadoop* software for the SAP HANA* in-memory database. Using SAP Smart Data Access* (watch for availability in coming weeks), this big data solution is able to leverage all types of data for processing in analytical applications.

And since it’s built on Intel^® architecture, and leverages the full power of Intel^® Xeon^® E7 processors, HANA has hardware-enhanced performance and security built in, with solid-state drives and cache acceleration for blazing speed and stability. Watch this interview from TDWI to learn more about SAP HANA and how the database helps address big data challenges.

If you’re looking for the technology to get the most impact out of analytics, look no further.

Follow Tim and Twitter @TimIntel and the SAP analytics team at @SAPAnalytics.

Additionally, @TDWI has some very interesting DW feeds.

IBM Impact 2013: Heroes of IT, Please Take a Bow

tim.allen@intel.com — Thu, 16 May 2013 16:39:05 GMT

At last week’s IBM Impact conference, the focus was on “Technology in Motion,” and the keynote presentations were both inspiring and thought-provoking. The program celebrated some of the 400 IBM Champions from around the world who, as heroes of IT, help open new horizons and new audiences to technology.

Wednesday’s keynote kept with the “tech in motion” theme, and the most stirring presentations involved how IBM is helping open up access to computing for underserved communities. Doug Schmidt of Pearson Publishing described how his company uses innovative technologies based on IBM WebSphere MQ*, IBM Message Broker*, and IBM Operational Decision Management* to bring higher education to students in over 70 countries and in a variety of formats.

We also viewed a video presentation from P-TECH, the Pathways in Technology Early College High School, an innovative collaboration between New York public schools, the City University of New York, and IBM. The P-TECH schools (which include branches in Chicago and Idaho in addition to the original location in Brooklyn) bring together private-sector partners to design an academic program that’s heavy on high tech workplace skills. Students all have IBM mentors to help make sure they get the skills to land good, technology-oriented jobs. P-TECH program has been so successful that President Obama highlighted it in this year’s State of the Union Address: “We need to give every American student opportunities like this,” he stated. Is there a P-TECH school in my town for my kids?

As the focus of Impact is WebSphere middleware, IBM highlighted some recent testing between generations of Xeon^® processors and just upgraded IBM WebSphere* 8.5 vs. 8.1 performance gains. The nearly 260% performance gain with newer versions of software and hardware is huge for the middleware community and hopefully will be sufficient reason to upgrade both processor middleware versions. Details of new SPECjEnterprise2010 test results are detailed here: spec.org/jEnterprise2010.

Intel and IBM have worked together for over 15 years to optimize software performance on the Intel infrastructure, and their collaboration and co-engineering on IBM WebSphere 8.5 has led to very impressive performance gains. And when it comes to technology in motion, it’s this kind of optimized joint engineering that creates new IT heroes. Watch this interview with our Intel hero, Pauline Nist (@panist), general manager of Enterprise Software Alliances, and learn more about the growing collaboration between Intel and software partners such as IBM.

Follow Tim on @TimIntel and @IntelAdrenaline.

SAP and Intel Showcase Big Data Solutions at Sapphire 2013

webadmin@intel.com — Wed, 15 May 2013 17:22:08 GMT

Intel and SAP have worked closely together for years. But, our co-engineering has reached a new level of integration with the recent launch of our Big Data solution based on the SAP HANA^® platform and Intel^® Distribution for Apache™ Hadoop^® software. The joint solution will be demo’d at SAP Sapphire 2013, SAP’s flagship conference in Orlando, held May 14 – 16.

Stop by the Intel booth #3215 and experience our Big Data solution.

The Intel Distribution of Apache Hadoop running on SAP HANA represents more than a typical collaboration. It’s a great example of how co-engineering can result in a “more-than-the-sum-of-its parts” success that changes the game by vastly improving the speed and resilience of Big Data analytics.

HANA is SAP’s blindingly fast database platform that consolidates transactional and analytical workloads into a single, in-memory process. Combining OLAP and OLTP structures into a unified landscape eliminates traditional relational limitations that have restricted the development of real-time business applications, and in particular Big Data analytics.

Apache Hadoop is the industry’s open source standard for managing Big Data, but to ensure that Hadoop’s data-intensive workflows can provide real-time analytics on an enterprise scale requires a comprehensive computing platform with muscle and intelligence.

The Intel Distribution for Apache Hadoop is specifically optimized for the advanced features of SAP HANA and the Intel® Xeon® processor E7 family. Intel’s version of Hadoop can leverage SAP HANA in-memory technologies to accelerate data analytics and also tap into extra performance and scalability that the Intel processor E7 family is optimized for. In turn, SAP HANA’s in-memory processes help eliminate the latencies found in Hadoop’s underlying file system to enable on-demand data analysis.

Through joint-engineering, SAP and Intel have delivered a breakthrough Big Data solution that can store and analyze massive volumes of structured and unstructured data in real time. The underlying platform has the performance to scale to continued exponential data growth and deliver rapid-fire insights to help boost business productivity and profits.

For more information go to: http://hadoop.intel.com/resources

How do you explain the cloud to your parents?

webadmin@intel.com — Wed, 15 May 2013 14:50:24 GMT

How important is storage to your business? My name is Ilene Aginsky and I work in the Storage Division. I’m new to this community and I think this is the perfect place to talk about how the explosion of digital data is changing the way businesses and even consumers need to manage this incredible growth.

It seems that storage should be a relatively easy concept however when data is being created at such a rapid pace it starts to get complicated very quickly. Given this complexity, I will reach out to my colleagues here at Intel and ask them to help explain the technology that surrounds the concept of storage.

For my first foray, I have asked my colleague, Gary McCulley, a Product Line Manager for the Storage Division to explain the concept of Cloud in storage. Below, in Gary’s own words, is that explanation.

How I would explain the “Cloud” to my parents

My parents recently asked me how work was going. Being a Product Marketing Engineer at Intel, I tend to give them vague, (yet accurate) feedback, like “oh, it’s going well” or “Intel as a company is doing well”, etc. In other words: “blah, blah, blah”; I tend not to get into the specifics, fearing I would be in long conversation that would span a diverse range of topics, including explaining how Al Gore really didn’t invent the Internet all the way to how the “Cloud” really has very little to do with the weather. But it got me asking myself if I could coherently tell them about important concepts like the “Cloud” and wondering if they would have a better understanding after my explanation.

Gartner Inc. recently forecasted that one-third of consumers' digital content would be stored in the “Cloud” by 2016. It's an interesting statistic that certainly would impress my parents, even though they may not initially understand it.

Aside -- My parents love me and are impressed by most anything I do or say– sort of like Ben Stiller’s parents in Meet the Fockers, who created a shrine in their son’s honor, including his 9^th place ribbon. By comparison, my mom has a framed picture of a duck that I painted when I was in 5^th grade hanging on their bathroom wall – I earned 2^nd place in my 5^th grade class of 25 students!

Upon telling my parents the “Cloud” statistic, they would likely ask the following questions:

“OK, what is digital content and what is the “Cloud”?

First, I would give them examples of digital content: pictures of their grand kids taken with a digital camera, email they send and receive songs they listen to on Pandora, videos of puppies on YouTube, and other examples. Then I would tell them why the “Cloud” is important. I would tell them the growth of multiple connected devices, most of which have cameras (my mom’s cell phone even has a camera), has led to a massive increase in digital content that users have created. And all that content has to be stored somewhere. I would tell them that people used to store content on their PCs (and still do), but with all this new digital content, people now like to store their data on social media sites, like Facebook, which offer free storage space for uploading videos and photos for social sharing.

I would explain to them that every time they access anything that isn’t stored in their computer they are engaging the “Cloud” - like when they access their Yahoo! email account, Facebook, Google, or YouTube. Basically, the “Cloud” allows anyone with an Internet connection to access information that is contained in a network of computers and servers (i.e. a data center) located around the world. This makes it easier to share pictures and videos, for example, of their kids and grand kids.

In addition to Facebook , Google, YouTube, and Apple - companies that provide free storage, I would also explain that some of them charge customers (on a monthly basis) to store their data – typically to customers that have a lot of data to store. Companies like those above and like Amazon Web Services, which have locations with server farms, (i.e. data centers), will store data for customers for a monthly service-based subscription.

Side note: while online backup services like Amazon Web Services, Google Drive, and others, are well-known cloud storage providers, their total storage allocated to consumers and "pro-sumers" is still small relative to that maintained by social media sites like Facebook, Twitter, and other sites.

Anyway, at this point, my parents would be losing interest, but would still act like they are really interested. So I would try and wrap it up by saying the term “Cloud”is really just a metaphor for the cloud-like shape used to represent the Internet’s infrastructure when engineers draw it.

And then I would tell my Mom that I took a picture of my duck picture and uploaded it to Facebook, where she could enjoy it on her smart phone anytime she liked. I would then be told that the picture isn’t that great, and there was a good reason it is displayed in the bathroom...

Gary McCulley is a product line manager in the Data Center and Connected Systems Group at Intel ® Corporation. Prior to joining Intel, Gary held strategic planning and product marketing positions at Broadcom Corporation, Philips Semiconductors, and General Dynamics. He earned a B.S. in Engineering from Arizona State University, an M.S. in Engineering from San Diego State University, and an MBA from the University of Arizona.

What to Discover at HP Discover US June 11-13

webadmin@intel.com — Wed, 08 May 2013 15:40:03 GMT

HP and Intel have a long standing partnership and it feels like there will be no shortage of exciting things to come the rest of 2013 and into 2014. We are very excited to once again be the premier Sponsor of HP Discover this year. Many of you who are reading this have attended Discover, either in the US or in Europe and we look forward to seeing you again and also seeing some new faces.

What have we got in store for you this year? Well, one thing is for sure, We won’t just bore you with speeds and feeds (okay, maybe a little) but we want to invite you into the platform and technology advancements that become essential to the way we work AND the way we live and interact socially. Our innovation theater sessions will cover both the technological keys to optimizing your datacenter for the future (with sneak peeks into Intel Labs) but also the human future and balance of social power that’s being morphed by the intersection of enlightenment and technology.

Whether you want to think of it as “Return on…” a)”Investment” b)”Information” c) “Innovation “ Intel continues to work with HP to provide the building blocks that are the heart of the datacenter. No longer can you think of the datacenter as silo’d compute servers that provide awesome performance and enable cloud and virtualized environments, but you have to think of the Storage, Networking, Solid State Drives and even geekier technologies that support and augment the datacenter to help you meet your customers’ needs and positively impact your bottom line.

At Discover, look for Intel and HP to take the lead and show you how we can help your organization tackle Hyperscale computing, Big Data,
Cloud and Virtualization. Get in touch! @ITSandhya www.intel.com/xeon

Exascalar and Cost Effective HPC

webadmin@intel.com — Wed, 08 May 2013 13:00:39 GMT

I know… you’re probably thinking, “what the *?” The phrases “Cost effective” and “HPC” seem as rarely seen together as are a double bill of Ferris Bueller’s Day Off and Rocky Horror Picture Show. But, in fact, with rapidly expanding efficiency and performance capability of supercomputing systems, electricity costs in large scale machines may warrant deeper scrutiny of the need for newer hardware.

What got me thinking about this was a startling realization about systems

near the lower left “Corner of Inefficiency” of the familiar Exascalar plot below.

The systems near that lower left corner are almost a factor of one hundred less efficient than the most efficient systems of comparable performance. In other words they consume about one hundred times the energy for comparable work. This can be a big deal, for instance a 20kW system or a 2.0 MW system. If you think about the cost of electricity, there could be some real ROI there.

So the question is how to visualize that difference in cost. The point of what I discuss below is not to provide an accurate cost analysis for every application, but to show how this general framework can be put to use.

Costs of supercomputers, especially those at the forefront of innovation, are difficult to estimate. For the purposes here I chose to use a published cost of the Lawrence Livermore Labs Sequoia computer as the anchor point for this analysis. For comparison read about the ORNL supercomputer here. Assuming a constant $/flops one can easily scale capital cost according to performance. This scaling is shown as the horizontal lines in the Figure below.

Electricity costs also vary widely from location to location. Industrial electricity costs are actually falling in the US, but for the sake of simplicity I have assumed $0.07/kWh with an assumed facility PUE of 1.6. $0.07 is about the average industrial electricity rate in the US. This translates, conveniently, to a total energy cost of about $1/(Watt*Year). You can see system-level annualized energy costs in the Figure.

From this point it is pretty straight forward to calculate a payback time for replacing inefficient servers. It’s interesting they work out to be vertical lines. It’s interesting that they times for return on investment show up as vertical lines. It’s astounding that they are so short. In several cases, less than a year!

Again, this is not intended to be a definitive analysis of return on investment or total cost of supercomputer ownership. But I think this initial estimate is provocative enough to warrant further investigation. To me it looks like millions are on the table.

So, what are you waiting for?

Data Center Power: Zooming in Where it Matters

webadmin@intel.com — Sat, 04 May 2013 15:00:08 GMT

This post originally appeared in Information Management on December 26, 2012

Politics aside, the subject of energy is of great concern in every large data center. Why, then, is power consumption still an afterthought for most server deployments? Because IT and facilities teams typically work independently and neither team can control consumption or predict requirements when data center energy costs are buried in the overall utility bill.

Let’s face it: Energy costs are spiking, server sprawl is pushing against site capacity limits, and the Internet and smart device adoption rates are calling for aggressive increases in data center compute densities. Industry analyst firms agree that power and associated cooling requirements account for the fastest increasing components of operational costs. To protect the bottom line, and to comply with the latest EPA Energy Star standards, data centers need to change the way they monitor and manage energy consumption for power-hungry assets, like servers. New power and cooling management approaches are available that offer greater energy efficiency and reduced costs.

Traditional approaches to managing power and cooling have failed to control costs, in large part because they typically force over-budgeting to ensure priority needs are met. Ironically, even with overestimating and over provisioning cooling, data center hotspots continue to crop up, thereby impacting server availability, reducing data center cooling efficiency and driving up operational costs. These factors and their impact demand that facilities and IT professionals find a better way to achieve their common objectives.

Zooming in on the Right Measurement Points

One of the most fundamental barriers to achieving greater power efficiency and curbing runaway energy spend rates has been the inability to obtain accurate readings of actual server energy consumption levels. Various models have been developed that translate temperature and power consumption into overall data center energy requirements for servers and their associated cooling systems. However, even the best of these models lack the real-time visibility required to accurately understand and predict energy trends. Actual usage can vary significantly (up to 40 percent) from modeled predictions, and the models do not provide the immediate feedback required to pinpoint hot spots before they impact services or identify areas of waste where conservation can lead to savings.

Energy models are limited in terms of day-to-day management of power consumption. For example, we know from in-field measurements that an average of 15 percent of data center servers are “ghost” or “zombie” servers (servers that are not producing useful work, drawing energy just to stand idle). When we do the math, assuming that a server draws approximately 400 watts of power, which currently costs about $800 per year, companies are spending on average more than $24 billion per year for these “ghost” servers in their data centers.

Aggregating Server Thermal and Power Data

A second problem has been the technical challenges of aggregating data from varied and disparate systems within the data center. Facilities managers have been forced to cobble together, manually or with crude homegrown systems, vital data such as power supply of the server, inlet and outlet temperatures, asset information contained in RFID tags as well as temperature and humidity readings of the air conditioning units. This prevents the achievement of a “big picture” perspective of facilities’ server inlet temperatures and power consumption data from rack servers, blade servers, and the power-distribution units and uninterrupted power supplies related to those servers. The crippling effects of this piecemeal view are analogous to a long-distance truck driver suffering from tunnel vision.

By shifting attention from the cooling systems to the servers which account for the majority of the power consumed in the data center, managers can introduce a holistic energy optimization solution. Accurate monitoring of power consumption and thermal patterns creates a foundation for enterprise-wide decision-making with the ability to:

Monitor and analyze power data by server, rack, row or room;
Track usage for logical groups of resources that correlate to the organization or data center services;
Automate condition alerts and triggered power controls, based on consumption or thermal conditions and limits; and
Provide aggregated and fine-grained data to Web-accessible consoles and dashboards for intuitive views of energy use that are integrated with other data center and facilities management views.

Identifying temperatures at the server, versus at the room or even rack levels, can also help data center managers more accurately understand what the real ambient temperature should be for individual servers to have optimal life spans. This assessment of real temperatures has enabled data centers to increase the overall room temperature by one to two degrees, which can create significant savings in the air-conditioning expense.

Disseminating the power and cooling data, without impacting ongoing processing in the data center, is another challenge. Invasive monitoring approaches have the potential for adversely affecting the performance of existing systems. Agentless monitoring capabilities should have little impact on the overall system performance, and therefore are virtually undetectable to the end users’ experiences.

Where should all of this energy monitoring and aggregation functionality be placed within the data center? Ideally, all of this would take place transparently and non-invasively, to avoid impacting the servers and end users. Agentless approaches, without the need for any software on the managed nodes, are available. Data center managers should also look for solutions that are easily integrated, such as those based on Web Services Description Language APIs, and able to coexist with other applications on the designated host server or virtual machine.

Where Power is Going

Today, the goal is improved efficiency and reduced costs, but energy management will become even more critical in the future, as compute models continue to tax power infrastructures. Whatever the goal, the monitoring and aggregation of server energy metrics set the stage for much more comprehensive energy management and a far deeper and richer set of usage models for IT assets. Besides enabling accurate power planning and forecasting, logging and trending power data provides knowledge for data center “right-sizing” and accurate equipment scheduling to meet workload demands.

The thermal data can also be used for more efficient designs of integrated facilities systems, such as cooling and air-flow solutions. Optimized resource balancing in the data center will always be closely tied to power; the expanded insight offered by intelligent energy management approaches will contribute to cost-saving decisions for years to come.

Jeff Klaus is the director of Intel Data Center manager (DCM). Jeff leads a global team that designs, builds, sells, and supports Intel® DCM.

Reflecting on Carbon: Why Invest now in Efficiency

webadmin@intel.com — Fri, 03 May 2013 14:25:38 GMT

I’ll admit it. I read some dismay the stories about the collapsing market for European carbon dioxide emissions allowances, as covered in the New York Times.

Without going into great detail, I feel the most economically efficient way to reduce the deferred costs of carbon emissions is to simply set a price for them today. Does buying roses grown in Africa produce more or less carbon than those grown in the Holland? I’d say a carbon analysis of that supply chain borders on too complex. But if carbon impacts were fairly encumbered at each point of use with a cost, an efficient market would prefer the lower impact source to the extent that carbon futures affect the price of the commodity.

Sadly it appears we have taken a step away not only from that efficiency, but also from addressing the carbon problem.

So are the carbon credits DOA? After some digging, I think not. Here's why - carbon futures are traded on a market like any other commodity and are affected by the same supply and demand economics that affect the price of everything else. So let’s look at both.

The first data I looked at was to compare the fluctuating price of carbon with economic data in this case the changes in combined GDP of four major European economies (Germany, France, UK, and Italy). This shows an important correlation in the demand side. The amount of money available to chase EUA credits is limited.

Another correlation is with electricity generation, which is a reasonable proxy for the demand to use credits. Here I just looked at France and Germany together. Although Europe uses multiple sources for electricity, the majority source is from fossil fuels.

Again, the trend (at least of available data) shows a pretty good correlation. This is yet another way to look at the demand side of the equation. With lower economic output driving lower demand for electricity in turn diving lower demand for carbon allocations, the drop in price seems natural.

On the supply side, of course, the decision by the commission to not limit the number of allocations guaranteed an abundant supply of credits.

So do I think the carbon market idea is dead? No, I don’t. The data “behind the curtain” support the idea that the falling price of carbon allocations is just a simple matter of supply and demand.

Will demand rise and EUA prices rise in the future? Of course they will. Hence, while there may not be any short term imperative to oinvenst in low carbon and efficient technolgies, smart industries should be, I believe, investing now, in the down turn, to gain advantages from efficiency in the longer term.

How can enterprise IT keep up with the lines of business?

webadmin@intel.com — Wed, 01 May 2013 17:18:37 GMT

We have tackled this fundamental question in The New CIO Agenda, a brochure aimed at helping you advance your cloud innovation strategies. Whether your business needs to transform the supply chain or go deeper into social media, cloud services are likely to be part of the solution. If you want to run as fast as the lines of business, capture opportunities from trends like big data and consumerization, or just keep the lights on as efficiently as possible, you need to transform your IT shop into an engine for service delivery.

The brochure, The New CIO Agenda tackles this challenge head-on by talking to some of the industry thought leaders. You will find insights from Intel’s CIO and other visionaries who are driving Intel’s cloud and technology strategies along with examples of companies that are leading their own cloud journeys.

Becoming an engine for service delivery doesn’t mean you need to do it all yourself—or that you immediately outsource everything to publicly hosted services. You start by building a plan which includes understanding the required services to run the business but also by looking for the services that are missing, but valued by the lines of business.

An essential step is to start by modernizing and standardizing your internal infrastructure, if you haven’t done so already. Chances are you’re well on your way to consolidating and virtualizing your server infrastructure—a key starting point for cloud computing. The next step in terms of infrastructure is to modernize and virtualize your storage architecture, migrating traditional scale-up storage platforms onto intelligent, distributed scale-out storage platforms. You’ll also want to unify your networks on 10/100 Gigabit Ethernet, and re-examine every layer of your security architecture. With these actions, you’re building a more secure, scalable foundation for cloud services and freeing up funds that can be used for innovative services.

In parallel, start developing a cloud services roadmap that spells out what services you’ll deploy and how. As a rule of thumb, you’ll want to keep core competencies and sources of competitive advantage on secure private clouds, so you can innovate quickly in areas that are critical to the business. This approach also retains critical expertise in-house. On the flip side, commodity functions are good candidates for externally hosted SaaS solutions.

Whether you’re hosting services internally or externally, you’ll need to clearly define each service, translating business goals into specific requirements and mapping service-level agreements back to the business owners. Pay close attention to requirements around performance, monitoring, auditing, and compliance, and identify ways of measuring them.

With each new service, you build your ability to rapidly deploy additional services, expand existing services, adjust on the fly—and create strategic value. Because one thing we know as IT professionals: The lines of business will continue to surprise us.

For more about building your cloud innovation strategies, watch a video about The New CIO Agenda and download the brochure.
Download technical presentations from the April 2013 Intel Developers Forum, including my presentation for Session CLDS007.
Follow Intel CIO Kim S. Stevenson on Twitter.

Could next-generation computing lead to precise cancer treatments? We think so.

webadmin@intel.com — Wed, 17 Apr 2013 14:26:59 GMT

Let’s arm doctors and biologists with the power of parallel processing to attack the roots of cancer

Attacking a complex genetic problem

Cancer is a genetic problem, and a very complex one at that. The genetic abnormalities that cause tumors manifest themselves differently in different individuals. To make a hard problem harder, a healthy human body creates millions of these mutations and distributes the cell signaling equilibriums. The scientific challenge lies in determining, for each individual, which mutations are relevant in treating the patient’s disease.

Today we are able to gather billions of data points on how a patient’s cells are malfunctioning. By comparing those abnormalities to a normal human genome, we have the potential to narrow down and eventually go to the root cause of the patient’s disease.

Now let’s think bigger. By collecting and analyzing enough data on individual cases, we may be able to detect patterns in how the common circuitry in everyone’s cells can be manipulated to shut down the vast majority of cancers. The science is evolving, and the prognosis is getting better.

The computing challenge

So the goal is to identify and shut down the causes of many cancers. This achievement will require analysis of massive amounts of data that is available today and also that is being collected and accumulated at a frenetic pace because of advances in genome sequencing. What is currently lacking is the right combination of this science with next-generation computing power and capabilities.

With current computing limitations, it can take weeks of nonstop data processing to analyze the genetic data from a single person to identify cancer-causing abnormalities. The other key issue is when more data comes specifically from more patients with a particular type of cancer, doing the related data mining and observing different trends requires too much compute power.

This pushes us to come up with energy-efficient computing devices to address a larger population with far more compute/storage capacity and more power. To make this vision affordable, we need to look at the total cost of computation and energy and drive further gains in energy-efficient computing.

At Intel, we can envision a future—a not-too-distant future—in which a new generation of extreme-scale computing will allow us to compare billions of genetic aberrations in a sick patient with the billions of data points in a healthy human genome in a matter of hours. This comparison will single out the differences that are most relevant to the progression of the individual’s disease—information that can then guide narrowly targeted, personalized treatments.

What is the forecast? Cancer is pervasive. One in three women and one in two men would at some point in life be affected by cancer. It is accidental DNA cell copy flaws and, added to that, carcinogen-based mutations, that lead to cancer. Clearly, this is a tough problem to solve—but we have to believe that we can indeed solve it.

Attacking cancer with high-performance computing isn’t just a futuristic goal. Today, Intel and Oregon Health & Science University (OHSU) have formed a strategic collaboration to make this vision a reality. We are working together to use extreme scale computing to explore solutions for the most challenging problems in delivering personalized medicine to treat cancer and other complex diseases. The ultimate goal is to create a system of algorithms, computations, and models that can select a “high probability of success” treatment for an individual patient.

Ambitious design goals

To tackle this computational genomics problem, we need to optimize servers and software to create solutions that can quickly sort through enormous amounts of data to isolate the genetic variations that contribute to disease. And we need to do this in a cost-efficient manner, using general-purpose processors and energy-efficient configurations.

Here are some of the key design goals for these new solutions that will stem from our collaboration with OHSU:

Develop algorithms and software that parallelize everything to allow dozens of processor cores to work simultaneously on different aspect of a problem.
Develop a processor capable of doing all of the data crunching on a minimal amount of energy-efficient servers to avoid the high energy costs that come with running multiple servers in a supercomputing cluster.
Use accelerators to speed up computations.
Make the communication between the processor cores and servers highly efficient.
Process data in memory to avoid the I/O latency that comes with data going back and forth between memory and disk.

The Intel contribution

Intel is uniquely positioned to deliver the computing architecture that will solve today’s computational genomics problem. We know the hardware better than anyone, and we know how to optimize systems to make them run faster and be more energy efficient.

We now have the capability to deliver, say, 72 cores (Intel™ Xeon™ Sandybridge server with one Intel Xeon Phi) in a single server. This number will grow in the future, as will the processing power of the cores, as we design architectures to address a new generation of computing challenges.

Our collaboration with OHSU may even lead to advances in Intel Architecture, as we incorporate our learnings into the design of next-generation processors.

Ultimately, we believe that Intel technology has the potential to serve as the foundation for more advanced diagnostic tests that can zero in on all relevant abnormalities in a patient’s genes in a matter of hours.

Gans Srinivasa is a Senior Principal Engineer for Intel Corporation.

Big Data and its Role in Driving Technology Trends – Intel’s Data Center CTO and Senior Fellow Steve Pawlowski Shares His Insights

webadmin@intel.com — Fri, 12 Apr 2013 15:29:52 GMT

How big is big data? Imagine a petabyte of data…which in video form represents about 10 years of HD viewing. Scientists, corporations and people are producing petabytes of data an amazing rate….an average Chinese city’s traffic cameras produce a petabyte every 12 hours or so, and the CERN supercollider produces a petabyte per second. These massive stores of data represent an enormous opportunity for analytics, but in order to scale technology to process data this size and scale represents an enormous technology challenge for the industry.

This is probably why Intel Senior Fellow Steve Pawlowski’s session at the Intel Developer Forum was a standing room only affair. While everyone talks big data today, no one has the complete map to where we need to go to fully unlock the benefits of big data. Steve broke this problem down with a focus on data center innovation, compute and storage innovation, data protection and context and location as areas for industry innovation focus to harness the full value of big data.

In order to scale data center capability, the efficiency of data generation and compute at a data center level need to be addressed.

Steve discussed Rack Scale Architecture as well as data center level innovations including ambient cooling, DC power, optimized PUE to help address this challenge. He also pointed to better management of infrastructure power instrumentation with technologies like Intel Data Center Manager as being focuses of Intel as driving the power dynamics across the data center. He then drilled into system efficiency discussing platform technologies including improved fan speeds, high efficiency power supplies and voltage regulators, and liquid cooling as technologies to improve server level efficiency, and processor advancements can push this efficiency further.

Steve then changed focus to discuss the importance of efficient scaling of memory capacity for big data. While today’s systems scale memory through increased DRAM, this scaling is expensive both from a cost and a power consumption perspective. Steve pointed to new memory technologies on the horizon that would introduce the performance capabilities of DRAM with power and cost dynamics of NAND memory as being an emerging alternative for the data center.

With a path defined towards memory innovation, Steve then moved on to the I/O and interconnect innovation required to move all of this data from data collection to data analysis. His first topic in this field was the breakthrough represented by silicon photonics, a new capability invented by Intel to use silicon lasers to dramatically reduce the cost of optical transmissions. He then discussed the growing challenge in wireless communication with an emerging spectrum shortage stating that by 2020 we will only be able to meet 50% of the demand for spectrum usage. Steve pointed to delivery of software defined radios, devices that could identify available spectrum and modulate point to point communications to available spectrum on demand as a potential solution to our spectrum challenge, but admitted that regulatory control of spectrum must be addressed as well to put this solution into practice.

Steve followed his focus on moving data to the challenge of security discussing ensuring data integrity within the data center and as data is in flight. While current solutions try to keep pace with today’s environments, the news is rife with stories of data exposure often for nefarious purposes. The industry needs to do more to ensure data integrity, and Steve called for a holistic approach that focuses on defending against firmware attacks, protecting privacy and data security, ensuring application security, and overcoming programming errors and developing applications for failure.

Finally, Steve focused on the analytics frameworks themselves highlighting the early progress of Hadoop and pointing to the need for the industry to develop many algorithms for analytics. While the parallel batch work of frameworks like Hadoop will get us started, some of the most compelling uses of big data such as genomics or social searching involve graph approaches. Steve provided an example of investment by Intel’s research team in development of genomics algorithms speeding results 10X and stated that in the future compute will be driven by the requirements of these algorithms vs. forcing algorithm development towards a static compute platform.

A New Era of Insight: Diane Bryant Outlines the Future of the Data Center and Beyond

webadmin@intel.com — Thu, 11 Apr 2013 14:44:19 GMT

One thing we all know about the Intel Developer Forum is that it provides an insight into the future of computing, and Diane Bryant did not disappoint providing the audience with a view into how data center computing is evolving to address the vast needs of our changing society and the rapid pace in which we are using technology to drive our businesses, manage our cities and keep us connected. Diane began her speech by outlining some of these macro changes in China: growing urbanization, rapid ramp of citizens with access to computing platforms, and technologies aimed at solving major challenges such as the rapid growth in urban traffic. Data centers play a critical role in delivering this seemless compute experience, and Diane described how behind each device from sensor to tablet to smart display, rests a server. It’s a simple concept but one that quickly emerges as a driver of the rapid growth in data center compute capability the world is experiencing when you consider our ramp towards 3 billion connected individuals and 10+ billion connected devices across the globe.

Diane outlined Intel’s approach to helping deliver the foundation for the underlying infrastructure to power the data center in a number of ways. The first focus was analytics. With Intel’s recent foray into the Hadoop distribution arena, Intel’s footprint in how unstructured data from sources such as social media streams has expanded from hardware to software and solution delivery. This was apparent in the end to end traffic management solution described by a leading Chinese manufacturer, Bocom, whose partnership with Intel has shifted the company from image capture to 2.6 Petabytes of data capture and real time analysis using Intel powered analytics from the edge to back end data center. The power of Intel’s Hadoop distribution was also demonstrated by Intel’s own Jason Waxman as he showed the power of infrastructure innovation at the processor, drive, and network matched with code optimization delivering a 30X improvement in response time vs. previously available solutions. More about the Hadoop distribution from Intel here.

Diane then branched into the land of the high density data center discussing the latest from Intel in rack scale architecture delivery as well as progress on microserver platform innovation with a world’s first demo of our next generation Avoton platform and announcement of availability of our first Atom based storage solutions only 4 months after the delivery of our first Atom based microserver platforms. To discuss the importance of this area of innovation, Diane welcomed the chair of China’s Project Scorpio to the stage. Project Scorpio is driving large scale data center design innovation and is an effort spearheaded by China Internet behemoths Tencent, Baidu, AliBaba and China Telecom. When you consider that over 550 million people in China access the net daily, the work by groups like Project Scorpio will help drive innovation in the data center on a massive scale. To help deliver the capability of the cloud to Chinese customers, Diane announced a new China Cloud Lab to be hosted in Beijing helping optimize cloud solutions on Intel Architecture. More info about rack scale architecture and Avoton here. (link to Raejeanne episode we just did)

Diane concluded with an update on data center platforms highlighting that 2013 will bring new solutions across the platform lineup. For E7 she announced the new Intel Run Sure technology and highlight how new innovations in IO and memory will help drive new capabilities for big data and in memory database solutions. In E3 she discussed the improvements in graphics capabilities which will help a wide array of applications such as described in Bocom’s traffic monitoring solution. She summed it up stating that for every workload that data center customers face, Intel is expanding its large technology portfolio to deliver the world’s best solutions optimized to address customer requirements. And really, with its heritage in working with the industry to deliver the best computing experiences on the planet, what else would you expect from Intel?

Two generations of Intel Atom SoC to power HP’s Moonshot servers in 2013

webadmin@intel.com — Mon, 08 Apr 2013 15:25:42 GMT

Intel Atom S1200 helps HP Moonshot to lift off today.

HP and Intel® have been collaborating for a few years on project Moonshot to bring new levels of density, efficiency and TCO for light weight web workloads such as static web and dedicated hosting. Yesterday, HP unveiled their first generation Moonshot systems and I am excited to see that the first and only production HP ProLiant Moonshot servers available today are based on the Intel® Atom® S1200 processor family.

HP chose to lead with the Intel Atom™ processor for many reasons. First, HP and Intel have a long history of collaboration and we have brought many innovations to market first on Intel and HP’s platforms. Second, the Intel Atom S1200 processor is the industry’s only available 64-bit SoC with critical data center class features such as full 64-bit software ecosystem support, ECC and Intel Virtualization Technology - all within an ultra-low power 6W TDP. This means that today the ProLiant Moonshot servers using Atom S1200 can drop into any environment and software applications will run seamlessly on the server without porting needed. The lower power you want, with the software applications you need. Third, this SoC was designed for targeted lightweight web scale workloads, including low-end dedicated hosting, simple content delivery, and offline batch analytics making Intel Atom S1200 the perfect SoC solution for HP Moonshot’s target markets.

Moonshot servers with the Intel Atom S1200 are shipping to customers today and receiving great reviews. We look forward to sharing more results from customers going forward as these implementations go public. We also look forward to seeing Moonshot systems that will take advantage of higher density HP ProLiant Moonshot servers using Intel’s next generation Atom SoC coming later this year. The next gen servers will be built on Intel’s 2nd generation 64-bit Intel Atom SoC, code named “Avoton”. Avoton is built on Intel’s leading 3D tri-gate 22-nanometer (nm) process technology and is based on a new microarchitecture codenamed “Silvermont”. It will feature an integrated Ethernet fabric controller and deliver improvements over today’s Intel Atom S1200 in performance per watt and energy efficiency through a combination of new capabilities, new microarchitecture and leadership manufacturing technology. Avoton is now being sampled to customers and the first systems are expected to be available in second half of 2013. Moonshot servers using Avoton will quadruple the density (4 Avoton SoCs per server) vs. the current generation just announced using Intel Atom S1200.

Want to see these two fabulous energy sipping SoCs? Here you go. Avoton is on my right, the Intel Atom S1200 on my left.

2013 is and will be a great year for Intel and HP Moonshot. We have not only enabled the first Moonshot system to lift-off but with Avoton we will also bring HP Moonshot’s customers a revolution in energy efficiency and performance per watt to drive major TCO improvements when processing lightweight web scale workloads. For more information on this announcement and the future of Intel’s microserver products, please visit www.intel.com/microserver or contact me on twitter with more questions - @RaejeanneS.