It's great server weather here in New Mexico today. The current temp is 74f , the high will be ~82f ,the low was 57f ,and the humidity will be ~30% all day. These are all "in range" of the air needed for a densely packed server to breath. Depending on how many servers are packed into a rack, they can heat this 74f degree air by as much as 50f degrees. Removing that heat from the server exhaust air consumes energy and expensive equipment. A key system trade off in data center design is that density reduces the unit cost of deploying and operating servers for most inputs but increases the cost of dealing with high density heat. If we can find ways to address this heat with less energy and cost then we can be more economical, and in today's words, more green.

It was on a day like today, a while back, where Tom Greenbaum, a fellow Intel engineer, and I were brainstorming how to get outside air directly to the servers. Because of Tom's experience in custom air handlers, we were focusing on economizers that are normally used for buildings. Economizers are commonly designed into office buildings, homes and in a simple fashion, most cars, but they are rarely are thought of for data centers. The key concept is that you have access to two air supplies: the outside air and the exhaust air. To be most economical, you want to continually decide "what is the best air to use next?". This takes measuring, deciding and switching. In your car, you are both the measurement device and decider as you press the re-circ button or the flow-through button on the dash. In a highend building air conditioner this is an outside weather monitor, an inside weather monitor, a simple controller and some extra duct work.

In our experiment we used an vane controller in a looped air duct that could blend outside air with the exhaust air to get us the best air for the servers to inhale at the lowest cost. The accuracy and controllability of our monitors and controllers were not as capable as we wished, but they did the job. The perfect controller would have measurements of outside air and exhaust air for: temperature, enthalpy and dew point. It would have policies that you could modify for best economy based on the requirements of your equipment. For example, it should be able to blend exhaust heat into the outside air to hold the minimum temperature at say 55f and no more in winter and it should be able to start incremental cooling loads as the temperature of the coolest of the two supplies rises above the maximum allowed by your equipment, in summer. It would decide to use some exhaust air when the dew point was higher than the outside temperature to control air humidity. You get the point, it would condition the air using all available inputs. Our PoC used DX cooling based units that usually are considered not as economical as water based cooling. But, in this mode of operation, they worked well and reduced complexity. In addition they used no water which is a plus in many desert locations. You can imagine evaporative systems in similar designs that could replace the DX units or work with the DX units for even more "economi-zation".

In the video you will see dust on the back of the servers. We had filtration on the air intakes and a control system that can indicate when the filter needs replacing, but we had a door system that let in dust near the inputs during really windy days. It was a design flaw in our temporary room. Once it got into the system we decided to let it run to see where it went. It collected in the exhaust areas but then created very little risk becasue most of the time we exhausted the air. Something that would not have been true in a closed loop system.

Don Atwood was able to negotiate for and create the production capable configuration for a sufficient number of production servers that were dense enough to run a proof of concept (PoC). These servers run high volume batch computing and are nearly always running above 90% utilized, perfect heaters for the job. It is important to note that this PoC was dependent on the concept of a "Compute Center". A Compute Center is the idea that high density servers can be isolated in their own air space a very short distance from the storage. The storage is left in the classical, and perhaps now more aptly named data center. Where this concept is able to be used, it can help free up traditional close loop environmental control for storage systems. If anyone knows of a great economizer controller, please lets us know. An Atom based design would be a plus

The temporary "compute center" we established and operated would not have be successful without the help of several great engineers contributing insight and innovation. Tom Greenbaum, Marvin Bailey, Steven Bornfield, Natasha Bothe, Greg Botts, Demetruis Ferguson, Ryan Henderson, Dan Links, Don Wright all contributed to learning what is possible during our PoC.

Well, it's 79f outside now and still great server weather, there are hot air balloons in the sky this morning, as we get ready for the Balloon Fiesta here in early October. Hot air balloons, like racks of servers, love this air becasue they can inhale cool air and then heat it to create work that moves people. Finally hot air balloons spend very little energy exhausting the resulting heat out the top. Its a simple model really, it just takes a smart controller.

Video: http://video.intel.com/?fr_story=2d6e0fbbef76b72c6119cc7fe7889bba20cb5192&rf=bm

Paper : http://www.intel.com/it/pdf/Reducing_Data_Center_Cost_with_an_Air_Economizer.pdf

As this is my first blog on this forum, I'd like to introduce myself. My name is Bill Sunderland and I have been working at Intel for 12 years primarily working on Server Hardware Engineering and the last three years of which I have focused my efforts on Program Managing the Virtualization Engineering release for Intel IT. I have recently published a WP demonstrating the methodology used as described below.

Intel IT planned, engineered, and has begun deploying a virtualized business-computing production environment at several data centers, a rollout that will continue through 2008. Our initiative has already confirmed anticipated virtualization benefits such as faster, more automated deployment. We are initially consolidating older servers running applications that are not mission-critical; we see opportunities to achieve 16:1 consolidation ratios.

Click here to read the WP: Implementing Virtualization in a Global Business-Computing Environment

I would be interested in hearing your experiences and/or questions regarding virtualizing IT environments!

Doug Garday continues his podcast series with part II, which continues the discussion around a heat recovery system to reduce the total cost of ownership. In this podcast, Doug plugs in numbers that show potential energy cost savings.

To listen to Part I go to What if you invested a dollar and it returned 10?. View the full brief at Data Center Heat Recovery Helps Intel Create Green Facility.

Most data centers have a very long history with the enterprises they provide services for. Data centers grew up around the users they provide services to and are generally located within a close proximity to the user base. As LAN capabilities improved, performance of local applications became less of an issue for enterprises but new capabilities were generally landed in existing facilities. As the enterprise grew through mergers and acquisitions, additional data center capacity was generally co-located with the new users.

When availability was the key metric for measuring performance of applications, it became customary for each new application that the enterprise was rolling out (whether developed or purchased) to have its own dedicated hardware, to protect the integrity of the application and improve stability. Over time this has led to data centers around the world that are full of hardware that operates at a fraction of what it's capable of! Application owners were loathe to consider the idea of ‘stacking' applications on the same server, looking to avoid potential conflicts that arise in a shared resource environment.

We now have multiple vectors for driving efficiency into our data centers: energy efficiency, sustainability and cost are some that are moving organizations toward initiatives that will transform the way we provide services from the data center and how we support those services going forward. Driving efficiency is a painful but necessary step in the overall transformation.

Transformation can mean many things to the enterprise but part of a data center transformation generally involves consolidation of data center facilities and compute resources that provide services to the enterprise. This can cause some amount of nervousness on the part of the end users, application developers and administrators who have grown accustomed to unfettered access to resources over time.

Why is this abstraction of resources from users, developers and administrators necessary? A primary driver is to standardize facilities, network, compute and storage resources so the operations staff is sustaining a small number of standard offerings, which provides them with a very predictable environment that they can become expert is sustaining. By centralizing to fewer facilities and driving toward higher utilization of existing resources (i.e. network, servers and storage) the enterprise can obtain more work out of their data centers for less energy and therefore less cost.

Removing the users, developers and administrators from the data center is a process and mindset change that will take resolve and ultimately executive support. So many IT pros feel they need to ‘touch' their environment but this leads to custom configurations, unknown/undocumented changes and instability in an environment that we're trying to standardize. Building the competency within the operations group will facilitate a change in the way excursions and outages are dealt with and will form the basis of a much more predictable environment that the operations team will feel proud to own. This is one piece of the transformation that cannot be overlooked when considering the evolution of the enterprise data center.

I've posted to this blog a few times in the past, but my "day job" (Data Center Efficiency) has kept me away from blogging for longer than I'd like. My colleague Brently and I were recently in Folsom for meetings, and took some time out to talk about some of the key elements of our data center strategy: working horizontally rather than tailoring solutions for a particular business unit, giving transparent access to a global pool of resources, and improving data quality.

I'll plan to increase my posting frequency, using this blog as a forum to share some of the things that have worked for us (or that haven't worked) as we reduce our data center footprint. I'd like for this to become a conversation rather than a broadcast, so please let me know if you have any questions or would like more details on any specific part of this initiative.

Is hypervisor based server virtualization basically a “useful” tactical consolidation “play” or does it form the basis for much more? For several years, server virtualization has been used with good success by many organizations to consolidate legacy workloads as well as to cost-effectively provision test and development virtual machines (VMs). During the same timeframe, there has been significant discussion pertaining to “the datacenter of the future” – a centrally managed “dynamic” datacenter where resources will be dynamically assigned “on demand” to workloads in an agile manner thereby allowing the datacenter operations teams to quickly adapt to fluctuating demands for compute resources. So, “where is the connection?” you ask.

The recently concluded VMworld 2007 may offer some hints. At that conference, VMware Chief Scientist, Dr. Mendel Rosenblum, demonstrated some key capabilities that show where compute virtualization is headed in its evolution to becoming the “datacenter operating system” of the future. Dr. Rosenblum demonstrated “storage live migration” and “lockstep VMs”. Transparent migration of storage “owned” by a VMs from one disk (LUN) to another (potentially on a different storage frame) without impacting application availability was demonstrated in the first demo. He also demonstrated the ability to transparently failover an executing VM to a “secondary” VM – on another host - that has been executing in lockstep with the primary VM! If you did not have a chance to see Dr. Rosenblum’s keynote, a webcast is still accessible here.

Per the VMworld demonstrations mentioned above, the hypervisor based virtualization capability and upcoming hardware assists for virtualization in combination with a “compute virtualization manager” that collectively enable policy-based management of the datacenter including virtualization hosts, VMs, storage is well on its way to evolving to the dynamic “Datacenter OS” of the future. This powerful Datacenter OS of the future – evolved from today’s “tactical” virtualization capabilities - will dynamically manage compute, memory, power, storage and network resources of the Datacenter in an automated, highly efficient manner. Pre-packaged applications in portable VMs will be landed on these dynamic datacenters in a flexible and agile manner.

While the hypervisor based virtualization based evolutionary path may not be applicable to highly specialized datacenters (e.g., HPC), its applicability to Business computing environments is clear. Am I oversimplifying and underestimating the technical challenges (security and scaling to name a few) along this evolutionary path? Probably. Am I glossing over the organizational and business process related challenges? Almost certainly. On the other hand, are all the major building blocks like efficient hypervisors, virtualization hardware assists, VM live migration, storage live migration, power management, record and replay, lockstep VMs…) successfully prototyped or better (in production)? The answer is an emphatic “Yes”. Do you agree or do you forsee insurmountable barriers that will thwart attempts to evolve the virtualization capabilities of today into the dynamic data-center of the future along the lines suggested by the ground-breaking capabilities demonstrated at VMworld? Either way, I’d really like to hear your perspective.

Like most other companies these days, Intel is facing a growing demand for computing resources. As a result, our computing costs are going up along with that demand. All of these issues prompted us to take a hard look at our data center strategy, to see where we could make it more efficient.

Therefore, we’ve launched into a significant undertaking. We’ve started the process to End of Life (EOL) or consolidate our data centers down to just eight strategic locations. This effort is planned to take us eight years, but we’re working to pull this in sooner. This initiative enables us to reduce costs, improve server and storage utilization, create higher density & more energy efficient data centers, and allows us to keep pace with our company’s rapid rate of innovation. The effort could deliver up to $750M in Net Present Value. View my video blog for more information.

I would be interested to hear your comments, questions and have you share with us what your company is doing to drive efficiency within your data centers.