I was recently involved in a project where Intel IT SMEs from disciplines including Server, Storage, Data Center, Network, and Finance reviewed and updated our Data Center Strategy (Intel IT Data Center Solutions: Strategies to Improve Efficiency) for Intel IT.  The primary focus of the paper was to provide an update on value realized, shifts in strategy, and key execution lessons learned.

Our execution highlighted the need for finance to participate as an active partner in the influence planning and internal communications.  At some point, especially in economically challenging environments, cross organization investment decisions boil down to a tradeoff between limited resources and a number of good projects. Being able to clearly articulate the value added by a "portfolio of projects" (like the Data Center Strategy) and how you will track progress doesn’t mean that the project(s) will be funded – but it does increase the likelihood that you will be in the game at the end.  For us, having this coordinated communication strategy for technology solutions,cost efficiency, and operational efficiency was a key consideration for successful execution. 

We currently estimate that the cumulative projected financial impact over eight years will be ~$500-650M NPV - this range has changed in upper and lower limits based on updates to forecasts.  Over the first three years, Intel IT has realized ~31% of the projected benefits through execution to the Data Center strategy.  The primary value driver has been the impact of our server strategies (multi-core refresh and virtualization) that enable demand growth within the existing data center footprint and affordability targets.  Moving into 2010, we are evaluating new forecasting and value metrics to enhance customer reporting of data center activities.  This approach will incorporate our activity driver methodology into comprehensive unit costing and forecasting framework, creating a holistic cost forecasting process to improve future decision making.

One area currently under review is establishing the right unit of measure for a data center infrastructure housing different compute environments.  Is this something you or your business partners are exploring or looking to explore?

At a recent event our CIO, Diane Bryant, talked about our continued plan to replace old servers in our Data Centers (http://www.tgdaily.com/content/view/44213/135/). Here is a summary of her key points:

• Not replaceing servers could have costed Intel $19 million due to high maintenance and cooling cost
• Our plan of refreshing old servers with Nehalem servers will save Intel $250 million over 8 years

If you are an IT manager looking at where you can find extra dollar in your IT budget to invest in new technology, new innovation and new competitive capability for your organization, this must be good news for you! Moreover, if you do nothing, you are opening a hole in your IT budget.

Here is a recent white paper and a video we published to discuss our server refresh strategy and how we are getting the cost benefit Diane Bryant shared:

Realizing Data Center Savings with an Accelerated Server Refresh Strategy

We have also developed a Server Refresh ROI estimator so you can calculater the amount of savings you can get from these cash machines:

http://www.intel.com/go/xeonestimator

If you ain't satisfied, here is a video showing you how to use the estimator!

Go and install those cash machines into your data centers now! 8-)

I hate fixing the roof.  In fact, I have been postponing a roof repair over my garage for about 2 years now.  I recently read an article by Peter Kretzmen titled “IT, The CIO, and the Business Need for Roof Projects” and realized that while I can put off my roof repair, IT may not be able to postpone routine upgrades. 

For businesses, technology refresh is a standard business process (ie a roof fix).  The question for IT often boils down to WHEN I should upgrade, not IF. Why? … because hardware technology ages, maintenance costs rise, and software solutions can become unresponsive or obsolete as business needs change, user needs evolve and new technology and software become available. In this economy, cost is king and reducing IT costs has clearly become a critical imperative.

My colleagues in Intel IT recently conducted two separate and independent studies on how frequent we should refresh our PC fleet and data center servers.

PC Fleet Management:  John Mahvi and Avi Zarfaty from Intel IT recently wrote a paper titled “Using TCO to Determine PC Upgrade Cycles”.  The conclusion of this analysis showed that a 3.5 year refresh rate was optimal for total cost management in our IT environment.  Despite the fact that delaying PC refresh this year was initially seen as a cash conservation approach, the analysis showed that not refreshing older PCs increased the business’s overall costs.  As a beneficiary of PC refresh (I got a new laptop a month ago ), I can also personally attest that my productivity has gone up.

Data Center Efficiency:  Matt Beckert and Diane Boyington of Intel IT recently published a paper titled “Realizing Data Center Savings with an Accelerated Server Refresh Strategy”.  This paper discusses Intel IT’s movement to a proactive 4-year server refresh cadence in 2007 and illustrates both the long term savings (up to $250M over eight years) and immediate benefit to the corporate bottom line ($45M saved in 2008). After plans to refresh our servers was slowed earlier this year to preserve capital funds, a re-assessment was done that showed that Intel IT could save $19M by refreshing now vrs waiting until 2010.

Just like you shouldn’t sleep in a house with a leaking roof … it is prudent to not let old hardware create a hole in your IT budget. In today’s economic environment, Intel IT can’t afford a leaky roof and so we are moving forward with proactive business client PC and Server refresh, proven approaches to reduce TCO and boost business value.

Chris Peters, Intel IT

twitter @chris_p_intel

I have just returned from the Intel sponsored Eco-Technology Great Debates where I was slotted into the topic of Thin vs. Thick Client Energy Efficiency. I had the opportunity to weigh in on the side of "Thick" clients as the most energy efficient. The bad news is that our team lost; the good news is that we didn't lose by much (29 to 24)! The best news is that all of the teams had some very strong arguments (and even several very entertaining exchanges).

Being a simple data center guy, I learned a lot, especially as it relates to thin client architecture and energy impacts. No contest, thin clients consume less energy at the device level than do thick clients (PCs and Laptops). But is that really the energy efficient answer?

For thin clients, compute and storage are necessarily displaced to the data center. Data centers with thier concentrated IT equipment are typically inefficient to power and cool relative to laptops and PCs which are distributed by nature and cooled by ambient air. Generally data centers require 1 watt of power for cooling and electrical distribution (house load) for 1 watt of IT load (newer data centers are more efficient but still incur additional power costs simply to power and cool). Therefore, every kW of power that is shifted from distributed thick client use to a data center causes more or less 2 kW of impact in the data center! Wow!

With the majority of the world's data centers facing power or cooling capacity constraints and some with no additional grid power available at all, total energy costs extend beyond the simple house load + IT load equation. Expansion and upgrade of facilities increases energy consumption, as well. There are too many areas to detail here but needless to say the total power consumption for extracting and manufacturing data center components, transporting them to a site and construction of new facilities is non-trivial and likely larger per unit of compute than for the typical laptop. This collateral consumption is not comprehended in any calculations of alternative client model power efficiencies of which I am aware..

I also have no specific data on the power efficiency of PCs or laptops to provide rigorous comparison to data center power utilization efficiency. The above arguments, however, do appear to be logical. More work needs to be done to collect the data and analyze these concepts in detail.....

If you want to see the instant replay of all of the debates (including the client debate, liquid vs. air cooling and ac power vs dc power in the data center), click on the web link above and look for the embedded webcast URL at the bottom of the resulting page. There are also a couple of links to other articles on the subject that are well worth reading.

TTFN!

Do Not Wait for an Alarm or Failure Give your Data Center a "Health Check" using a simple hand held Infra Red (IR) Gun. This tool can provide early warning for electrical breaker overload, CRAC unit calibration issues, server air supply stratification, source of CRAC short-cycling. See the image below and use the number references for legend. The cost of the tool is between $100 and $500 the higher priced guns are recommended for the multiple features

1. Check temperature range of breakers

Check panel cover for ambient temperature, then breaker temperature range. Look for outliers hot and cold. Hot could be loose wire or overloaded circuit.

2. Check under floor for poor air flow

Floor tile temperature is a quick check for restricted air flow or range beyond CRAC.

3. Check actual temperature of delivered air (Supply air)

Concrete in front of CRAC should be around 55 degrees Fahrenheit.

4. Server in-take temperature on rack frame low

Rack frame at first server position compared to temperature at top of rack shows air temperature stratification or rack heating from conductive heat loads. Temperature range of 6 degrees is good. If more than 10 degrees, look for hot air mixing from above or behind servers. Max intake air temp greater than 90 degrees is a great risk to the server platform.

5. Server in-take (supply) temperature on rack frame high

Plus 6 to 10 degrees is the range from good to poor. (See note in 4 previous)

6. In-coming air (return air) temperature off sheet metal frame

Temperature in center of CRAC filter bank is a good indication of actual ambient mixed air returned to CRAC. Compare this temp with CRAC thermal readout for indication of short cycling or bad CRAC temp sensor.

See previous Blogs at

Data Center Toolbox for Power and Cooling

Data Center Toolbox \\"Watts per Square Foot of What\\"?

See Published articals at

http://searchdatacenter.techtarget.com/originalContent/0,289142,sid80_gci1275008,00.html

http://www.cio.com.au/index.php/id;537667845;fp;4;fpid;51245

http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9028098&pageNumber=1

Please comment on and rate this Blog.

New topics coming soon:

"Generic Data Center Racking, Cost and Space Benifits"

"Data Center Layer One and Structured Cabling Designs, Without Costly Patch Panel Installations"

"Server Power Cord Management"

"Humidity Management to "Humidify or Not Humidify"

Disclaimer

The opinions, suggestions, management practices, room capacities, equipment placement, infrastructure capacity, power and cooling ratios are strictly the opinion and observations of the author and presenter.

The statements, conclusions, opinions, and practices shown or discussed do not in any way represent the endorsement or approval for use by Intel Corporation.

Use of any design practices or equipment discussed or identified in this presentation is at the risk of the user and should be reviewed by your own engineering staff or consultants prior to use.

Recently, a colleague and I spoke to a group of IT administrators in Washington, DC. We left our car in a self-park parking lot in which the attendants had everyone leave their keys in their car, in lieu of keeping them on a valet "key board". They seemed to be depending on reasonably honest customers (we were in a secure area past a government checkpoint) and their own memories to ensure no cars were "lost". We returned to find that the parking lot attendants had completely rearranged the vehicles. Since it was a rental car, it was hard to describe the car and therefore hard to find. (By this point you're probably thinking that I've posted to the wrong board or that Intel pays me by the word, but bear with me)

It took a rather lengthy iterative search, but we eventually found the car. As we walked, my colleague and I joked about this as "parking lot virtualization". Our vehicle was moved from one slot to another to better fulfill the changing needs of the parking environment over time. This struck a chord with us, having just been discussing some of the challenges with virtualization.

In the data center, most virtualization suites allow an administrator to manually move a workload from one host to another. This is a very powerful concept - instead of having to negotiate for a 3:00am Sunday morning maintenance window to do preventative hardware maintenance, we can move all of the workloads to another physical machine, perform maintenance during normal working hours, and eventually move the workload back to its original location. We can also migrate workloads from a less powerful machine to a newer machine for performance or in order to retire hardware.

Combining this capability with the ability to host multiple workloads on a single piece of hardware, the data center can quickly become very complex. Without a robust database to map workload to physical machine (and vice-versa) or an automated update mechanism to adjust these mappings after a move, we can easily lose track of our services. These mappings are needed in order to answer questions like "host/rack/row/room x went down - what services need to be restarted?"

My colleague noted that ITIL has mature, well-defined mechanisms to deal with many of these types of events. Change orders, maintenance escalations, and configuration databases were all designed with these business processes in mind, albeit at a much slower (and more manual) pace. It would defeat much of the benefit of virtualization if one had to get a signed piece of paper, email approval, or file a trouble ticket in order to offload a workload in response to a failed CPU fan. Instead, we should use policy to anticipate and enact these types of responses. The discipline and rigor of change management is critical within the virtualized data center, but it must be directly encapsulated by our tools in order to be effective. In essence, the CMDB needs to be dynamically updated in order to maintain fidelity to the Data Center's logical state at any given instant.

For those of you who have deployed virtual machines in large-scale production, what techniques have been most successful for managing the chaos of moving services and images? Are you using a glue layer for your legacy CMDB and other management tools, or are you finding it easier to throw them out and depend on the tools provided by your virtualization stack?