The energy spent spinning up a flywheel, recharging a battery or pumping water up a reservoir can be recovered at a later time with with the appropriate infrastructure, minus a percentage loss lost to heating. This behavior is governed by the First Law of Thermodynamics.
Alas, there is no such luck in the operation of a data center. There is energy stored in the UPS batteries and the capacitors in the equipment, but this amount is minuscule compared to the total amount of electric energy fed into the data center. Hence, it is fair to say that all the electricity fed into a data center eventually gets converted into heat, warming up the air, ground or water around the facility. Again there is no way around the First Law of Thermodynamics.
In any case, the useful output for a data center is not the amount of energy that eventually gets to the UPS batteries or recharges capacitors in servers. It's the amount of computation done at the data center. However, counting CPU instructions is difficult and controversial. Hence it is common practice to settle for the next best metric as a proxy for computation, namely, the power consumed by the CPUs in all servers.
Measured as percentage of total data center power consumed, the CPU power consumption is rather small. Ainsworth, Echenique et al. from IBM (Figure 1-1, page 3) report that only 35 percent of the data center power goes to the IT equipment load. Likewise, power consumed by processors represent 30 percent of the IT equipment load. The number needs to be further derated to the CPU utilization, 20% on the average. If we do the math, the power dedicated to computation is about 2 percent of the total data center power.
John Pflueger from Dell (figure 1, page 9) reports a remarkably similar result. He estimates that 41 percent of the data center power is consumed by IT equipment, broken down into compute servers, storage and communication devices and other IT equipment. The compute server portion is 63 percent, and out of that 31 percent is consumed by the CPUs. If we apply the same 20 percent CPU utilization ratio from IBM, the end result is 1.6 percent, still within the ballpark.
Where does this analysis leave us in terms of actions we can take as part of a first order strategy? The data above is hierarchical, and hence a pyramid is a useful way to organize it:
Changes toward the top, namely in the CPU application workload will have a minuscule impact power consumption for the data center as a whole, yet they can have a dramatic impact in the data center efficiency, that is in the amount of useful computations done as defined above. These changes can take place in two ways.
First, due Moore's Law, a server refresh can potentially double the per output CPU if the servers are two years old, or more than quadruple it if the servers are four years old.
Second, a consolidation or virtualization exercise can address the low utilization numbers for CPUs from less than 20 percent to 60 to 80 percent. Higher numbers are possible, but it is desirable to reserve some headroom to make the servers more responsive to workload peaks. These benefits are attained through the deployment of software technology from VMware, Xen, or, more recently, Microsoft Hyper-V technology that comes with the newly released Microsoft Windows Server 2008, formerly code named Longhorn. A Microsoft white paper, Windows Server 2008 Power Savings reports up to 10X linear power savings in a study with Hyper-V. Results may vary. A basic assumption is that utilization factors are low to start with. Workloads that take multiple servers or workloads that need a server cluster to run, such as large database applications or mail servers might not see such large benefit if the utilization factor is initially high. However, that also means that that the CPU utilization efficiency was high to start with, so there is less room for improvement.
Near the bottom of the pyramid we are talking real megawatts. In many cases the low hanging fruit comes not from from pulling all the stops with technology, but from plain energy conservation. A homeowner intent on lowering electricity bills should not rush to install solar cells. The first step is to conduct an energy audit to identify areas of greatest impact. A data center is no different. In an engagement I was involved with, a team was investigating whether 300 servers could be landed in an aging 25,000 square foot data center without hot spots developing. The energy audit using thermal modeling tools indicated that the data center could actually support a whooping 1,800 additional servers with very minor changes, essentially plugging air leaks in the floor tiles and repositioning a few rows to define hot and cold aisles. Of course, these results must be taken with caution, becase supporting the extra servers would probably have required a power feed upgrade.
So far we have analyzed possible actions that can be taken at the top an at the bottom of the pyramid. What happens in the middle? This is a more complex question and requires the inclusion of process factors. Furthermore, a specific answer always requires a context. Below is a case study presented by Gregg Wyant and James Chen at the recent Intel Developer Forum in San Francisco. Gregg is the Intel IT CTO, Chief Architect and General Manager; James Chen is the Director for Engineering Computing. In this case study, a server refresh was conducted over 4-year old servers. The application requirements did not change, yet running the application in the newer servers allowed reducing the number of machines from 126 to 17. The potential payoff from Moore's Law is a bit over 4X, yet the actual power draw reduction was 8X. The rest comes from application optimization and IT process improvement, a tribute to the Intel IT engineers carrying the application migration.
The reduction from six cabinets to one actually understates the gain. If the cabinet is populated with 1U servers, it will be only half full. The energy density per cabinet however will have gone up. These cabinets need to be housed in a data center designed to handle higher power densities.
