Prior to the Intel Xeon X5500 Server Platforms*, measuring server power was done via expensive equipment and could only be performed in a discrete fashion.  Unless you had tons of monitoring equipment to mash-up your power data - it was a tedious process.  Now, using Intel DCM and Node Manager - you can pull multiple servers worth of power info to make some important power decisions in your datacenter.

First of all, you need to baseline your workload.  If you're confident that you can replicate workload patterns then you've got a starting point.  Otherwise, it's usually a good idea to start monitoring and looking for some cyclical patterns and/or common data points (time, power, thermals, etc) to keep track of.

In this scenario (like in my last blog) we're using a SQL workload which can be modified to run the CPU at high levels for a relatively set amount of time.  The base workload runs for 7 min 30 seconds, as shown in the Intel DCM screencap below.

In this test case: Idle power for the 4 servers is 782W, and under load - the power increases to 1174W - which is a delta of 392W.  This power increase occurs when work is given to the server and the P/T states react to the workload and increase power/voltage to the system to increase performance.  Exactly what we've been used to seeing even since EIST was introduced several years ago.

Now, what I'll show you is something that may be very interesting in scale... I will power cap the servers by 20W each, and set the Intel DCM Power Policy to only allow 1095W for the 4 servers in the rack.

What is awesome here is that we can still finish the workload in the same 7 minutes 30 seconds.  So essentially, we have saved 80W of power for each set of 4 servers and still get the same amount of work completed!  In a large datacenter this can be HUGE in energy savings.

Let's do some quick math:  20W power savings per serer x 10,000 servers = 20kW power savings and you still get the work done.  I hope I just helped some of you server admins get some new ideas on your next "I need a raise" talk with your manager

*your mileage may vary, so test your own workloads and report out!

Do you ever wonder where Spam comes from?  I have no idea where the meat-like version of Spam comes from (nor do I wish to ponder that mystery). But it is pretty well established that a huge component of the e-mail and IM Spam that we all know and hate is generated by automated programs (bots) installed on thousands or even millions of unsuspecting systems.  These bots are remotely controlled via command-and-control or even peer-to-peer networks (botnets) to do the bidding of the bot developer—such as propagate Spam or other malicious software or generate denial of service attacks against designated targets.  And all of this could happen without most people even knowing their system is doing anything. 

Botnets are the end result of many malware exploits—as viruses, worms, Trojans, drive-by or click-through attacks may deliver and propagate the bot payload. They are also a crystal clear example of how the objective of attacks have changed from hit-and-run high-profile grabs for fame to instead focus on stealth and establishing and retaining control of assets. Botnets are an ideal tool for the nefarious—they can command huge numbers of widely distributed systems at trivial costs.  While it is hard to estimate how many systems are part of a botnet, the potential is staggering.  For example, the much-publicized Conficker worm is estimated* to have placed more than 4 million unique IP addresses under the control of “bot-masters”. And this huge resource base allows the bot-masters to rent control of these resources to spammers or other agents looking for ways to generate attacks or other nuisances with low risk of being detected.  In essence, they are allowing criminals and spammers to outsource the generation of their malicious activities. It is a frightening business model indeed.

It is also a difficult challenge for IT. Thanks to botnets, it is possible for an IT manager or CIO to get a call from out of the blue asking why their systems are attacking some other company or government entity’s systems.  Or discover a botnets of 100’s of computers with their company.  These type of events can happen to the best IT departments (even Intel or the US Government). Clearly, IT needs tools to help prevent such scenarios, and the antivirus and intrusion detection/prevention industry is working hard to keep up with the rapid growth in the delivery vehicles for bot code.  The other weapon for IT managers is traffic analysis – looking for strange patterns of activity (such as bursts of e-mail traffic from selected systems or floods of network traffic generated against specific targets) that falls outside of business norms to determine if there is another business being conducted with their assets.  While being part of a networked world has wonderful, powerful benefits, it is not without enhanced risk. A botnet is not a network you ever want a member of.

Intel technologies like Trusted Execution Technology (TXT) and instruction set optimizations such as STTNI can be part of these solutions.  Intel® TXT can be used in solutions that help protect systems from software attacks which provide the malware payloads to compromise systems.  In fact, Intel TXT (to be available with Westmere server systems) provides an entirely new protection capability for most systems—providing evaluation of the launch environment and enforcing “known good” code execution. This is important because most malware tools execute only once the system is booted—so Intel TXT provides a valuable complementary protection. And to help with the growing burden of run-time malware and attack analysis, new (with Nehalem) instructions that accelerate string manipulation can boost content inspection software ability to detect anomalies.  And research and development will ensure Intel continues to develop and deploy building blocks to help IT address today’s challenges and tomorrow’s.

We can do that most effectively only if we’re trying to solve the right problems.  Are your systems under attack? (yes, they are). What types of solutions are most effective for you?  Where is the greatest exposure? Is the pain in stopping attacks or cleaning up after them? This is certainly worth thinking about—before some Government agency comes calling asking why your systems are sending them so much spam!

*http://www.confickerworkinggroup.org/wiki/pmwiki.php/ANY/InfectionTracking

     As I’m new to The Server Room, I offer this brief introduction:  I am a marketing manager in Intel’s Software and Services Group – looking after Intel’s collaborative marketing efforts with virtualization solution providers.

     A couple weeks ago, Ken Lloyd blogged about the incredible changes in compute capability and performance brought by the Nehalem microarchitecture – and gave credit to the advances in software, too.  I’d like to take the conversation a step further:  did you know that the launch of VMware™ vSphere 4.0 in April 2009 represented a milestone of collaborative development?  The combination of VMware vSphere and Intel Xeon processor 5500 based systems delivers astonishing performance in part because it is the result of a full cycle of collaboration.

     Intel has a well established rhythm of technology innovation – and a lot of really smart architects who know a thing or two about cpu design – but we get innovative ideas from the outside, too.  Over the years of the VMware alliance, Intel has received (and acted on) many requests for small changes in cpu circuitry…changes that would make virtualizing the cpu easier, more efficient, or add capability.  A whole raft of hardware optimizations for virtualization were included in the Nehalem architecture.  As Intel started to deliver early silicon for Xeon 5500 based platforms, Intel software engineers worked closely with VMware engineers – optimizing vSphere code to take advantage of the new hardware features to improve performance, increase efficiency, and add new functionality.  The results?  Check out this video from the launch of VMware™ vSphere 4.0 to see for yourself what “better together” really means.  And the cycle continues – what can you imagine in the next round of collaborative innovation??

There’s a video going around from one of Intel’s top external customers.  Before you see this (video linked below) I wanted to position this correctly.  I caught up with Mr. X at an undisclosed coffee shop and got his approval to share publicly the messages that we would have rather had him go out with. Those messages are as follows:

Mr. X’s 4 year old servers were a burden on his organization, he spent all of his budget on just maintenance, nothing left for innovation.

He looked at his old infrastructure and determined that replacing them with more powerful-energy efficient servers from Intel was a strategic investment.

The New intel Xeon 5500 based servers provided the opportunity for him to innovate again.  He claimed that these new Intel Xeon Processor 5500 (Nehalem-EP) are the best enabler of IT business value that he's seen in years.

They boosted energy efficiency, saved him big $ and extended his facility lifespan – now he doesn’t have to go build a new data center. 

He replaced his old servers in a 9:1 ratio (getting rid of 9 old and replacing with 1 new) that enabled him to cut operational expenditures by 90% …And that savings alone is paying for the investment in these new servers in just 8 months. 

By strategically investing in IT when his competitors hunkered down and cut spending – he is now positioned to grow faster and gain share as the economic upturn arrives.

Ok, now that I’ve had a chance to convey his real messages, you can check out this video.

These are dog years for servers.   Pretty much every year Intel introduces a new Xeon processor.  Those who have heard the story recognize this as the Tic Tock model.  On Tic years the manufacturing process is updated, on Tock years the chip architecture is updated.  Every year customers get a boost in performance, and often a cut in power.  Typically this boost is in the 50% neighborhood, enough to make it worth the upgrade, and still achievable by engineering teams on a two year cycle.  Except, we are in dog years.

The Nehalem – Xeon 5500 – processor broke all prior boundaries on single generation performance gain.  Delivering two to three times the compute capacity of the Xeon 5400 (Harpertown) generation.  This is a big change, probably a once in a lifetime change – unless that quantum thing happens in my lifetime.  Roughly a 10X performance boost in less than 5 years.

During this same five years we have seen virtualization technology go from a lab project – something for test and dev – to mainstream data center process.  In 2005 it would have been heresy to suggest virtualizing the corporate ERP.  At that point virtualization overhead on the server could be as high as 25% and the entire server was needed to do “real work”.  Fast forward to today.  Virtualization technology in both the hypervisor and processor have reduced overhead to only a few percent, AND servers are 10X faster.  Not only can you virtualize the ERP, you are irresponsibly wasting resources if you do not.  Unless your ERP demands have grown 10X in 5 years, your ERP alone won’t even make a new Xeon 5500 system sweat.

If this advancement wasn’t enough, the announcements last month from Intel about the coming Xeon 7500 (4+ socket) processor were amazing.  All the benefits of the Xeon 5500, but on steroids.  The  new biggest leap ever.  With up to eight cores and four memory channels per socket, this is a monster.  Your ERP system will be barely a blip in perfmon.  It isn’t unreasonable that an entire data center for a SMB business could be virtualized onto one of these beasts.  And, how big is a Xeon 7500 server?  My guess is about the size of a breadbox…

With the Intel Xeon 5500 series (Nehalem) based processors, the X5500 chipset and instrumented power supplies, you can start with the most basic use case for Intel Node Manager - monitoring the power usage of your servers.

As you can see in the Intel Datacenter Manager (DCM) screen below - there are multiple servers configured into logical units:  HF2-EIL is the lab that these servers are located in.  Rack 1 and Rack 2 are the physical location of these servers, and each Rack contains 2 servers each.

When you highlight one server (as above in DCM)- you can see the power characteristics over a certain time period.  The time period shown gives you the idle power, max power, and thermal measurement.  The 'hump' in the graph is a SQL workload which creates 'work' for the server and the process runs for about 5 1/2 minutes with no power capping.

Here's a graph of the 2nd server in that rack, performing a similar workload.  As you can see, the 2nd server power usage is different than the first.

The Intel Datacenter Manager SDK console can monitor multiple systems as well.  The next graph, is both of those servers in the rack, which accounts for both servers power usage during the same timeframe.

Finally, here is the final graph, showing the accumulation of all 4 servers, in both Rack #1 and Rack #2.  This shows the maximum power utilized during the workload, the minimum power (idle) and the inlet thermal temperature in the lab.  Something that hasn't been able to be done before without expensive equipment in the datacenter.

My next power based blog will show how power-capping can give you more effience use of your workload power while using Xeon 5500 series platforms.

Nehalem-EX has been in the news quite a bit over the past several months. 

First, in May, Intel described how Nehalem-EX will be at the heart of the next generation of intelligent and expandable high-end Intel server platforms, delivering a number of new technical advancements (Intel Nehalem Architecture, Quick Path Interconnects, 16 threads, 24MB cache, new RAS features like MCA-Recovery, 16 DIMM slots per socket, 128 threads on 8 Socket systems) and boost enterprise computing performance (the greatest gain in generational performance ever seen at Intel.)

Next at IDF in September Intel described how Nehalem-EX would deliver a bigger generational performance improvement than that delivered by the Intel Xeon 5500 processor (including a 3X Nehalem-EX gain in database performance); a large shift in Xeon scalability with over 15 >8S systems anticipated and expandability for the most data demanding enterprise applications, the addition of about 20 RAS capabilities traditionally found in the Intel® Itanium processor family – along with a demonstration of MCA-Recovery. IBM announced their upcoming BladeCenter products that will support 4S Nehalem-EX blades and Super-Micro announced a 1U box, specifically targeted at HPC.  Staying on the HPC theme, Mark Seager from the Lawrence Livermore National Laboratory was also quoted with stating that “Nehalem-EX allows us to invest in science, not the computer science of porting and adapting software to new architectures, but real science.  Nehalem EX is an innovative SMP on a chip solution that provides us access to a “super node” … The result is an astonishing new level of performance.”

And Oracle Open World on October 13^th, the drumbeat for Nehalem-EX continued.  Michael Dell in his Oracle Open World Keynote today discussed how Nehalem-EX will provide a true leap in performance, with up to 9x the memory bandwidth and 3x the database performance vs. prior generation.  And he mentioned that Dell’s unique implementation of the memory architecture will allow the most cost effective scaling, with 4S systems up to 1TB of DRAM (64 Dimms x 16GB Memory sticks) enabling customers to run their entire database in system memory.  He also mentioned that standard based systems are driving new efficiencies with applications like Oracle, where Dell’s data shows Oracle apps run better on x86 vs. proprietary architectures, up to 200% better.  Check out this short video from the keynote and watch what Michael Dell had to say. 

Keep your eyes on the Server Room for more Nehalem-EX news as it comes between now and launch.  And visit the Intel booth at South Moscone Booth #1621 to learn more.

Bryce

Software scalability has been a big issue recently.  While modern servers are incredibly fast, many software solutions simply are not able to take advantage of it.  There are many reasons for this.  Some are easy to address and some require changes to the software.  Intel performance engineer and Oracle WebLogic performance engineer will jointly give a talk at Oracle OpenWorld on this topic.

Here is the session information:

ID#: S309892

Title: Ten Ways to Improve J2EE Application Performance on Multi-Core Systems

Track: Oracle Develop: Enterprise Java and Oracle WebLogic

Date: 13-OCT-09

Time: 17:30 - 18:30

Venue: Hilton Hotel, Room: Yosemite B

Here is the abstract:

The current economic environment and the new focus on being green demand greater efficiency from every IT shop, big and small alike. In this session, you will learn how to improve Java application scalability by using Oracle WebLogic Server on the latest multi-core systems. It examines various software and hardware features for getting the best performance out of your applications. In particular, it explores the pros and cons of 32-bit versus 64-bit environments and how having multiple Java virtual machine instances can reduce heap pressure and improve cache locality. It also discusses operating system and hardware features such as large pages and solid-state drives and their impacts on J2EE application performance.

As a bonus, we will be giving away a Netbook at the end of the talk.

We also wrote a technical paper on the topics that will be covered in this talk.  You can find this technical paper at http://software.intel.com/en-us/articles/java-application-server-optimization-for-multi-core-systems/ .