The Server Room Blog

A 45nm 6-core QnA

C_Peters — Fri, 03 Oct 2008 23:29:14 GMT

Following my earlier blog, I promised to share answers to some of the more common questions I get from customers on 45nm and mostly about the newest product we have on 6-core 45nm: the Xeon processor 7400 series.

1. What does 45nm really mean? A nanometer represents a distance that is one billionth of a meter in length. 45nm represents the width of a single transistor and is used to describe the manufacturing technology Intel uses to create our latest generation of processors. Because of the small 45nm transistor size, Intel is able fit 2 million transistors on the period at the end of this sentence.

2. Are all 45nm transistors the same? No. Materials used in silicon manufacturing process can vary from manufacturer to manufacturer. Intel switched over to a high-k dielectric material (Halfnium) that helps dramatically reduce leakage current – improving the performance/watt characteristic of our processors.

3. What OEM products feature 6-core 45nm products? Servers based on the processor are expected to be announced from over 50 system manufacturers around the world, including four-socket rack servers from Dell, Fujitsu, Fujitsu-Siemens, Hitachi, HP, IBM, NEC, Sun, Supermicro and Unisys. There are four-socket blade servers from Egenera, HP, Sun and NEC and there are server designs that scale up to 16-sockets from IBM, NEC and Unisys.

4. How does 6-core affect my software licensing? Just like with other multi-core processors, licensing will depend on the software vendor. With quad-core most ISVs elected to license by socket or processor meaning that the performance enhancements came “for free” as the number of cores are increased. Recently VMware updated their definition of a “processor” to include up to 6-cores per processor (learn more) meaning that with VMware ESX 3.5 update 2 and Intel Xeon processor 7400 series, IT can deploy a higher density of virtual machines per server without an incremental increase in licensing costs. Everyone does it differently – so do your homework.

Other common questions circle around IT usage trends and how this technology can really be applied. Here is an interesting (and somewhat long) video where Intel VP and CIO Diane Bryant discusses with executives from Yahoo, Oracle, MySpace and Verisign about the challenges they face and how technology is helping them. If you choose to listen you will find answers to questions (paraphrased) like?

What are some of the top challenges IT faces today? How can technology help?
Is 6 core performance too much? Does IT have the ability inside their environment to take advantage of this additional compute capacity?
Is the software ecosystem is ready for multi-core? Can today's applications take advantage of it?
How are customers using Virtualization today and how do they see it changing over time?
When virtualizing ... how does IT view MP servers (4 socket) vs DP (2 socket)?
When deploying next generation technology, how important is the power capacity of the IT environment when selecting technology?
Are Intel Xeon servers powerful and reliable enough to consider moving away from RISC or other proprietary architectures?

If I missed your burning question, just ask … I’d be happy to share. Chris

I/O Virtualization - Round 2!

BenHacker — Fri, 03 Oct 2008 21:55:17 GMT

In my last I/O Virtualization blog, earlier this year, I discussed a fundamental problem with virtualizing I/O and one the solution that Intel and VMware have teamed up to deliver - VMDq and VMware NetQueue. These queuing technologies together can help to offload some of the virtual switching (vswitch) functionality to the network adapter from the hypervisor. VMDq provides a method for the Hypervisor to do less work, and also provides a way to share the I/O processing across multiple cores; improving system bandwidth and more fully utilizing its processing power.

Now, VMDq and NetQueue are a great solution together that scale well, support Vmotion, and are relatively simple to manage. However, is there a way to get even better performance from your Virtualized I/O?

What if there was a way to completely cut the Hypervisor software switch out of the picture and remove the associated latency and CPU overhead? The ideal scenario for optimum performance is for the VM to communicate directly with LAN hardware itself, and bypass the vswitch completely. For example, you could have a single 10 Gigabit port expose multiple LAN interfaces at the hardware level (on the PCI-e bus), and each VM could be assigned directly to a hardware interface. Alternatively, you have multiple physical NICs in the system that could be directly assigned to a given VM. Below is a diagram that summarizes the 3 main variations of attachment for I/O in a virtualized server. Below we will get into more detail to put the diagram in context.

In the diagram above, the left side represents an implementation of a virtualized environment with a standard I/O setup using the Hypervisor vswitch and VMDq for I/O performance enhancement. In the middle is an example of direct I/O assignment between a single physical LAN interface and a single Virtual Machine. The implementation on the right is showing what is possible with a single NIC that supports SR-IOV (we'll discuss this later) for a fuller, hardware level I/O virtualization. After taking a moment to understand the basic differences in these three implementations, there are immediately a few obvious benefits here for bypassing the Hypervisor vswitch and going with either of the two directly assigned designs...

By allowing the Virtual Machines to talk directly to the networking hardware, throughput, latency, and CPU utilization of the I/O traffic processing will be greatly improved. So the question is, "why hasn't this been done before?" Well, the answer is that there are several gotchas to make this implementation work well...

First, in order to implement this properly, the LAN hardware needs to support some physical capabilities to successfully route the networking traffic in this kind of virtualized system. In addition to all of the above the actual server hardware itself must also support VT-d so that the memory mapping between the Virtual Machine PCI-e memory space and the systems physical memory space are correlated correctly. Also, the actual system itself must also support VT-d so that the memory mapping between the Virtual Machine (I/O data memory address) and the systems physical memory address are correlated correctly.

Finally, and this is a big one, this kind of implementation while very good for performance just happens to break the ability to move a VM from one physical server to another (VMware Vmotion). This is one of the more widely used aspects of VMware's software that has been utilized heavily by most IT shops. Seamless vmotion support is critical for making any I/O performance improvement deployable in the real world.

Now, if you stop at the 2^nd^ diagram, and use separate NICs for each VM, you will also miss out on a few key advantages of new Ethernet capabilities. You won't be able to allocate your overall bandwidth between your VMs (each VM will get a single Gig or 10Gig port), and more importantly, you won't be able to effectively share higher bandwidth pipes. For example, a server with a few 10 Gigabit ports may have enough I/O horse power to handle traffic for 30 VMs, but there would be no way to assign only a portion of the bandwidth of the pipe to an individual VM.

Additionally, the LAN hardware needs to support the ability for each virtual function of the LAN device to be able to support bandwidth segregation (think QoS per VM) and the ability to support multiple queues and traffic classes per LAN virtual function. This last piece is necessary for those who remember the discussion on Fiber Channel over Ethernet (FCoE), as the ability to support multiple traffic classes, and dedicated bandwidth links, are key needs for the storage over Ethernet market.

Now that I've set up what is needed to make this directly assigned virtualized I/O environment work, and called out the potential problems, you don't need to worry; I won't throw cold water on this idea. In fact, most of the pieces are in place today and there is already work being done to complete the solution as we speak.

First, Intel network adapters now support some fancy hardware capabilities related to virtualization. In addition to all the hooks for VMDq, our newest NICs support PCI-SIG SR-IOV (I know... technologist love acronyms) which provides the ability to virtualize the LAN at the lowest hardware level. The networking hardware also supports some smart logic to be able to function properly in a virtualized system. For example, VM to VM communication in the same server must be looped back before it gets to the wire or the switch connected to the machine won't know how to route the packet. This is all taken care of in the LAN hardware. And of course, all the support for bandwidth segregation, and support for multiple queues and traffic classes is there as well to make sure Storage and other QoS sensitive applications are still going to work well.

As for VT-d support, Intel platforms now come with this basically standard, so there is no issue there. But the last most important piece is the ability for an individual VM to be moved between physical servers while still being able to ‘renegotiate' with its physical network connection. The ability to do this is under development by Intel, VMware and others in the industry, and the end goal is to have an architectural framework in place to make this kind of handoff seamless from a hardware and software perspective.

This architectural framework will be the topic of a future post, as I think I've used up all the lines I can before I start putting my readers to sleep. Until next time!

Ben Hacker

Anand says there is more to Dunnington than Cores

K_Lloyd — Tue, 30 Sep 2008 19:32:26 GMT

Saying more good things about Dunnington ( Intel Xeon 7400 ) feels a bit like piling on. There are a myriad of posts out there about how great Dunnington is. If you are looking for some data to support enterprise selection of the 7400, the article in Anand Tech Intel Xeon 7460: Six Cores to Bulldoze Opteron is very compelling. One of the exciting parts of this article is in the section on ESX performance, especially with vm's configured with multiple "virtual cpu's". This is a configuration some of my large enterprise customers seem married to - even when not needed... The 7400's use of highly efficient 45nm penryn cores delivers the dominant performance for this usage model. There is a lot more to this processor than "2 more cores".

To quote from the article "This 45nm Intel core features slightly improved integer performance but also significantly improved "VM to Hypervisor" switching time. On top of that, synchronization between CPUs is a lot faster in the X74xx series thanks to the large inclusive L3 cache that acts as filter. Memory latency is probably great too, as the VMs are probably running entirely in the L2 and L3 caches. That is the most likely reason why we see the X7460 outperform all other CPUs."
The ESX section concludes with "Xeon X7460 is again the winner here: it can consolidate more servers at a given performance point than the rest of the pack"

Xeon 7400 is the processor for virtualization.

Have you got the "VIBE"?

whlea — Mon, 29 Sep 2008 22:00:03 GMT

Check out this video with Robert Harley of Virtual Blocks and Chris Parisi with HP Canada talking about a new virtualization solution for SMB's.

VMWorld Forecast: Partly "Cloudy" with a chance for change

J_Smith — Thu, 18 Sep 2008 21:49:19 GMT

Each year for the last 10 years, the innovators of VMWare, have hosted a users and partner conference to discuss virtualization technologies, ideas and services for the IT industry. This years event, in Las Vegas, brought together over 14,000 of the world's foremost thought leaders, developers and users from around the world. As the "Virtualization World" converged on Las Vegas their was a prevailing forecast that has begun to permeate our virtualization landscape: Cloud Computing. Paul Maritz, in his initial keynote address as CEO of VMWare, outlined the importance Cloud computing and the role that VMWare and their customers will play in defining the Enterprise Computing "forecast" over the next several years. It was a thoughtful direction for the world's leading innovator in virtualization software technology. I personally found it rather gratifying to see Mr. Maritz thoughtful demeanor and acknowledgement of the VMWare Co-Founders Diane Greene and Mendel Rosenblum, role in shaping this new direction. His understated prose also failed to acknowledge the role he himself has played over the years in establishing this direction.....it also clearly placed in my mind why he may be the ideal leader to help us realize the forecast for cloud-based compute models.

So what does it all mean? Cloudy forecasts are always difficult to predict and predictions can become self-fulfilling prophecies or embarassing missteps. What is clear, in my opinion, is that Cloud computing will drive meaningful change across a wide range of industries in rapid succession.

Let me explain the logic: Organizing and managing compute, network and application usage models has been a very elusive endeavor for many years. IT departments cannot always predict application load, network requirements and storage availability. If you provision for the worst (or highest use) case scenario you often over build. In other cases, application popularity or changing business conditions create under capacity and infrastructure failure. Those of us who have launched Application Service Provisioning infrastructures bear the scars of failures, excitement of success and hope for the future. VMWare, Microsoft, EMC, Google, Amazon and many others have made a concerted effort to "get it right" this time. Cloud infrastructures using virtualization technologies are providing a opportunistic ways for developers and end users to test scalability theories of traditional client/server compute models. These same "Clouds" are providing internal cost reduced resource infrastructures to make available vast computing, network and application resources for everyday usage with relatively low entry points (a la Amazon's EC2). However, determining which part of the "Cloud" to make available for public vs. internal consumption will be defined by innovative new technologies that have yet to be announced. Interoperability, compatibility, performance and scalability are all design points which the industry must consider.

Visionaries in this space abound: Vin Cerf (deserves more credit than he is given), Ray Ozzie, Reuven Cohen (you may not of heard of him yet), Alan Gin, Marc Benioff, Ed Bugnion, K.B. Chandrasekhar, Pete Manca and many others have been working diligently for years behind the scenes to make the promise of Cloud computing real. Industries such as Big Pharma, Telecom, Financial Services and Oil & Gas will reap tremendous benefit from well defined industry "clouds". The role of ethernet will be a critical design point for these next generation infrastructures as 10Gbe+ reduces latency, response times and delivers application QoS. At Intel, we are very proud of our engineering and process manufacturing prowess for the development of multi-core compute technologies, rightfully so in my opinion, but the future of the "Cloud" will challenge us to re-examine our design methodology, increase our price-performance-per watt cadence and deliver exciting new innovations throughout our server/client platforms.

Virtualization innovation has provided a "sliver lining" for today's Cloud infrastructures. Where there is transitions or inflection points in the technology industry, there is opportunity. At VMWorld 2008, the virtualization industry has begun the process of delivering technologies in a world beyond the hypervisor. Virtualization 2.0 as outlined by Doug Fisher, Intel VP of Software and Solutions Group and Steve Herrod, CTO of VMWare is a step towards providing the innovation required to make Cloud infrastructures real. The next steps, the new pioneers ( a la Simon Crosby of Citrix) are building tools which provide increased ROI in decreased cycle times for IT managers. The future of the IT cloud is in their capable hands and in the hands of the IT innovators within each company focused on providing compute infrastructures designed to scale (and shrink) with the businesses we serve. VMWorld has yet to disappoint, in 2008, VMWorld reminds us that even on a "Cloudy" day there is a chance for change.

Here's a short video talking to Dave Martin of VMware around VT Flex Migration....

“VMWorld08: Intel Keynote-Collaboration for Leading Virtualization Deployments”

whlea — Thu, 18 Sep 2008 18:28:49 GMT

More news from VMWorld 2008, Las Vegas. Doug Fisher, Intel V.P. gave a keynote during the VMWorld conference. One of the more interesting elements brought Steve Herrod, Sr. V.P. and CTO of VMware on stage to talk about how Intel and VMware are collaborating to deliver leading Virtualization Deployments. Click on the video to see what they have to say.....

Which Server is Best for Virtualization?

K_Lloyd — Tue, 16 Sep 2008 17:26:40 GMT

Virtualization is the big thing, everybody is doing it - just read the in-flight magazine to see why you should be virtualizing your data center... While it is true that Virtually everyone in the fortune 500 has begun to virtualize their data center, it is also true that most servers are still not virtualized.
i.e. The data center landscape is still mostly an opportunity. The software is mature, there are multiple viable solutions, but there are still many questions about how "best" to proceed.

As an enterprise engineer working with enterprise customers, I am inevitably asked where the sweet spot is. The reality is, there isn't one. Or "It Depends". In general larger ( 4 socket servers) provide an edge in efficiency as there are more shared components - board, memory, power supplies, etc. Large servers can also provide more head room if most of your VMs are low utilization, but any of them can spike way up. The launch of Intel's six core Xeon 7400 series based servers ( and their record breaking virtualization performance) have added to the interest - is it time to go big?
What does it depend on?

How big are your VMs? Machines today are quite powerful. We have seen a 10X growth in compute capacity in just the last 6 years. The application that filled 37% of your 2003 vintage server won't even make a dent in a modern Xeon based server. i.e. Most VMs are much smaller than your server 2 socket or 4 socket. There are still tasks - like decision support that scale as big as your machine will go, but with average enterprise utilization down around 12% ( on old hardware) most physical machines fit tidily inside a VM.
How spiky are your VMs ( in resource demand - compute, memory, network)? By doing some resource profiling, you can understand where your servers fit best.
How many VMs do you want on each PM(physical machine)? You can put more on 4 socket hardware ( efficiency) but have greater redundancy on a bunch of 2 socket hardware ( depth).

Fortunately you do not have to solve this linear programming problem before you start. In reality the tools are making it easier you solve. Using your favorite VMM manager (choosing this is another discussion). With Intel's VT Flex Migration Technology you can pool together 1, 2, 4 socket current and future generation Xeon platforms and move the workloads ( automatically, or manually) to optimize your resource utilization.

VMworld - Intel's online booth - check it out.

Server.Dude — Thu, 11 Sep 2008 21:57:43 GMT

For the event next week @ VMWorld we have created a virtual booth, check it out here: Intel Virtual Booth

Also follow Hank Lea as he explores vmworld and blogs about the event.

Intel & Turtle Entertainment Collaborate on Virtualization of Gameservers

whlea — Thu, 21 Aug 2008 21:10:56 GMT

IDF SF08-Online gaming and sports leagues are growing every day and here at IDF this week we had the opportunity to see how Intel is making an impact. I was visiting the Virtualization Community in the IDF Showcase where I met Bjoern Metzdorf, Director of Information Technology at Turtle Entertainment who was speaking with Alan Bumgarner of Intel. Check out the video for a major success story including an 18:1 server consolidation ratio, 85-90% power savings and no observable latency for the gamer, this is cool stuff!

If you want to learn more about Turtle Entertainment and the Electronic Sports League (ESL) Click Me

Big Servers are Back!

bryceolson — Wed, 28 May 2008 20:05:23 GMT

One trend that is really starting to take shape in the server industry is that big servers are back! That doesn't mean big servers ever disappeared off the map. Historically bigger servers with 4 or more processor sockets have been 7-8% of the server market from a volume perspective. And bigger servers have always been used for scalable, data-demanding enterprise applications which IT values for it's performance, headroom and reliability. What we're seeing now is a greater shift in popularity towards these servers as IT invests more and more in this direction.

So, why is that? Well, check out this video and then let me know if you agree or disagree. After you watch it I'd also be curious to learn more about what you value as the most important buying criteria when you go big.

Intel & VMware-World's First Flex Migration Demo

whlea — Wed, 23 Apr 2008 18:19:48 GMT

After coming back from IDF a couple weeks ago, I've had some time to go through the mountains of online material, presentations mostly and a few interesting videos. This video is from Pat Gelsinger's keynote address and features Mendel Rosenblum from VMware. Pat and Mendel discuss new technologies in virtualization and demonstrate "Flex Migration", just hit the play button below to view...

This is very interesting for those IT shops with multiple legacy platforms and new generation servers coming online. We will have more discussion on this topic in the future, and so in the meantime, let us know if you have questions on how this could benefit your datacenter.