I wanted to follow-up on the pre-IDF blog I wrote and what I and Sean conveyed regarding comprehensive IO optimization for enterprise cloud (based on virtualization infrastructure). A blog I owed to those who could not attend the IDF session.

In the last blog we identified 4 important vectors that drive I/O evolution.

1)      Balanced system that maps to the increases in CPU performance

2)      Scalability

3)      Unified fabric

4)      Security

In my view I state it an evolution as I feel that is the natural state things will head towards in the (near) future.

In a cloud environment you would anticipate automation and policies determine the consolidation possible on a system. If SSDs get broader adoption and virtualization performance increases due to hardware assists, I/O and fabric could become the bottleneck for the degree of consolidation and efficiency as it cannot map to the increased data rates from the storage and CPU performance.

Ways to address this is either to reduce or eliminate the overheads in the I/O stack caused by software emulation of devices in the VMM. VMDq is an example of a technology that can reduce the overhead or offload some of the VMM tasks through hardware assists in the NIC. Direct assignment with PCI-SIG SR-IOV support is a way to eliminate the overheads by by-passing the VMM. With SR-IOV, a single device can be divided into many logical devices known as Virtual Functions (like a pair or independent transmit receive queue). Each virtual function can be directly assigned to a VM using Intel VT-d thereby bypassing the VMM. This can work with Live VM Migration too. At IDF we showcased 4 demos of prototype SR-IOV software solutions running on Intel Xeon 5500 based hardware with prominent VMM vendors like VMware, Citrix and Redhat that have different hypervisor technology. The networking demos showcased working live migration with SR-IOV and VT-d based direct assignment. Direct assigned VMs could be even moved to an emulated mode and brought back to direct assigned mode. Intel has not only been working with software providers but also with other hardware vendors like LSI to demonstrate this capability. These technologies are as important to storage as networking particularly as SSDs gather steam. You can learn more from these blogs below on the demos.

Demo with Intel Xeon 5500 based Dell servers

An analyst view on the LSI solution demonstrated

If those multi 1GbE cables (that make your fabric look like pasta) be replaced by 10GbE and if SR-IOV and VT-d be used for performance, then it answers both the I/O performance and scalability requirements for a flexible datacenter.

Beyond those VT-d provides greater protection by allowing I/O devices to access only the memory regions allocated to them, and SR-IOV allows VMs to access only their portion of the device and restricts access to other Virtual Functions (owned by other VMs) on the I/O device or the entire device itself. Better security through better isolation.

Last but not the least of the requirements is the unified fabric. When IT can use a single I/O device for storage or for LAN traffic, the rigidity associated with provisioning of servers with some number of HBAs and some number of NICs is reduced. The I/O capacity becomes fungible and flexible. FCoE and iSCSI are key technologies in this direction. Adding capability to monitor QoS and shape the traffic makes it a good match for flexible datacenter.

Many of the technologies I discussed above (VT-d, SRIOV, FCoE, iSCSI) are here today… software such as Red Hat Enterprise Linux is already delivering the solution. In my view just a matter of time that ecosystem builds further and hardware is well tuned.

With these in perspective how do you see your datacenter shaping up?

I have written in the past about key IT considerations while implementing virtualization.

One of the key elements that change going from a non-virtualized environment to virtual environment is the security model. The security model needs some additional considerations going to virtual environment.

I and a few of my colleagues who meet with IT end customers deploying virtualization on a regular basis have realized that there are some frequently asked questions/concerns and also misconceptions about protection in virtualized environment.

We also did a bit of research on types of documents available to help IT understand the topic of security model in virtualized environment better, but found most articles to be either outright dismissive of security concerns or took a very opposite theoretical and conservative view on lack of security.

So with the help of our architects we developed the below white paper with an intent to help IT managers, strategists and implementers understand resource protection in virtualized environment better. We also address some of the frequently asked questions and typical misconceptions with security in virtual datacenter.

The white paper essentially takes a balanced view and provides an overview of security model changes, challenges and considerations that organizations must address when implementing virtualization. It introduces hardware, software, and policy measures available to help address those challenges, including their strengths and limitations and then closes with a brief discussion of some key issues associated with security in emerging cloud computing usage models.

Let us know what you feel.

Non-x86 RISC architectures, Power or SPARC, have been used in high end business critical virtualization solutions for a long while now. These come with a vertical stack of solution including the hardware, software, manageability tools and services provided by one vendor. This often leads to lock-in to the proprietary virtualization solution and services, and can be expensive from an end user perspective.

There are reasons why companies that can afford RISC based solutions have subscribed to it. This has been mainly due to Reliability, Availability and Serviceability (RAS) features, scalability and dedicated resources for quality of service (QoS) and isolation.

The world of virtualization however has significantly changed in the last 5 years. x86 based hardware and software products today offer well accepted and high performance virtualization solution. With the eminent availability of highly scalable and resilient Nehalem-EX products with 16-threads per socket and extensive RAS capabilities in the near future, the line between an expensive RISC solution and x86 based virtualization solution could blur further.

From an end user’s perspective, Nehalem-EX could provide sufficient capabilities that they have come to expect out of a RISC based virtualization infrastructure. Looking at it:

• Hardware partitioning of Nehalem-EX platform would be possible. Along with this OS virtualization and full commercial hypervisor support for logical partitioning already exists on Xeon processors.
• Nehalem-EX hardware infrastructure allows software ecosystem to deliver capacity on demand. For example extra CPU capacity can be dynamically added as needed. Moreover VM migration and policy based load balancing capabilities that already exist in commercial hypervisors complement this and provides IT easy methods to manage capacity at the datacenter level.
• Memory can be dedicated by not oversubscribing the available physical memory.
• CPUs can be dedicated by creating CPU affinity.
• Dedicated I/O assignment is possible using VT for Directed I/O. It can also restrict DMA access from devices to certain areas in memory, increasing isolation and system reliability.
• Single Root IO Virtualization feature would be available as part of Intel VT for connectivity in the networking devices. This allows a single NIC to be shared amongst multiple VMs directly, while isolating the traffic from a NIC queue to a VM for better reliability. Per VM bandwidth allocation can also be supported.
• Nehalem EX adds virtualization feature that could help increase VM performance in a processor oversubscribed environment with high system utilization.
• Nehalem-EX will add new reliability, availability and serviceability (RAS) such as Machine Check Architecture (MCA) Recovery that allows error detection, error recovery and VM isolation.
• Inherent power technologies in the CPU, Turbo mode, and Dynamic Power Node Manager for system wide power capping all deliver IT the essential keys to balance power and performance.

While Nehalem-EX measures up to the infrastructure needs, it also enables horizontal solution that would allow customers to take advantage of best of breed software from the virtualization ecosystem thus reducing lock-in. This could result in faster innovation leading to an array of choices for business critical virtualization.

Based on http://www.itjungle.com/tfh/tfh042808-story03.html, a Power virtualization solution with Power6 based 4 Socket P550 box (~$93,000) and PowerVM Enterprise Edition for large system ($1,969 per core, with $220 per year on the maintenance) will totally cost an enterprise $109,000, just in one server acquisition.

While pricing of NHM-EX 4S system is not available, approximating a cost using current 4-Socket Intel server pricing and commercial VMM software would suggest that Intel based solution could cost at-least 50% less in just infrastructure. Other savings like not requiring specialized RISC based hardware, services, solution and staff would add to the lower cost of ownership in the long run.

Given the economy and Nehalem-EX features, would it not make sense to take RISC out of your investment?

When it comes to implementing virtualization, the saying ‘if you fail to plan, you plan to fail’ is just simply true. Upfront and detailed planning for virtualization implementation in the datacenter can get the right results both financially and operationally.

In the document below, I explain a simple six phase approach to implementing virtualization with a lot of emphasis on how to plan for such implementation. These phases are generalized best known methods and are not an exact blue print to implement virtualization. While it is true that one size fits all approach will not work for virtualization implementation, knowing the basics will help guide through the process or ask the right questions if you are using a solutions provider.

http://communities.intel.com/docs/DOC-2355

-RK Hiremane

Intel's launch of Xeon 7400 processors this week marked yet another great product from Intel that simply delivers to basic virtualization infrastructure need of a datacenter. In my view, what sets Intel apart is the consistency with which Intel has been providing the hardware capabilities essential for virtualization adoption and acceleration. These hardware capabilities have delivered incremental power efficient performance for virtualization and platform wide solution that makes virtualization adoption efficient.

Just drawing a year back Xeon 7300 processors based platforms when launched set industry leading performance results for virtualization for 4 Socket mainstream servers. Now, Xeon 7400 series processor with six cores and built on energy efficient 45nm technology, provides the industry best performance for virtualization for 4 Socket mainstream servers. On VMware's VMmark, Xeon 7400 scaled up the performance (over best published Xeon 7300 score) by appx 35% (http://www.vmware.com/products/vmmark/results.html). On Hyper-v with vConsolidate virtualization benchmark Xeon 7400 delivered 40% better performance and 52% better performance per watt (over Xeon 7300) as published at http://www.principledtechnologies.com/Clients/Reports/Intel/vConHV2Sys0908.pdf. This performance trend is fairly similar to how 45nm Quad Core Xeon 5400 (launched Q4 '07) delivered up to 20% performance over Quad Core Xeon 5300 (launched Q4 '06) in 2 Socket space. The key to IT managers from my view point are not just these statistics in performance but also the ability to get these performance increments on a predictive cadence with in the same power envelope. Socket based virtualization software means better TCO as well.

In the same vein of performance I mentioned platform solutions for efficient deployment as the key element of these hardware capabilities. Now efficient deployments of virtualization and emerging usage models of virtualization require performance and some more... what I refer as capabilities. But why?. It requires some simple understanding.

New emerging usage models of virtualization beyond consolidation, referred as virtualization 2.0, like load balancing, high availability and disaster recovery (HA/DR) require resource pooling. Once these resource pools are architected within the datacenter the IT managers do not typically want to change them just because they want to add new generation of servers to the resource pool (and retire a few older ones). To support this requirement Intel delivered a new capability called Intel VT FlexMigration (Intel VT FlexMigration). With appropriate software support like Enhanced VMotion in VMware ESX 3.5 update 2, IT managers can simply roll in a Xeon 7400 processor based server with Core Microarchitecture based previous generation servers (like Xeon 5300, 5100, 7300 series processors) already in a resource pool.

Another requirement for efficiency in highly utilized servers as in the case of large consolidation or load balancing is robust and efficient networking solution that supports the increased processing capability. Load balancing and HA/DR usage model in particular rely on VM's moving over the network. Efficient networking solution means efficient virtualization 2.0 usage model deployment. Intel networking adapters that can be used on even the Xeon 7400 based servers has a feature known as VMDq, which can accelerate the networking performance. On a 10GbE NIC using ESX 3.5 update 1 software, VMDq delivered >2x the improvement in throughput, which means higher performance and also VMDq being a hardware assist reduces the VMM overhead relieving the CPU cycles for applications to run more than VMM. New Ethernet adapters also add QoS capabilities like bandwidth allocation that could provide even better control in terms of latency and traffic.

Finally the virtualization 2.0 usage models rely heavily on centralized storage. Becoz when VM is moved from one physical server to another server in the resource pool, if the entire resource pool had a ubiquitous view of the data a VM was using, then the transition and resuming of VM on any server in the pool would be fast and seamless. Hence cost effective centralized storage connectivity would be very desirable for these virtualization 2.0 usage models. Intel hence has been a leading force in working with industry standards to make Ethernet robust and developing Fiber Channel over Ethernet standards and products that can carry both SAN and LAN traffic on the same fabric.

Collectively, all these highlight how Intel is showcasing leadership in products that matter to both consolidation and usage models of virtualization 2.0 beyond consolidation.

A common question that most IT managers are faced with while they mull on virtualization is what kind of a system should I use? Do I need a 2 socket system? a 4 socket system?

Reality in my view, it is more like designer wear and there is no ‘one size fits all' generic answer for such question. There are few things (not exhaustive by any means) IT managers that are planning virtualization need to understand to come to that conclusion are

• How many servers are being consolidated?

• Workload and compute horsepower: What are the workloads being consolidated? What is the average utilization of the workload? What is the maximum utilization expected from the workload (so that you anticipate for the max and the datacenter capability does not fall apart if the workload utilization increase)? An overall look at the compute horsepower requirement to run the VM with the workload.

• Memory: How much memory is required per VM to run at the acceptable or required quality of service guarantees of performance?

• Manageability comfort and VM variation/headroom: The number of VMs the IT manager is comfortable putting on a same system either for ease of manageability, downtime managements, resource scaling if VMs get over utilized or over subscribed at peak demand, and/or also any intuitive comfort level of mixing different workloads or OS environments.

In my opinion, for high level of server consolidation with memory and I/O intensive workloads or VMs, less predictive workloads, and workloads that demand more headroom for peak demands, a 4 (or higher) socket system could be more beneficial. For consolidation to raise server utilization with very predictable and stable workload that may be smaller applications, a 2 socket system could be beneficial.

Feel free to write your opinion, or experience.

Some resources: http://www.intel.com/it/pdf/memory-sizing-for-server-virtualization.pdf