The virtualization that is the basis of all cloud computing is the death knell for any HPC application. The virtualization does two things that are counter to effective usage of clouds for HPC. Virtualization kills memory foot print control essential to HPC applications. An HPC data structure is carefully constructed as a function of system memory and microprocessor cache and TLB attributes. Not having control over that makes the HPC application uncontrollable and thus unpredictable. The second problem of clouds is that the gear used is not uniform and typically cheap, that is, cost is extracted from the system on components that are essential for HPC: network latency and throughput. Variable cpu gear creates problems in load balancing and low cost networking creates problems for the efficiency of the algorithms.

Having gone through that exersize we now understand cloud computing and virtualization much better and what it means for the applications that are good candidates. The MapReduce concept and applications that can leverage MapReduce are very natural fits for cloud computing. So we are working with Hadoop/Lucene/Nutch and working through the issues for those types of applications. In that Hadoop space it is interesting that there are already quite a few companies that leverage Hadoop on EC-2/S3.

In summary, our assessment of cloud computing is that for workloads that do not care about latency, clouds are useful. However, cloud computing combined with data sets, such as Google or Amazon, cloud computing is much more interesting. We believe that cloud computing as a service for computes is not viable: it has to be coupled with an application, and even more attractively, with a valuable data set. When you think about the success of outsourced web servers and web application servers this should not come as a surprise. Organizations such as Rackspace attract an end user of an application, in their case, web servers, but they don't have any leverage to add value to their offering. Google and Amazon do have that option and the richness of their offerings easily surpasses Rackspace.

Personally, I think we'll see a huge increase in the "dark matter' of the internet over the next couple of years. Dark matter defined as the data created by organizations with valuable data assets on the internet for sale. Examples would be market makers like Dun and Bradstreet, Gartner, IDC, or financial service brokers, and of course any product organization like GE and Boeing. These organizations are either motivated by generating revenue from their data asset or by leveraging the internet to amplify their geographical diversity. This makes it more plausible that federated systems are the end game, not cloud computing.

In one way, you can think of Google as an federated system. The world's web servers are the federated systems that Google aggregates, caches and adds value to the data contained. The world can self-organize these web servers into a collection of server farms managed by IBM, Google, eBay, and Amazon, or these web servers will become larger hubs themselves aggregating the data behind the firewall for profit or productivity.