The Data Stack

Data Security & Cisco Live 2013

webadmin@intel.com — Wed, 19 Jun 2013 23:25:12 GMT

I recently wrote about the fact that I just passed my 19th anniversary as an employee of Intel Corporation. This has caused me to reflect more than once on the state of computing, the industry, and the challenges we at Intel work on in support of our customers. Cloud computing promises agility, resiliency, and cost savings - but it also introduces new challenges that must be addressed. Security, a challenge and concern for IT leadership throughout my career at Intel, remains a top of mind concern but has taken on new characteristics in today’s world. Intel has a significant focus and investment with our partners and customers in creating innovations to address key security concerns for today and into the future.

Multi-tenancy and virtualization, as examples, both introduce architectural elements that require new and different security considerations for IT organizations. Virtualization and IaaS (Infrastructure as a Service) bring flexibility and scale in support of enterprise customers needs. However, virtualization can abstract where your workloads actually run, and often IT Managers need to know and control where their workloads are running in order to meet compliance requirements for their businesses. One example of Intel’s innovation to address these challenges is Intel® Trusted Execution Technology (Intel® TXT). Intel TXT provides a mechanism to enable visibility, trust, and control in the cloud and is a feature of our Intel Xeon™ product line. When coupled with an enabled operating system, hypervisor, and enabled applications, these capabilities establish an automated hardware root of trust method for enforcing and monitoring such things as geolocation restrictions for cloud servers. For an in-depth explanation of trust-based computing pools and geolocation-based secure migration refer to this document from the National Institute of Standards and Technology as great reading material on your trip to Orlando!

Intel has an interesting set of demonstrations and topics lined up for Cisco Live in Orlando on June 24-27th. Please stop by our booth and see how enterprises can enhance security and create a trusted cloud environment with geolocation controls whether they have a private VMware environment as in our demonstration with Hytrust or a when using the public cloud as we demonstrate with Virtustream. We look forward to seeing you there! Feel free to reach me via Twitter at @RaejeanneS if you have questions you’d like answered ahead of time on TXT.

HP Moonshot: Lifting Off with Intel Atom and Xeon Processors

webadmin@intel.com — Fri, 14 Jun 2013 20:31:28 GMT

On my way back from Vegas now – a jam packed, 36 hours where I never saw the sun and never left the hotel (and didn’t sleep much) – but well worth the trip all the same. I attended HP’s Discover conference for two days with a primary focus to listen to HP’s top executives and a number of their customers discuss the new Moonshot system that launched just this past April. And it wasn’t hard to do – Moonshot certainly was the rock star of the event!

Honestly, I was expecting a lather, rinse, repeat from the recent launch, but was pleasantly surprised to see HP do a double click down into exactly when, where and why a customer would want to use Moonshot…and even more importantly, where not to use Moonshot. This type of clarity was exactly what the audience was looking for in my opinion and Paul Santeler, VP and GM of HP’s Hyperscale division did an excellent job of it in his break out seminar. A few key points:

Moonshot is not a replacement for their traditional server systems like the DL/SL line and blades – it’s an augmentation. Designed for light weight, scale out workloads and consumable only in a fully loaded chassis (45 Intel® Atom® S family SOC-based cartridges today, with option to quadruple the density with 180 cartridges with the next gen “Avoton” based cartridges coming later this year) – it’s a platform targeted at web-scale environments with applications written and optimized for light weight, distributed workloads
The application is “back in charge”. Moonshot is configurable across compute, network and storage based on the application need. Instead of running every DC app on a general purpose server, light weight, scale out applications can run on a flexible, easy to configure infrastructure. One can now optimize across such workloads as static web, batch analytics and simple content delivery, all by turning the knobs on the HP Moonshot system. And although there is only one production cartridge shipping today (based on the world’s first 64-bit, 6W, data center class SOC - the Atom S 1200), soon there will be a range of cartridges that span software compatible Atom to Xeon® processors to address a broader range of workloads.
Good things come from collaboration. Intel and HP have been working together on Moonshot for a number of years, since the initial design and development stages. We are proud that HP chose Intel to be the lead partner for Moonshot and we will continue to innovate together and deliver more value to our end users. Together we bring decades of experience, a full platform portfolio and a developed ecosystem just waiting to explore Moonshot. The best is yet to come.

Congrats to HP on a successful show. Big kudos to Meg for weaving in Kevin Bacon into the opening keynote (hit me up on twitter at @RaejeanneS if you want more on this!). And thank you for choosing Intel as your collaboration partner.

What’s next for Intel in this space? Well, my whirlwind travel schedule continues next week at the ODCA Forecast’13 and GigaOm Structure’13 events in San Francisco, a great confluence of enterprise experts talking cloud adoption at Forecast and industry insiders discussing the future of cloud infrastructure innovation at Structure. In the meantime, for the latest on what Intel is doing with our products, solutions and customers, please visit the IT Center on Intel.com.

Partners Extend the Benefits of Intel Distribution for Apache Hadoop Software

tim.allen@intel.com — Thu, 13 Jun 2013 16:57:28 GMT

There’s more to the Intel^® Distribution for Apache Hadoop* software (IDH) than meets the eye.

Intel is building an entire partner ecosystem around IDH that extends optimized hardware, data storage and analytics support to help ensure that IT orgs get the value and intelligence they need out of big data.

IDH helps organizations store and analyze big data by providing an open source data management platform with the hardware-level security, manageability and performance acceleration features of Intel^® Xeon^® processors. It also comes with technical support, training, and professional services from Intel. IDH is the distribution of choice for enterprises seeking to deploy open source Hadoop for processing big data at multi-petabyte scale.

However, the compute intensive processes of Hadoop require a combination of hardware and software optimizations and specialized analytics and visualization tools to deliver the insights, scale and ROI demanded of big data. In addition to server architectures and IDH, Intel provides a number of tools to help manage Hadoop, including Intel^® Graphbuilder, which enables distributed graph analytics on top of Hadoop. However, Intel turns to its partners to help create a larger vendor ecosystem of optimized and co-engineered solutions and to build a more complete IDH computing environment.

RainStor Takes Complexity, Cost out of Big Data Querying

I estimate that up to 90 percent of organizations that have deployed Hadoop clusters today are just using them as an ETL offload. In other words, they are using Hadoop to store big content, but haven’t gotten around to taking advantage of Hadoop’s benefits as an engine for big data analysis. That’s where RainStor comes in. RainStor helps organizations achieve business insights at lower costs than other data stores, and uses familiar query and BI tools to reduce the complexity of big data analytics.

Rainstor for Hadoop* is a big data infrastructure that runs natively on the Intel Distribution for Hadoop. It’s made to not just handle the velocity and growth of today’s data, but also tackle the changing nature of data itself—log files, web clickstreams, Twitter content, machine generated data, and more, all in great volume.

But most IT admins don’t want to become query specialists or data scientists just to analyze their big data. Using RainStor, they can run real SQL queries on Hadoop stores, taking much of the complexity out of big data analysis. Rainstor is standards-based, and it uses specialized JDBC and ODBC drivers to peer into persistent Hadoop data. It can then run queries through the data using the familiar SQL environment—and DBAs don’t have to be stuck with HiveQL. To some that could be like telling a C++ developer that JavaScript coding requires no further training.

Rainstor also offers data compression and de-duplication capabilities that can lower the storage footprint by as much as 20 to 40 times. These compression features not only reduce the hardware needs and costs of big data, they also speed up querying. Rainstor is a great example of how partners are building out the Intel Hadoop ecosystem with innovative technologies. Learn more about Rainstor for Hadoop, and follow Tim and the growing #Intel #BigData Hadoop community at @TimIntel.

Intel Atom S1200 helps HP Moonshot to lift off in Vegas.

webadmin@intel.com — Fri, 07 Jun 2013 14:39:11 GMT

HP and Intel have been collaborating for a few years on project Moonshot to bring new levels of density, efficiency and TCO for light weight web workloads such as static web and dedicated hosting. Recently, HP unveiled their first generation Moonshot systems and I am excited to see that the first and only production HP ProLiant Moonshot servers available today are based on the Intel Atom S1200 processor family.

HP chose to lead with the Intel Atom processor for many reasons. First, HP and Intel have a long history of collaboration and we have brought many innovations to market first on Intel and HP’s platforms. Second, the Intel Atom S1200 processor is the industry’s only available 64-bit SoC with critical data center class features such as full 64-bit software ecosystem support, ECC and Intel Virtualization Technology - all within an ultra-low power 6W TDP. This means that today the ProLiant Moonshot servers using Atom S1200 can drop into any environment and software applications will run seamlessly on the server without porting needed. The lower power you want, with the software applications you need. Third, this SoC was designed for targeted lightweight web scale workloads, including low-end dedicated hosting, simple content delivery, and offline batch analytics making Intel Atom S1200 the perfect SoC solution for HP Moonshot’s target markets.

Moonshot servers with the Intel Atom S1200 are shipping to customers today and receiving great reviews. We look forward to sharing more results from customers going forward as these implementations go public. We also look forward to seeing Moonshot systems that will take advantage of higher density HP ProLiant Moonshot servers using Intel’s next generation Atom SoC coming later this year. The next gen servers will be built on Intel’s 2nd generation 64-bit Intel Atom SoC, code named “Avoton”. Avoton is built on Intel’s leading 3D tri-gate 22-nanometer (nm) process technology and is based on a new microarchitecture codenamed “Silvermont”. It will feature an integrated Ethernet fabric controller and deliver improvements over today’s Intel Atom S1200 in performance per watt and energy efficiency through a combination of new capabilities, new microarchitecture and leadership manufacturing technology. Avoton is now being sampled to customers and the first systems are expected to be available in second half of 2013. Moonshot servers using Avoton will quadruple the density (4 Avoton SoCs per server) vs. the current generation just announced using Intel Atom S1200.

2013 is and will be a great year for Intel and HP Moonshot. We have not only enabled the first Moonshot system to lift-off but with Avoton we will also bring HP Moonshot’s customers a revolution in energy efficiency and performance per watt to drive major TCO improvements when processing lightweight web scale workloads.

Learn all about it at this year’s #HPDiscover in Vegas June 11-13. Sign up for the exclusive NDA session BB BB4355 on Wednesday, June 12th at 12:45PM in Murano 3206 and follow all the action on Twitter at @IntelITS.

19 Years of Data Center

webadmin@intel.com — Thu, 06 Jun 2013 16:58:22 GMT

I woke up this morning and realized that yesterday marked my 19th year at Intel. Just as turning 29 and 39 freaked me out more than the actual decade birthday in my life, year 19 at Intel woke me up that soon I’d be 2 decades into a career and quite honestly, most of it went by so fast that I can’t quite say how I got here some days. Should I have tried other companies/industries, should I have tried other paths than marketing or should I have ventured beyond the data center ?!

So, I did some deep contemplation during my [very long] commute this morning and here is what I came up with….I love my job, I am proud to work for Intel and despite the [very long] commute – one couldn’t ask for a better employer and career than with Intel’s data center group. But, I also realized that there were a few things along the way that helped me stay “me” as I’ve gone through it. Listed in increasing or of importance:

I will not wear a "logo’d" polo shirt, or for that matter any logo’d apparel, as I work. Polo shirts make me look like a teenage boy and quite honestly, high tech logos really aren’t my style. I will also not wear a sweater set or sensible shoes to work. If I am not meeting with a customer, I wear jeans. Period. Even better when paired with 4 inch heels. And don’t even get me started on the “lanyard” concept…
I will not apologize or make an excuse why I have spent most of my career in the Enterprise / Data Center domain. If you think consumer, tablets, phones, etc are “sexier”…well you obviously haven’t really looked into the big cloud service providers, rack disaggregation or what is powering the voice/gesture recognition technologies on those sexy phone and clients. Game changing tech trends that are rooted in the data center
I choose my manager as much as I choose my job. During my 19 years I have had some exceptional managers. The commonality across them – I honestly believe they care as much about me as they care about my output. They’ve had my back and in return I will always have theirs. They also build their organizations with that culture – respect, no tolerance for politics, investment in career development and support for work/life balance. I am now a working mom with elementary age twins and I travel too much. Did I also mention the very long commute? That means I only have time to work on real work. I couldn’t do what I do at home and at work if I had to worry about watching my back or battling bureaucracy.

So although my life has dramatically changed over 19 years, and Intel with it, I am glad to say that I have landed in a highly relevant, fast paced, super cool technology space where I feel supported to be ME – data center junkie, lover of new tech, Intel loyalist and devoted mom who never wants to miss a talent show or soccer game.

Question to you –

How have you struck balance and kept interest in your career over one, two or more decades in the data center business?

How to Optimize Hadoop Performance on Intel Architecture

tim.allen@intel.com — Thu, 06 Jun 2013 00:11:20 GMT

If you are dealing with data analytics, you have probably heard about or are using Apache Hadoop*. By leveraging the compute power of multiple nodes in a cluster, Hadoop can analyze practically unlimited volumes of data—both structured and unstructured—with lightning-fast speed.

But did you know you can optimize Hadoop to deliver even better performance on Intel architecture? The key is to tune the underlying Java so that it takes advantage of capabilities in Intel hardware. When you do that, you can expect to see up to 70 percent faster performance on Hadoop sort operations. Keep reading to learn how.

Understanding Hadoop’s Java Foundation

A Hadoop cluster comprises a master node and multiple slave nodes, where data is stored and where analytics processing occurs. An incoming analytics request invokes several Java* services that enable efficient replication and large-scale analytics across nodes in a cluster.

Processing begins with two Java services on the master node. These two services then communicate with additional Java services on the slave node where the data are stored—see the image below for a simple view of what’s going on.

When MapReduce* completes all the assigned analytics tasks, it returns the results to the master node, which compiles results from multiple nodes and returns them to the requester.

Important note: Hadoop spawns a new Java Virtual Machine* (JVM) for each MapReduce function on each slave node. This means that a large analytics job can result in the creation of thousands of individual JVMs. Because Hadoop does not share memory resources across nodes, each JVM and Java service must perform optimally. Reduced performance on any single node can hamper data analytics performance across the cluster.

Tune Java to Optimize Hadoop Performance

Given this foundation of Java services, it is easy to see that optimizing Java for Intel® architecture can deliver significant Hadoop performance enhancements. Optimizations are built into Java and Intel architecture, which make these improvements easy to achieve. When Intel releases a new microarchitecture and platform, Intel and Oracle software engineers work together to tune the JVM to take advantage of the new hardware advances. These optimizations can provide faster Hadoop performance on each Intel-based node as well as on the entire cluster.

How much faster? Up to 50 percent faster on a TeraSort* benchmark test and 70 percent faster on the Hadoop Sort benchmark. Read the white paper, Optimizing Java and Apache Hadoop for Intel Architecture, to see the details.

Since 2007, Oracle and Intel have improved Java performance up to 14 times by tying specific Java optimizations to advancements in the underlying hardware. Read the paper to see a complete list of optimizations, but here’s a taste of some that are important to Hadoop performance:

Fast CRC increases file checksum and compression/decompression checksum, which increases Hadoop network and file system performance.

Large-page usage increases performance of large analytics jobs.

Intel® Advanced Vector Extensions improve the performance of MapReduce operations that contain array and string manipulation, such as sub-string or character searches; also improves integer and floating point calculations.

Intel® Integrated Performance Primitives (Intel® IPP) compression increases compression performance, which reduces network and disk input/output (I/O) across the Hadoop cluster.

If you’re using Hadoop, or analytics on large data sets are part of your job, then check out the white paper for a full description of the Java optimizations that are available to you to enhance Hadoop performance on Intel architecture.

Follow me @TimIntel.

Go Ahead and Virtualize IBM InfoSphere* Information Server

tim.allen@intel.com — Thu, 23 May 2013 15:42:13 GMT

If you want to virtualize IBM InfoSphere* but have put it off, I have some new information that might give you the bump you need to move forward: How does 98 percent throughput sound?

In a nutshell: extensive testing shows that InfoSphere can perform very well when virtualized. How well? We achieved between 90 and 98 percent of the throughput that we typically see in a physical environment, with an overhead tax of only about 10 percent on I/O-intensive workloads.

You can get all of the configuration and results details in the white paper about InfoSphere virtualization here, but I’ll hit the highlights for you in this post.

While virtualizing InfoSphere Information Server is cool, here’s the icing on the cake: virtualization with VMware vSphere* is now supported as an IBM PureApplication* pattern. “Patterns” are a growing ecosystem of software stacks for IBM PureSystems* running on Intel® Xeon® processors. Hence, adding the leading VMware vSphere 5.1 stack indeed broadens the use of these rich appliance systems versus other industry systems. PureSystems are innovative, quick-to-uptime, and run industry-standard software stacks.

Testing InfoSphere Virtualization

IBM, VMware, and Intel teamed up recently to see how IBM InfoSphere performs when it’s virtualized. Specifically, we looked at the runtime performance of:

IBM InfoSphere DataStage* 8.7
Running on VMware vSphere 5.0
On a server powered by the Intel Xeon processor E7 processor family

The tests found that InfoSphere DataStage scaled smoothly as we cranked up the number of virtual CPUs (vCPUs), while clocking throughput at up to 98 percent of that found in a physical environment.

More good news: we saw only a slight performance difference when using VMFS (Virtual Machine File System) versus RDM (Raw Device Mapping) data stores. This means that you can reap the benefits of VMFS for storage provisioning with virtualized workloads without concern over performance.

Now let’s get down into some of the details so you can put these results in context.

InfoSphere Virtualization Test Configuration

Server

We used an IBM* System x3850 X5 with a network-attached IBM System Storage* DS5300:

Four socket system
40 physical cores
Intel Xeon E7-8870 processors
Configured as “optimized for performance”

To get a good comparison between physical and virtual environments, we controlled RAM and processor availability to the native environment so that we could match the virtual environment as closely as possible. We enabled all of the virtualization-related options, except for Intel® Hyper-Threading Technology (Intel® HT Technology), which we disabled to simplify the comparison.

IBM InfoSphere

We installed all InfoSphere DataStage components on one physical server in the native environment and on one virtual machine in the virtualized environment:

IBM WebSphere* 8.1 Application Server (WAS)
XMeta repository
DataStage engine

We found that DataStage engine running in a single virtual machine had higher throughput. The specific reasons for this result weren’t clear, but we’re hopeful that further testing will provide more detail.

ETL Workload Virtualization Test Results

As expected, throughput in both environments increased as the number of processor cores increased, with performance in the two environments varying only between 2 and 10 percent. The overhead for the virtual environment was only 10 percent for all tested configurations.

Host Server Memory-Management Test Results

We also wanted to see what would happen with DataStage performance when overcommitting the host server. We measured a 34 percent drop in throughput when the system was 100 percent committed, and throughput continued to drop as the host processors were further committed.

Working with VMware engineers, we determined that this drop was because the eight vCPUs were not mapping neatly onto the 10-core, Non-Uniform Memory Access (NUMA)–node design of the Intel Xeon processor E7 family’s microarchitecture.

To work around this issue, you could use a five vCPU configuration instead of an eight vCPU configuration. Five vCPUs would map well to the 10-core NUMA nodes. Another workaround would be to turn off NUMA scheduling in BIOS or VMware vSphere, which would allow all of the CPU cores to be used, though you would see a lag in memory performance. The lesson? Understand NUMA configuration to optimize VM performance.

We repeated this over-commitment test with the VMware vSphere “reserve CPU” option for the DataStage guest set to maximum, and the result showed minimal performance impact. However, this move can potentially impact the performance of the other non-reserved virtual machines running on the same host.

If your system is overcommitted and you’re not seeing the DataStage runtime performance you want, the best option would be to increase the host system capacity or to move the VM to another host.

Go Ahead and Virtualize InfoSphere

Our tests showed InfoSphere Information Server runtime performed very well in a virtualized environment hosted on a platform powered by the Intel Xeon processor E7 family and using VMware vSphere 5.0. So if you’ve wanted to virtualize your InfoSphere applications but were concerned about performance, now might be the time.

Take a look at the white paper to see the test configuration, procedure, and results. And follow me on Twitter, @TimIntel, to get more useful InfoSphere and DB2* tidbits. You can also get the latest news and technology updates at the joint Intel and IBM DB2 website.

Server Storage Caching Considerations

webadmin@intel.com — Wed, 22 May 2013 14:00:16 GMT

Caching is storage. Here’s how Wikipedia defines caching:

“… .a cache (pron.: /ˈ k æ ʃ / KASH) is a component that transparently stores data so that future requests for that data can be served faster.”

With the data explosion and consumers creating content and wanting to access it immediately, caching is becoming more and more important.

My colleague, Susan Bobholz is a Marketing Director in Intel’s Data Center Software Division and talks about the considerations for server storage caching.

Right now, one of the hottest storage topics is storage caching. It seems hardly a week goes by without some type of caching software showing up in the press. I thought I’d spend a bit of time talking about this trend and provide some things to think about when choosing caching software.

In many datacenters, the hottest, most frequently accessed data is stored on 15K serial attached SCSI (SAS) hard drives. But those hard drives can become a bottleneck because they are mechanical devices with moving parts. They simply can’t move fastest enough to keep up with some application demands. One solution to resolve this is to replace all those 15K SAS hard drives with Solid State Drives (SSDs) but this can be an expensive undertaking. The beauty of storage caching is that it protects your investment in hard drives, because your application performance is improved without replacing all those hard drives with SSDs.

To be as simple as possible, storage caching allows the hottest, most important data to be stored in a SSD instead of hard drives, allowing that data to be accessed significantly faster. Often caching is implemented as a server application, but sometimes it’s actually part of the firmware on a RAID HBA

So you’ve decided you want to implement storage caching. There are several caching options out there. Other than cost, what are some key questions to consider about when deciding which to use?

Where does your cache physically reside?

As mentioned above, some caching solutions are integrated into a RAID HBA. This means that the Cache SSD must be attached to the RAID HBA itself and only data on hard drives connected to that RAID HBA can be accelerated.

Other caching solutions allow the Cache SSD to be anywhere inside the server itself. This provides additional flexibility as the data being cached can be anywhere on the server - behind a RAID HBA, behind a SAS HBA or even attached to the chipset SATA ports.

In addition to being able to have the Cache SSD inside the server, some caching solutions allow the Cache SSD to be outside the server, in a SAN or NAS. This is important in virtualized servers as this allows virtual machine migration to occur automatically. The cache remains active while the virtual machine moves from host to host.

Consider where you want the cache SSD to connect to your server when choosing a caching solution.

Which OSes are supported?

Think about what OSes exist in your datacenter. Windows? Linux? Virtualized OSes such as VMware ESX or Xen? Think about whether it’s important to have a common caching solution from one vendor across all these environments. Not all caching solutions support all these OSes.

Being able to choose what goes into the cache

This may sound unimportant, but imagine having an SLA with a customer that requires you to deliver the lowest latency to the data associated with that application. What if you could guarantee that specific data was always in the cache, ready to be accessed? Several caching solutions available today offer proprietary ways to pin data into the cache. This is becoming so important that standards bodies such as ANSI T10 are looking into ways to standardize ways to determine whether data should be kept into a cache at all times.

Read caching or write caching?

Look at the applications you want to accelerate. Do they mostly read data from hard drives or do they mostly write data? Or is it a mix? Some caching solutions are better are accelerate reads, others are better at writes. Choose a caching solution that meets the needs of the applications you want to accelerate.

Caching algorithms aren’t all the same

We all learned about Least Recently Used caching in school. Just as the name implies when the cache is full but new data needs to be added to the cache, the data that has been sitting in the cache the longest without being use will be evicted to make room for the new data. This can be an effective algorithm and is very common. But some caching solutions add intelligence to the caching algorithm and are able to decide to keep specific most popular/active data in the cache longer, protecting it from being evicted by more recent, but less popular data. This reduces the probability that important data is evicted from the cache, improving overall application performance.

So, these are just some of the areas to consider when choosing a storage caching solution. What is important to you when you choose a caching solution? Let me know!

Full disclosure: Intel has its own caching solution: Intel® Cache Acceleration Software that works with Intel® Datacenter SSDs. We think it’s pretty cool. A 30 day trial is available on intel.com.

Susan Bobholz is a Marketing Director in Intel’s Datacenter Software Division. She’s been with Intel for 20 years, doing everything from software development to initiative management to product marketing, focused on storage technologies and products. Prior to joining Intel, Susan developed software at Siemens Medical Labs and firmware for Motorola cell phones. She graduated from the University of Wisconsin with a BS in Electrical and Computer Engineering. She holds 3 patents.

At TDWI: Finding a middle way between Hadoop and relational data warehousing

tim.allen@intel.com — Fri, 17 May 2013 22:23:37 GMT

If you’re thinking big data analytics will solve all your BI challenges, you may be looking at it wrong…

I realized this when I was in Chicago the first week of May, attending The Data Warehousing Institute (TDWI) conference, where the theme was “Preparing for the Practical Realities of Big Data.”

There are high expectations around big data at the moment. Many people in marketing, product development, and analytics teams can’t wait to get their hands on big data intelligence to better understand and target audiences.

However, TDWI isn’t their show. At TDWI, the focus instead is on traditional database administrators, data analysts, and data scientists, and it’s a very technical conference firmly based on OLAP and OLTP analytics and hands-on issues of data warehousing.

With this in mind, I attended the keynote address by Ken Rudin, director of Analytics at Facebook—a leader in cool, cutting-edge big data processing and analysis addressing a conference of (what some would consider) old-school DBAs. The message from Rudin, who has held senior leadership positions at Zynga, Salesforce.com, and Oracle, was fascinating: Don’t get caught in the tyranny of either/or when it comes to data analysis—businesses need both traditional relational database processing and Hadoop*-based big data analysis.

@krudin said that Facebook started out relying almost solely on Hadoop and big data when the social media giant first launched, but now is increasingly incorporating OLAP and OLTP processes into its analytics. Hadoop is best at exploring huge data sets—putting all the data into one system to discover patterns. Traditional relational data analytics is best at business, looking at focused data to derive metrics and more actionable, granular analysis. Both are valuable technologies: which one is best depends on what kind of impact you are looking for.

So the question is not, how do you get from old-school to cutting edge as soon as possible. Rather, ask which technology is right to generate impact out of data.

This was a conclusion that appealed to many at TDWI, as several people I spoke to registered a bit of skepticism about the value of Hadoop as an engine for analytics. For them, the main attraction of Hadoop is its potential to act as a backend data storage mechanism, where unstructured data can be warehoused.

Then the question becomes: What’s the best way to integrate data stored in Hadoop into a traditional OLAP or OLTP infrastructure for processing? The answer is just around the corner.

At the SAP booth, I presented a discussion of the newly released joint solution from SAP and Intel that has optimized Intel^® Distribution of Apache Hadoop* software for the SAP HANA* in-memory database. Using SAP Smart Data Access* (watch for availability in coming weeks), this big data solution is able to leverage all types of data for processing in analytical applications.

And since it’s built on Intel^® architecture, and leverages the full power of Intel^® Xeon^® E7 processors, HANA has hardware-enhanced performance and security built in, with solid-state drives and cache acceleration for blazing speed and stability. Watch this interview from TDWI to learn more about SAP HANA and how the database helps address big data challenges.

If you’re looking for the technology to get the most impact out of analytics, look no further.

Follow Tim and Twitter @TimIntel and the SAP analytics team at @SAPAnalytics.

Additionally, @TDWI has some very interesting DW feeds.

IBM Impact 2013: Heroes of IT, Please Take a Bow

tim.allen@intel.com — Thu, 16 May 2013 16:39:05 GMT

At last week’s IBM Impact conference, the focus was on “Technology in Motion,” and the keynote presentations were both inspiring and thought-provoking. The program celebrated some of the 400 IBM Champions from around the world who, as heroes of IT, help open new horizons and new audiences to technology.

Wednesday’s keynote kept with the “tech in motion” theme, and the most stirring presentations involved how IBM is helping open up access to computing for underserved communities. Doug Schmidt of Pearson Publishing described how his company uses innovative technologies based on IBM WebSphere MQ*, IBM Message Broker*, and IBM Operational Decision Management* to bring higher education to students in over 70 countries and in a variety of formats.

We also viewed a video presentation from P-TECH, the Pathways in Technology Early College High School, an innovative collaboration between New York public schools, the City University of New York, and IBM. The P-TECH schools (which include branches in Chicago and Idaho in addition to the original location in Brooklyn) bring together private-sector partners to design an academic program that’s heavy on high tech workplace skills. Students all have IBM mentors to help make sure they get the skills to land good, technology-oriented jobs. P-TECH program has been so successful that President Obama highlighted it in this year’s State of the Union Address: “We need to give every American student opportunities like this,” he stated. Is there a P-TECH school in my town for my kids?

As the focus of Impact is WebSphere middleware, IBM highlighted some recent testing between generations of Xeon^® processors and just upgraded IBM WebSphere* 8.5 vs. 8.1 performance gains. The nearly 260% performance gain with newer versions of software and hardware is huge for the middleware community and hopefully will be sufficient reason to upgrade both processor middleware versions. Details of new SPECjEnterprise2010 test results are detailed here: spec.org/jEnterprise2010.

Intel and IBM have worked together for over 15 years to optimize software performance on the Intel infrastructure, and their collaboration and co-engineering on IBM WebSphere 8.5 has led to very impressive performance gains. And when it comes to technology in motion, it’s this kind of optimized joint engineering that creates new IT heroes. Watch this interview with our Intel hero, Pauline Nist (@panist), general manager of Enterprise Software Alliances, and learn more about the growing collaboration between Intel and software partners such as IBM.

Follow Tim on @TimIntel and @IntelAdrenaline.

SAP and Intel Showcase Big Data Solutions at Sapphire 2013

webadmin@intel.com — Wed, 15 May 2013 17:22:08 GMT

Intel and SAP have worked closely together for years. But, our co-engineering has reached a new level of integration with the recent launch of our Big Data solution based on the SAP HANA^® platform and Intel^® Distribution for Apache™ Hadoop^® software. The joint solution will be demo’d at SAP Sapphire 2013, SAP’s flagship conference in Orlando, held May 14 – 16.

Stop by the Intel booth #3215 and experience our Big Data solution.

The Intel Distribution of Apache Hadoop running on SAP HANA represents more than a typical collaboration. It’s a great example of how co-engineering can result in a “more-than-the-sum-of-its parts” success that changes the game by vastly improving the speed and resilience of Big Data analytics.

HANA is SAP’s blindingly fast database platform that consolidates transactional and analytical workloads into a single, in-memory process. Combining OLAP and OLTP structures into a unified landscape eliminates traditional relational limitations that have restricted the development of real-time business applications, and in particular Big Data analytics.

Apache Hadoop is the industry’s open source standard for managing Big Data, but to ensure that Hadoop’s data-intensive workflows can provide real-time analytics on an enterprise scale requires a comprehensive computing platform with muscle and intelligence.

The Intel Distribution for Apache Hadoop is specifically optimized for the advanced features of SAP HANA and the Intel® Xeon® processor E7 family. Intel’s version of Hadoop can leverage SAP HANA in-memory technologies to accelerate data analytics and also tap into extra performance and scalability that the Intel processor E7 family is optimized for. In turn, SAP HANA’s in-memory processes help eliminate the latencies found in Hadoop’s underlying file system to enable on-demand data analysis.

Through joint-engineering, SAP and Intel have delivered a breakthrough Big Data solution that can store and analyze massive volumes of structured and unstructured data in real time. The underlying platform has the performance to scale to continued exponential data growth and deliver rapid-fire insights to help boost business productivity and profits.

For more information go to: http://hadoop.intel.com/resources

How do you explain the cloud to your parents?

webadmin@intel.com — Wed, 15 May 2013 14:50:24 GMT

How important is storage to your business? My name is Ilene Aginsky and I work in the Storage Division. I’m new to this community and I think this is the perfect place to talk about how the explosion of digital data is changing the way businesses and even consumers need to manage this incredible growth.

It seems that storage should be a relatively easy concept however when data is being created at such a rapid pace it starts to get complicated very quickly. Given this complexity, I will reach out to my colleagues here at Intel and ask them to help explain the technology that surrounds the concept of storage.

For my first foray, I have asked my colleague, Gary McCulley, a Product Line Manager for the Storage Division to explain the concept of Cloud in storage. Below, in Gary’s own words, is that explanation.

How I would explain the “Cloud” to my parents

My parents recently asked me how work was going. Being a Product Marketing Engineer at Intel, I tend to give them vague, (yet accurate) feedback, like “oh, it’s going well” or “Intel as a company is doing well”, etc. In other words: “blah, blah, blah”; I tend not to get into the specifics, fearing I would be in long conversation that would span a diverse range of topics, including explaining how Al Gore really didn’t invent the Internet all the way to how the “Cloud” really has very little to do with the weather. But it got me asking myself if I could coherently tell them about important concepts like the “Cloud” and wondering if they would have a better understanding after my explanation.

Gartner Inc. recently forecasted that one-third of consumers' digital content would be stored in the “Cloud” by 2016. It's an interesting statistic that certainly would impress my parents, even though they may not initially understand it.

Aside -- My parents love me and are impressed by most anything I do or say– sort of like Ben Stiller’s parents in Meet the Fockers, who created a shrine in their son’s honor, including his 9^th place ribbon. By comparison, my mom has a framed picture of a duck that I painted when I was in 5^th grade hanging on their bathroom wall – I earned 2^nd place in my 5^th grade class of 25 students!

Upon telling my parents the “Cloud” statistic, they would likely ask the following questions:

“OK, what is digital content and what is the “Cloud”?

First, I would give them examples of digital content: pictures of their grand kids taken with a digital camera, email they send and receive songs they listen to on Pandora, videos of puppies on YouTube, and other examples. Then I would tell them why the “Cloud” is important. I would tell them the growth of multiple connected devices, most of which have cameras (my mom’s cell phone even has a camera), has led to a massive increase in digital content that users have created. And all that content has to be stored somewhere. I would tell them that people used to store content on their PCs (and still do), but with all this new digital content, people now like to store their data on social media sites, like Facebook, which offer free storage space for uploading videos and photos for social sharing.

I would explain to them that every time they access anything that isn’t stored in their computer they are engaging the “Cloud” - like when they access their Yahoo! email account, Facebook, Google, or YouTube. Basically, the “Cloud” allows anyone with an Internet connection to access information that is contained in a network of computers and servers (i.e. a data center) located around the world. This makes it easier to share pictures and videos, for example, of their kids and grand kids.

In addition to Facebook , Google, YouTube, and Apple - companies that provide free storage, I would also explain that some of them charge customers (on a monthly basis) to store their data – typically to customers that have a lot of data to store. Companies like those above and like Amazon Web Services, which have locations with server farms, (i.e. data centers), will store data for customers for a monthly service-based subscription.

Side note: while online backup services like Amazon Web Services, Google Drive, and others, are well-known cloud storage providers, their total storage allocated to consumers and "pro-sumers" is still small relative to that maintained by social media sites like Facebook, Twitter, and other sites.

Anyway, at this point, my parents would be losing interest, but would still act like they are really interested. So I would try and wrap it up by saying the term “Cloud”is really just a metaphor for the cloud-like shape used to represent the Internet’s infrastructure when engineers draw it.

And then I would tell my Mom that I took a picture of my duck picture and uploaded it to Facebook, where she could enjoy it on her smart phone anytime she liked. I would then be told that the picture isn’t that great, and there was a good reason it is displayed in the bathroom...

Gary McCulley is a product line manager in the Data Center and Connected Systems Group at Intel ® Corporation. Prior to joining Intel, Gary held strategic planning and product marketing positions at Broadcom Corporation, Philips Semiconductors, and General Dynamics. He earned a B.S. in Engineering from Arizona State University, an M.S. in Engineering from San Diego State University, and an MBA from the University of Arizona.

What to Discover at HP Discover US June 11-13

webadmin@intel.com — Wed, 08 May 2013 15:40:03 GMT

HP and Intel have a long standing partnership and it feels like there will be no shortage of exciting things to come the rest of 2013 and into 2014. We are very excited to once again be the premier Sponsor of HP Discover this year. Many of you who are reading this have attended Discover, either in the US or in Europe and we look forward to seeing you again and also seeing some new faces.

What have we got in store for you this year? Well, one thing is for sure, We won’t just bore you with speeds and feeds (okay, maybe a little) but we want to invite you into the platform and technology advancements that become essential to the way we work AND the way we live and interact socially. Our innovation theater sessions will cover both the technological keys to optimizing your datacenter for the future (with sneak peeks into Intel Labs) but also the human future and balance of social power that’s being morphed by the intersection of enlightenment and technology.

Whether you want to think of it as “Return on…” a)”Investment” b)”Information” c) “Innovation “ Intel continues to work with HP to provide the building blocks that are the heart of the datacenter. No longer can you think of the datacenter as silo’d compute servers that provide awesome performance and enable cloud and virtualized environments, but you have to think of the Storage, Networking, Solid State Drives and even geekier technologies that support and augment the datacenter to help you meet your customers’ needs and positively impact your bottom line.

At Discover, look for Intel and HP to take the lead and show you how we can help your organization tackle Hyperscale computing, Big Data,
Cloud and Virtualization. Get in touch! @ITSandhya www.intel.com/xeon

Exascalar and Cost Effective HPC

webadmin@intel.com — Wed, 08 May 2013 13:00:39 GMT

I know… you’re probably thinking, “what the *?” The phrases “Cost effective” and “HPC” seem as rarely seen together as are a double bill of Ferris Bueller’s Day Off and Rocky Horror Picture Show. But, in fact, with rapidly expanding efficiency and performance capability of supercomputing systems, electricity costs in large scale machines may warrant deeper scrutiny of the need for newer hardware.

What got me thinking about this was a startling realization about systems

near the lower left “Corner of Inefficiency” of the familiar Exascalar plot below.

The systems near that lower left corner are almost a factor of one hundred less efficient than the most efficient systems of comparable performance. In other words they consume about one hundred times the energy for comparable work. This can be a big deal, for instance a 20kW system or a 2.0 MW system. If you think about the cost of electricity, there could be some real ROI there.

So the question is how to visualize that difference in cost. The point of what I discuss below is not to provide an accurate cost analysis for every application, but to show how this general framework can be put to use.

Costs of supercomputers, especially those at the forefront of innovation, are difficult to estimate. For the purposes here I chose to use a published cost of the Lawrence Livermore Labs Sequoia computer as the anchor point for this analysis. For comparison read about the ORNL supercomputer here. Assuming a constant $/flops one can easily scale capital cost according to performance. This scaling is shown as the horizontal lines in the Figure below.

Electricity costs also vary widely from location to location. Industrial electricity costs are actually falling in the US, but for the sake of simplicity I have assumed $0.07/kWh with an assumed facility PUE of 1.6. $0.07 is about the average industrial electricity rate in the US. This translates, conveniently, to a total energy cost of about $1/(Watt*Year). You can see system-level annualized energy costs in the Figure.

From this point it is pretty straight forward to calculate a payback time for replacing inefficient servers. It’s interesting they work out to be vertical lines. It’s interesting that they times for return on investment show up as vertical lines. It’s astounding that they are so short. In several cases, less than a year!

Again, this is not intended to be a definitive analysis of return on investment or total cost of supercomputer ownership. But I think this initial estimate is provocative enough to warrant further investigation. To me it looks like millions are on the table.

So, what are you waiting for?

Data Center Power: Zooming in Where it Matters

webadmin@intel.com — Sat, 04 May 2013 15:00:08 GMT

This post originally appeared in Information Management on December 26, 2012

Politics aside, the subject of energy is of great concern in every large data center. Why, then, is power consumption still an afterthought for most server deployments? Because IT and facilities teams typically work independently and neither team can control consumption or predict requirements when data center energy costs are buried in the overall utility bill.

Let’s face it: Energy costs are spiking, server sprawl is pushing against site capacity limits, and the Internet and smart device adoption rates are calling for aggressive increases in data center compute densities. Industry analyst firms agree that power and associated cooling requirements account for the fastest increasing components of operational costs. To protect the bottom line, and to comply with the latest EPA Energy Star standards, data centers need to change the way they monitor and manage energy consumption for power-hungry assets, like servers. New power and cooling management approaches are available that offer greater energy efficiency and reduced costs.

Traditional approaches to managing power and cooling have failed to control costs, in large part because they typically force over-budgeting to ensure priority needs are met. Ironically, even with overestimating and over provisioning cooling, data center hotspots continue to crop up, thereby impacting server availability, reducing data center cooling efficiency and driving up operational costs. These factors and their impact demand that facilities and IT professionals find a better way to achieve their common objectives.

Zooming in on the Right Measurement Points

One of the most fundamental barriers to achieving greater power efficiency and curbing runaway energy spend rates has been the inability to obtain accurate readings of actual server energy consumption levels. Various models have been developed that translate temperature and power consumption into overall data center energy requirements for servers and their associated cooling systems. However, even the best of these models lack the real-time visibility required to accurately understand and predict energy trends. Actual usage can vary significantly (up to 40 percent) from modeled predictions, and the models do not provide the immediate feedback required to pinpoint hot spots before they impact services or identify areas of waste where conservation can lead to savings.

Energy models are limited in terms of day-to-day management of power consumption. For example, we know from in-field measurements that an average of 15 percent of data center servers are “ghost” or “zombie” servers (servers that are not producing useful work, drawing energy just to stand idle). When we do the math, assuming that a server draws approximately 400 watts of power, which currently costs about $800 per year, companies are spending on average more than $24 billion per year for these “ghost” servers in their data centers.

Aggregating Server Thermal and Power Data

A second problem has been the technical challenges of aggregating data from varied and disparate systems within the data center. Facilities managers have been forced to cobble together, manually or with crude homegrown systems, vital data such as power supply of the server, inlet and outlet temperatures, asset information contained in RFID tags as well as temperature and humidity readings of the air conditioning units. This prevents the achievement of a “big picture” perspective of facilities’ server inlet temperatures and power consumption data from rack servers, blade servers, and the power-distribution units and uninterrupted power supplies related to those servers. The crippling effects of this piecemeal view are analogous to a long-distance truck driver suffering from tunnel vision.

By shifting attention from the cooling systems to the servers which account for the majority of the power consumed in the data center, managers can introduce a holistic energy optimization solution. Accurate monitoring of power consumption and thermal patterns creates a foundation for enterprise-wide decision-making with the ability to:

Monitor and analyze power data by server, rack, row or room;
Track usage for logical groups of resources that correlate to the organization or data center services;
Automate condition alerts and triggered power controls, based on consumption or thermal conditions and limits; and
Provide aggregated and fine-grained data to Web-accessible consoles and dashboards for intuitive views of energy use that are integrated with other data center and facilities management views.

Identifying temperatures at the server, versus at the room or even rack levels, can also help data center managers more accurately understand what the real ambient temperature should be for individual servers to have optimal life spans. This assessment of real temperatures has enabled data centers to increase the overall room temperature by one to two degrees, which can create significant savings in the air-conditioning expense.

Disseminating the power and cooling data, without impacting ongoing processing in the data center, is another challenge. Invasive monitoring approaches have the potential for adversely affecting the performance of existing systems. Agentless monitoring capabilities should have little impact on the overall system performance, and therefore are virtually undetectable to the end users’ experiences.

Where should all of this energy monitoring and aggregation functionality be placed within the data center? Ideally, all of this would take place transparently and non-invasively, to avoid impacting the servers and end users. Agentless approaches, without the need for any software on the managed nodes, are available. Data center managers should also look for solutions that are easily integrated, such as those based on Web Services Description Language APIs, and able to coexist with other applications on the designated host server or virtual machine.

Where Power is Going

Today, the goal is improved efficiency and reduced costs, but energy management will become even more critical in the future, as compute models continue to tax power infrastructures. Whatever the goal, the monitoring and aggregation of server energy metrics set the stage for much more comprehensive energy management and a far deeper and richer set of usage models for IT assets. Besides enabling accurate power planning and forecasting, logging and trending power data provides knowledge for data center “right-sizing” and accurate equipment scheduling to meet workload demands.

The thermal data can also be used for more efficient designs of integrated facilities systems, such as cooling and air-flow solutions. Optimized resource balancing in the data center will always be closely tied to power; the expanded insight offered by intelligent energy management approaches will contribute to cost-saving decisions for years to come.

Jeff Klaus is the director of Intel Data Center manager (DCM). Jeff leads a global team that designs, builds, sells, and supports Intel® DCM.