1 2 3 Previous Next

The Data Stack

1,595 posts

iStock_000024629815_Large.jpg

 

Here’s a prediction for 2016: The year ahead will bring the increasing “cloudification” of enterprise storage. And so will the years that follow—because cloud storage models offer the best hope for the enterprise to deal with unbounded data growth in a cost-effective manner.

 

In the context of storage, cloudification refers to the disaggregation of applications from the underlying storage infrastructure. Storage arrays that previously operated as silos dedicated to particular applications are treated as a single pool of virtualized storage that can be allocated to any application, anywhere, at any time, all in a cloud-like manner. Basically, cloudification takes today’s storage silos and turns them on their sides.

 

There are many benefits to this new approach that pools storage resources. In lots of ways, those benefits are similar to the benefits delivered by pools of virtualized servers and virtualized networking resources. For starters, cloudification of storage enables greater IT agility and easier management, because storage resources can now be allocated and managed via a central console. This eliminates the need to coordinate the work of teams of people to configure storage systems in order to deploy or scale an application. What used to take days or weeks can now be done in minutes.

 

And then there are the all-important financial benefits. A cloud approach to storage can greatly increase the utilization of the underlying storage arrays. And then there are the all-important financial benefits. A cloud approach to storage can greatly increase the utilization of the storage infrastructure; deferring capital outlays and reducing operational costs.

 

This increased utilization becomes all the more important with ongoing data growth. The old model of continually adding storage arrays to keep pace with data growth and new data retention requirements is no longer sustainable. The costs are simply too high for all those new storage arrays and the data center floor space that they consume. We now have to do more to reclaim the value of the resources we already have in place.

 

Cloudification isn’t a new concept, of course. The giants of the cloud world—such as Google, Facebook, and Amazon Web Services—have taken this approach from their earliest days. It is one of their keys to delivering high-performance data services at a huge scale and a relatively low cost. What is new is the introduction of cloud storage in enterprise environments. As I noted in my blog on non-volatile memory technologies, today’s cloud service providers are, in effect, showing enterprises the path to more efficient data centers and increased IT agility.

 

Many vendors are stepping up to help enterprises make the move to on-premises cloud-style storage. Embodiments of the cloudification concept include Google’s GFS and its successor Colossus, Facebook’s HDFS, Microsoft’s Windows Azure Storage (WAS), Red Hat’s Ceph/Rados (and GlusterFS), Nutanix’s Distributed File System (NDFS), among many others.

 

The Technical View

 

At this point, I will walk through the architecture of a cloud storage environment, for the benefit of those who want the more technical view.

 

Regardless of the scale or vendor, most of the implementations share the same storage system architecture. That architecture has three main components: a name service, a two-tiered storage service, and a replicated log service. The architectural drill-down looks like this:

 

The “name service” is a directory of all the volume instances currently being managed. Volumes are logical data containers, each with a unique name—in other words a namespace of named-objects. A user of storage services attaches to their volume via a directory lookup that resolves the name to the actual data container.

 

This data container actually resides in a two-tier storage service. The frontend tier is optimized for memory. All requests submitted by end-users are handled by this tier: metadata lookups as well as servicing read requests out of cache and appending write operations to the log.

 

The backend tier of the storage service provides a device-based, stable store. The tier is composed of a set of device pools, each pool providing a different class of service. Simplistically, one can imagine this backend tier supporting two device pools. One pool provides high performance but has a relatively small amount of capacity. The second pool provides reduced performance but a huge amount of capacity.

 

Finally, it is important to tease out the frontend tier’s log facility as a distinct, 3rd component. This is because this facility key to being able to support performant write requests while satisfying data availability and durability requirements.

 

In the weeks ahead, I will take up additional aspects of the cloudification of storage. In the meantime, you can learn about things Intel is doing to enable this new approach to storage at intel.com/storage.

DCD Magazine, contributed article

 

While every facet of data center management is changing at a rapid pace, operating budgets rarely keep up. Data volume doubles every 18 months and applications every two years; in contrast, operating budgets take eight years to double (IDC Directions, 2014).

 

IT has always been asked to do more with less, but the dynamic nature of the data center has been accelerating in recent years. Smart devices, big data, virtualization, and the cloud continue to change service delivery models and elevate the importance of flexibility, elasticity, and scalability.

 

Every facet of data center management, as a result, has been complicated by an incredibly rapid rate of change. Thousands of devices move on and off intranets. Fluid pools of compute resources are automatically allocated. Does this ultra-dynamic environment make it impossible for IT and facilities management teams to identify under-utilized and over-stressed resources?

 

If so, energy consumption in the data center will continue to skyrocket. And data centers already consume 10 percent of all energy produced around the globe, according to recent Natural Resources Defense Council reports.)

 

Fortunately, IT is far from powerless even within these challenging data center conditions.

 

Discovering some secret weapons

 

Ironically, in today’s data centers consisting of software-defined resources, the secret weapon for curbing energy costs lies in the hardware. Rack and blade servers, switches, power distribution units, and many other data center devices provide a wealth of power and temperature information during operation. Data center scale and the diversity of the hardware make it too cumbersome to manually collect and apply this information, which has led to a growing ecosystem of energy management solution providers.

 

Data center managers, as a result, have many choices today. They can take advantage of a management console that integrates energy management, have an integrator add energy management middleware to an existing management console, or independently deploy an energy management middleware solution to gain the necessary capabilities.

 

Regardless of the deployment option, a holistic energy management solution allows IT and facilities teams to view, log, and analyze energy and temperature behaviors throughout the data center. Automatically collected and aggregated power and thermal data can drive graphical maps of each room in a data center, and data can be analyzed to identify trends and understand workloads and other variables.

 

Visibility and the ability to log energy information equips data center managers to answer basic questions about consumption, and make better decisions relating to data center planning and optimization efforts.

 

Best-in-class energy management solutions take optimization to a higher level by combining automated monitoring and logging with real-time control capabilities. For example, thresholds can be set to cap power for certain servers or racks at appropriate times or when conditions warrant. Servers that are idle for longer than a specified time can be put into power-conserving sleep modes. Power can be allocated based on business priorities, or to extend the life of back-up power during an outage. Server clock rates can even be adjusted dynamically to lower power consumption without negatively impacting service levels or application performance.

 

Energy-conscious data centers take advantage of these capabilities to meet a broad range of operating objectives including accurate capacity planning, operating cost reduction, extending the life of data center equipment, and compliance with “green” initiatives.

 

Common uses and proven results

 

Customer deployments highlight several common motivations, and provide insights in terms of the types and scale of results that can be achieved with a holistic energy management solution and associated best practices.

 

  • Power monitoring. Identifying and understanding peak periods of power use motivate many companies to introduce an energy management solution. The insights gained have allowed customers to reduce usage by more than 15 percent during peak hours, and to reduce monthly data center utility bills even as demands for power during peak periods goes up. Power monitoring is also being applied to accurately charge co-location and other service users.

  • Increasing rack densities. Floor space is another limiting factor for scaling up many data centers. Without real-time information, static provisioning has traditionally relied upon power supply ratings or derated levels based on lab measurements. Real-time power monitoring typically proves that the actual power draw comes in much lower. With the addition of monitoring and power capping, data centers can more aggressively provision racks and drive up densities by 60 to more than 80 percent within the same power envelope.

  • Identifying idle or under-used servers. “Ghost” servers draw as much as half of the power used during peak workloads. Energy management solutions have shown that 10 to 15 percent of servers fall into this category at any point in time, and help data center managers better consolidate and virtualize to avoid this wasted energy and space.

  • Early identification of potential failures. Besides monitoring and automatically generating alerts for dangerous thermal hot spots, power monitoring and controls can extend UPS uptime by up to 15 percent and prolong business continuity by up to 25 percent during power outages.

  • Advanced thermal control. Real-time thermal data collection can drive intuitive heat maps of the data center without adding expensive thermal sensors. Thermal maps can be used to dramatically improve oversight and fine-grained monitoring (from floor level to device level). The maps also improve capacity planning, and help avoid under- and over-cooling. With the improved visibility and threshold setting, data center managers can also confidently increase ambient operating temperatures. Every one-degree increase translates to 5 to 10 percent savings in cooling costs.

  • Balancing power and performance. Trading off raw processor speed for smarter processor design has allowed data centers to decrease power by 15 to 25 percent with little or no impact on performance.


Time to get serious about power

 

Bottom line, data center hardware still matters. The constantly evolving software approaches for mapping resources to applications and services calls for real-time, fine-grained monitoring of the hardware. Energy management solutions make it possible to introduce this monitoring, along with power and thermal knobs that put IT and facilities in control of energy resources that already account for the largest line item on the operating budget.

 

Software and middleware solutions that allow data center managers to keep their eyes on the hardware and the environmental conditions let automation move ahead full speed, safely, and affordably – without skyrocketing utility bills. Power-aware VM migration and job scheduling should be the standard practice in today’s power-hungry data centers.

Software-defined storage (SDS) and hyper-convergence provide a flexible and efficient alternative to traditional “one server, one application” Storage Area Network (SAN) and Network Attached Storage (NAS) configurations. Hyper converged infrastructure is a growing trend. So what is hyper-convergence, and is it right for your business?

 

A scale-out, hyper-converged system allows the management of compute, storage and network resources as a single integrated system through a common tool set. Virtual servers, virtual networks and virtual storage are converged within a single piece of standard IA server, along with tools for management and security. Bottlenecks of traditional system (where compute, storage and networking are all separate physical resources) essentially don’t exist with hyper-converged systems. Everything is (virtually) “in the box”. Compute, storage and network capacity grow (“scale out”) by simply adding more hyper-converged systems as business needs demand. It’s a pay-as-you-grow approach to building capacity, and helps to better manage the cost of IT.

 

Hyper-converged scale-out systems differ from the older scale-up approach. In a scale-up system, the compute capacity is confined as storage is added, while in a scale-out system, new compute nodes can be added as the need for compute and storage arises. Scaling-up systems have often been cost prohibitive and often lacks the necessary random IO performance (IOPS) needed by virtualized workloads. The scale-out approach is a more efficient use of hardware resources, as it moves the data closer to the processor. When scale-out is combined with solid-state drive (SSD) storage it offers far lower latency, better throughput, and increased flexibility to grow with your business. Scale-out is commonly used for virtualized workloads, private cloud, data bases, and many other business applications.

 

So, with all of those benefits, why not spring for it and be done? Unfortunately, hyper-convergence has existed as a combination of disparate hardware and software, which comprise a “tool kit” to be assembled by technical staff. The skillset required is often beyond the capabilities of smaller businesses and the local IT and integration vendors who serve them. Many vendors offer complete, packaged “appliance” solutions … but the price tag is often high. So it’s been a bit of a “cool toys for big boys” story.

 

Now, StarWind Software has introduced a new hyper-converged appliance that brings the benefits of software defined scale-out storage and hyper-convergence to small business and remote-office/business-office installations. StarWind HCA, a turn-key hyper-converged appliance, delivers high performance compute and storage powered by Intel® SSD Data Center Family drives.

 

Intel® SSDs are designed with fast, consistent performance for smooth data center operation. The architecture of Intel's SSDs ensure the entire read and write path and logical-block address (LBA) mapping has data protection and error correction. Many enterprise workloads depend not only on reliable data, but consistency in how quickly that data can accessed.  Consistent latency, quality of service, and bandwidth no matter what background activities are happening on the drive is the basis of the Intel Data Center Family.  Rigorous testing ensures a highly reliable SSD with consistent performance.

 

StarWind HCA brings enterprise class features to your business scale-out storage and hyper-convergence “for the rest of us”. This turn-key appliance - featuring Intel® SSD Data Center Family drives - eliminates scalability and performance bottlenecks and allows computing and storage capacity to grow with the business needs.

teachermatch-cover.jpg

Elementary and secondary school principals must solve a challenging optimization problem. Faced with a deluge of applicants for teaching positions, demanding teaching environments, and very little time to spend on the applicant review process, school principals need a search algorithm with ranking analytics to help them find the right candidates. This is a classic data science problem.

 

Elsewhere, I have described the ideal data scientist, a balanced mix of math and statistics sophistication, programming chops, and business savvy: A rare combination, indeed. To solve the teacher applicant ranking problem, does every school in the country need to hire one of these “unicorn” data scientists to create a system to automatically identify the best teacher candidates?

 

I propose that the answer is “No!” and Big Data startup TeacherMatch agrees with me.  It is not a good use of resources for every school to hire a data scientist to help analyze teacher applications with advanced analytics, natural language processing and machine learning, yet the need to make teacher candidate selection more effective and efficient is huge.  The solution is to leverage the work of an expert who has already done that analysis.

 

TeacherMatch is such an expert. Based on a huge amount of historical data, TeacherMatch has developed a score for ranking teacher applicants, the EPI score, based upon a prediction of how likely a candidate is to be successful in the environment that the principal is looking to fill. Suddenly, it is nearly instantaneous to identify the top handful of candidates out of a list of potentially hundreds of applicants.

 

 

I met Don Fraynd, CEO of TeacherMatch, last year when he joined a [data science] (http://www.cio.com/article/2931082/data-analytics/discussing-data-science.html).  panel that I hosted I was impressed with his deep understanding of the challenges of hiring good teachers and with his practical approach to analytics. He has created a big data analytics solution that is sensible to incorporate into any organization that needs to hire teachers. See for yourself in this very interesting video about TeacherMatch.

 

Looking more broadly at the needs of all industry for more powerful analytics, the shortage of data scientists available to hire is a challenge. TeacherMatch’s model represents a real solution. In fact, I suspect that analytics-as-a-service will help drive a new era of advanced data analytics because it allows business users of analytics to leverage the output of a small number of data scientists who solve common problems across different organizations. In this regard, TeacherMatch represents the future of analytics.

 

Through the example of TeacherMatch, it appears that our principals and teachers are taking us to school on analytics.

iStock_000065756513_Large.jpg

In May 2015, I wrote the first in a series of blog posts exploring the journey to software-defined infrastructure. Each blog in the series dives down into different stages of a maturity model that leads from where we are today in the typical enterprise data center to where we will be tomorrow and in the years beyond.  During that time, I also delved into the workloads that will run on the SDI platform.

 

As I noted in last month’s post, traditional analytics leads first to reactive and then to predictive with the ultimate destination of prescriptive analytics.  This state is kind of the nirvana of today’s big data analytics and the topic we will take up today.

 

Prescriptive analytics extends beyond the predictive stage by defining the actions necessary to achieve outcomes and the inter-relationship of the outcomes to the effects of each decision. It incorporates both structured and unstructured data and uses a combination of advanced analytics techniques and other scientific disciplines to help organizations predict, prescribe, and adapt to changes that occur.  Essentially, we’ve moved from, “why did this happen,” to, “what will happen,” and we’re now moving to, “how do we make this happen,” as an analytics methodology.

 

Prescriptive analytics allows an organization to extract even more value and insight from big data—way above what we are getting today. This highest-level of analytics brings together varied data sources in real time and makes adjustments to the data and decisions on behalf of an organization. Prescriptive analytics is inherently real-time—it is always triggering these adjustments based on new information.

 

 

Let’s take few simple examples to make this story more tangible.

 

  • In the oil and gas industry, it can be used to enable natural gas price prediction and identify decision options—such as term locks and hedges against downside risk—based on an analysis of variables like supply, demand, weather, pipeline transmission, and gas production. It might also help decide when and where to harvest the energy, perhaps even spinning up and shutting down sources based on a variety of environmental and market conditions.

     

  • In healthcare, it can increase the effectiveness of clinical care for providers and enhance patient satisfaction based on various factors across stakeholders as a function of healthcare business process changes.  It could predict patient outcomes and help alleviate issues before they would normally even be recognized by medical professionals.

     

  • In the travel industry, it can be used to sort through factors like demand curves and purchase timings to set seat prices that will optimize profits without deterring sales.  Weather and market conditions could better shape pricing and fill unused seats and rooms while relieving pressure in peak seasons.

     

  • In the shipping industry, it can be used to analyze data streams from diverse sources to enable better routing decisions without the involvement of people. In practice, this could be as simple as a system that automatically reroutes packages from air to ground shipment when weather data indicates that severe storms are likely to close airports on the usual air route.

 

I could go on and on with the examples, because every industry can capitalize on prescriptive analytics. The big takeaway here is that prescriptive analytics has the potential to turn enormous amounts of data into enormous business value—and do it all in real time.

 

With the impending rise of prescriptive analytics, we are entering the era in which machine learning, coupled with automation and advanced analytics, will allow computers to capture new insights from massive amounts of data in diverse datasets and use that data to make informed decisions on our behalf on an ongoing basis.

 

At Intel, we are quite excited about the potential of prescriptive analytics. That’s one of the reasons why we are a big backer of the open source Trusted Analytics Platform (TAP) initiative, which is designed to accelerate the creation of cloud-native applications driven by big data analytics. TAP is an extensible open source platform designed to allow data scientists and application developers to deploy solutions without having to worry about infrastructure procurement or platform setup, making analytics and the adoption of machine learning easier. To learn about TAP, visit trustedanalytics.org.

This article originally appeared on Converge Digest

 

 

We're in the depth of winter and, yes, the snow can be delightful… until you have to move your car or walk a half block on icy streets. Inside the datacenter, the IT Wonderland might lack snowflakes but everyday activities are even more challenging year round. Instead of snowdrifts and ice, tech teams are faced with mountains of data.

klaus-blog-image.png

 

So what are the datacenter equivalents of snowplows, shovels, and hard physical labor? The right management tools and strategies are essential for clearing data paths and allowing information to move freely and without disruption.

 

This winter, Intel gives a shout-out to the unsung datacenter heroes, and offers some advice about how to effectively avoid being buried under an avalanche of data. The latest tools and datacenter management methodologies can help technology teams overcome the hazardous conditions that might otherwise freeze up business processes.

 

Tip #1: Take Inventory

 

Just as the winter holiday season puts a strain on family budgets, the current economic conditions continue to put budget pressures on the datacenter. Expectations, however, remain high. Management expects to see costs go down while users want service improvements. IT and datacenter managers are being asked to do more with less.

 

The budget pressures make it important to fully assess and utilize the in-place datacenter management resources. IT can start with the foundational server and PDU hardware in the datacenter. Modern equipment vendors build in features that facilitate very cost-effective monitoring and management. For example, servers can be polled to gather real-time temperature and power consumption readings.

 

Middleware solutions are available to take care of collecting, aggregating, displaying, and logging this information, and when combined with a management dashboard can give datacenter managers insights into the energy and temperature patterns under various workloads.

 

Since the energy and temperature data is already available at the hardware level, introducing the right tools to leverage the information is a practical step that can pay for itself in the form of energy savings and the ability to spot problems such as temperature spikes so that proactive steps can be taken before equipment is damaged or services are interrupted.

 

Tip #2: Replace Worn-Out Equipment

 

While a snow shovel can last for years, datacenter resources are continually being enhanced, changed, and updated. IT needs tools that can allow them to keep up with requests and very efficiently deploy and configure software at a rapid pace.

 

Virtualization and cloud architectures, which evolved in response to the highly dynamic nature of the datacenter, have recently been applied to some of the most vital datacenter management tools. Traditional hardware keyboard, video, and mouse (KVM) solutions for remotely troubleshooting and supporting desktop systems are being replaced with all-software and virtualized KVM platforms. This means that datacenter managers can quickly resolve update issues and easily monitor software status across a large, dynamic infrastructure without having to continually manage and update KVM hardware.

 

Tip #3: Plan Ahead

 

It might not snow everyday, even in Alaska or Antarctica. In the datacenter, however, data grows everyday. A study by IDC, in fact, found that data is expected to double in size every two years, culminating in 44 zettabytes by 2020. An effective datacenter plan depends on accurate projections of data growth and the required server expansion for supporting that growth.

 

The same tools that were previously mentioned for monitoring and analyzing energy and temperature patterns in the datacenter can help IT and datacenter architects better understand workload trends. Besides providing insights about growth trends, the tools promote a holistic approach for lowering the overall power budget for the datacenter and enable datacenter teams to operate within defined energy budget limits. Since many large datacenters already operate near the limits of the local utility companies, energy management has become mission critical for any fast-growing datacenter.

 

Tip #4: Stay Cool

 

Holiday shopping can be a budget buster, and the credit card bills can be quite a shock in January. In the datacenter, rising energy costs and green initiatives similarly strain energy budgets. Seasonal demands, which peak in both summer and the depths of winter, can mean more short-term outages and big storms that can force operations over to a disaster recovery site.

 

With the right energy management tools, datacenter and facilities teams can come together to maximize the overall energy efficiency for the datacenter and the environmental conditions solutions (humidity control, cooling, etc.). For example, holistic energy management solutions can identify ghost servers, those systems that are idle and yet still consuming power. Hot spots can be located and workloads shifted such that less cooling is required and equipment life extended. The average datacenter experiences between 15 to 20 percent savings on overall energy costs with the introduction of an energy management solution.

 

Tip #5: Reading the Signs of the Times

 

During a blizzard, the local authorities direct the snowplows, police, and rescue teams to keep everyone safe. Signs and flashing lights remind everyone of the rules. In the datacenter, the walls may not be plastered with the rules, but government regulations and compliance guidelines are woven into the vital day-to-day business processes.

 

Based on historical trends, regulations will continue to increase and datacenter managers should not expect any decrease in terms of required compliance-related efforts. Public awareness about energy resources and the related environment impact surrounding energy exploration and production also encourage regulators.

 

Fortunately, the energy management tools and approaches that help improve efficiencies and lower costs also enable overall visibility and historical logging that supports audits and other compliance-related activities.

When “politically correct” behavior and cost savings go hand in hand, momentum builds quickly. This effect is both driving demand for and promoting great advances in energy management technology, which bodes well for datacenter managers since positive results always depend on having the right tools. And when it comes to IT Wonderlands, energy management can be the equivalent of the whole toolshed.

To see the challenge facing the network infrastructure industry, I have to look no farther than the Apple Watch I wear on my wrist.

 

That new device is a symbol of the change that is challenging the telecommunications industry. This wearable technology is an example of the leading edge of the next phase of the digital service economy, where information technology becomes the basis of innovation, services and new business models.

 

I had the opportunity to share a view on the end-to-end network transformation needed to support the digital service economy recently with an audience of communications and cloud service providers during my keynote speech at the Big Telecom Event.

 

These service providers are seeking to transform their network infrastructure to meet customer demand for information that can help grow their businesses, enhance productivity and enrich their day-to-day lives.  Compelling new services are being innovated at cloud pace, and the underlying network infrastructure must be agile, scalable, and dynamic to support these new services.

 

The operator’s challenge is that the current network architecture is anchored in purpose-built, fixed function equipment that is not able to be utilized for anything other than the function for which it was originally designed.  The dynamic nature of the telecommunications industry means that the infrastructure must be more responsive to changing market needs. The challenge of continuing to build out network capacity to meet customer requirements in a way that is more flexible and cost-effective is what is driving the commitment by service providers and the industry to transform these networks to a different architectural paradigm anchored in innovation from the data center industry.

 

Network operators have worked with Intel to find ways to leverage server, cloud, and virtualization technologies to build networks that cost less to deploy, giving consumers and business users a great experience, while easing and lowering their cost of deployment and operation.

 

Transformation starts with reimagining the network

 

This transformation starts with reimagining what the network can do and how it can be redesigned for new devices and applications, even including those that have not yet been invented. Intel is working with the industry to reimagine the network using Network Functions Virtualization (NFV) and Software Defined Networking (SDN).

 

For example, the evolution of the wireless access network from macro basestations to a heterogeneous network or “HetNet”, using a mix of macro cell and small cell base-stations, and the addition of mobile edge computing (MEC) will dramatically improve network efficiency by providing more efficient use of spectrum and new radio-aware service capabilities.  This transformation will intelligently couple mobile devices to the access network for greater innovation and improved ability to scale capacity and improve coverage.

 

In wireline access, virtual customer premises equipment moves service provisioning intelligence from the home or business to the provider edge to accelerate delivery of new services and to optimize operating expenses. And NFV and SDN are also being deployed in the wireless core and in cloud and enterprise data center networks.

 

This network transformation also makes possible new Internet of Things (IoT) services and revenue streams. As virtualized compute capabilities are added to every network node, operators have the opportunity to add sensing points throughout the network and tiered analytics to dynamically meet the needs of any IoT application.

 

One example of IoT innovation is safety cameras in “smart city” applications. With IoT, cities can deploy surveillance video cameras to collect video and process it at the edge to detect patterns that would indicate a security issue. When an issue occurs, the edge node can signal the camera to switch to high-resolution mode, flag an alert and divert the video stream to a central command center in the cloud. With smart cities, safety personnel efficiency and citizen safety are improved, all enabled by an efficient underlying network infrastructure.

 

NFV and SDN deployment has begun in earnest, but broad-scale deployment will require even more innovation: standardized, commercial-grade solutions must be available; next-generation networks must be architected; and business processes must be transformed to consume this new paradigm. Intel is investing now to lead this transformation and is driving a four-pronged strategy anchored in technology leadership: support of industry consortia, delivery of open reference designs, collaboration on trials and deployments, and building an industry ecosystem.

 

The foundation of this strategy is Intel’s role as a technology innovator. Intel’s continued investment and development in manufacturing leadership, processor architecture, Ethernet controllers and switches, and optimized open source software provide a foundation for our network transformation strategy.

 

Open standards are a critical to robust solutions, and Intel is engaged with all of the key industry consortia in this industry, including the European Telecommunications Standards Institute (ETSI), Open vSwitch, Open Daylight, OpenStack, and others. Most recently, we dedicated significant engineering and lab investments to the Open Platform for NFV’s (OPNFV) release of OPNFV Arno, the first carrier-grade, open source NFV platform.

 

The next step for these open source solutions is to be integrated with operating systems and other software into open reference software to provide an on-ramp for developers into NFV and SDN. That’s what Intel is doing with our Open Network Platform (ONP); a reference architecture that enables software developers to lower their development cost and shorten their time to market.  The innovations in ONP form the basis of many of our contributions back to the open source community. In the future, ONP will be based on OPNFV releases, enhanced by additional optimizations and proofs-of-concept in which we continue to invest.

 

We also are working to bring real-world solutions to market and are active in collaborating on trials and deployments and deeply investing in building an ecosystem that brings companies together to create interoperable solutions.

 

As just one example, my team is working with Cisco Systems on a service chaining proof of concept that demonstrates how Intel Ethernet 40GbE and 100GbE controllers, working with a Cisco UCS network, can provide service chaining using network service header (NSH).  This is one of dozens of PoCs that Intel has participated in in just this year, which collectively demonstrate the early momentum of NFV and SDN and its potential to transform service delivery.

 

A lot of our involvement in PoCs and trials comes from working with our ecosystem partners in the Intel Network Builders. I was very pleased to have had the opportunity to share the stage with Martin Bäckström and announce that Ericsson has joined Network Builders. Ericsson is an industry leader and innovator, and their presence in Network Builders demonstrates a commitment to a shared vision of end-to-end network transformation.

 

The companies in this ecosystem are passionate software and hardware vendors, and also end users, that work together to develop new solutions. There are more than 150 Network Builder members taking advantage of this program and driving forward with a shared vision to accelerate the availability of commercial grade solutions.

 

NFV and SDN are deploying now - but that is just the start of the end-to-end network transformation. There is still a great deal of technology and business innovation required to drive NFV and SDN to scale, and Intel will continue its commitment to drive this transformation.



I invited the BTE audience – and I invite you – to join us in this collaboration to create tomorrow’s user experiences and to lay the foundation for the next phase of the digital services economy.

Internet of Things (IoT) technologies from Intel and SAP enable innovative solutions far from the data center

 

Can a supervisor on an oil rig know immediately when critical equipment fails? Can a retail store manager provide customers with an up-to-date, customized ex¬¬perience without waiting for back-end analysis from the parent company’s data center? A few years ago, the answer would have been a clear “no.” But today, real-time, actionable data at the edge is a reality.

 

Innovative technologies from Intel and SAP can enable automated responses and provide critical insights at remote locations. The unique joint solutions enable companies to dramatically improve worker safety, equipment reliability, and customer engagement, all without an infrastructure overhaul. For example, technicians on a remote, deep-sea oil rig can be equipped with sensors that detect each technician’s location, heart rate, and exposure to harmful gasses. Additionally, sensors powered by Intel Quark SoCs can be placed on equipment throughout the oil rig to monitor for leaks or fires. The collective data from these sensors is fed to an Intel IoT Gateway and processed to provide data visualization and a browser interface that is easily accessible from any device.

 

From any location on the rig with Wi-Fi access, supervisors can monitor worker health and safety data from an app running on a tablet device. In addition, automated alerts and alarms can signal when an employee is in danger or a critical malfunction has occurred. All of this processing can happen in real time, on-site, without depending on a reliable wide-area network (WAN) connection to a back-end server that might be hundreds or thousands of miles away.

 

When the WAN connection is available, the SAP Remote Data Sync service synchronizes data with SAP HANA running in the cloud or in the data center. This synchronization provides cloud-based reporting and back-end SAP integration for long-range analysis

 

sap-iot-blog-image1.jpg

 

With IoT sensors, Intel IoT Gateway, and SAP software, businesses can improve safety and gain real-time insights right at the edge.

 

To learn more about the joint Intel and SAP solution at the edge, read the solution brief Business Intelligence at the Edge.

Tis the season to go shopping—at least if you are one of the millions of people buying gifts for the holidays. If so, perhaps you’ve noticed that your shopping strategies have changed in recent years. Fewer and fewer people go window shopping or wander the merchandise aisles, seeking inspiration. Instead, the retail experience for many begins by researching products with web and mobile tools before going to a bricks-and-mortar store to make further evaluations and a purchase decision: While more than 90 percent of retail transactions still occur in stores, over 60 percent of purchase research begins online. That means that consumers come to retail locations better informed than ever, often with their minds already made up about what they wish to purchase.

 

This can be a challenge for retailers who rely on traditional marketing strategies that focus on sales associates engaging with customers and helping make purchase decisions—with a consequent increase in the size of the sales basket.

 

How can a retailer compete? Well, it turns out that consumers aren’t the only ones with access to new sources of data. Innovative retailers can use new technologies—including a network of smart sensors, Internet of Things (IoT) gateways, and cloud-based data analytics—to know more about individual consumers and their preferences; personalize the customer’s shopping experience to increase engagement and sales; and to deliver a unified retail experience from online to in-store via smart signage, loyalty programs, targeted sales assistance, mobile POSs, and more.

 

To enable these kinds of data-driven experiences for retail requires a secure, end-to-end solution that can seamlessly extend from sensor-based data capture at the store level, to transaction data collected at retail locations, and all the way to analytics processing in the cloud—to help retailers improve engagement with their customers, better understand what’s happening in their stores, and improve the size of individual transactions. To help retailers gain these kinds of data-driven insights, Intel and SAP are working together to create an end-to-end IoT solution that can deliver actionable retail insights on a near real-time basis. This joint IoT solution, which includes Intel® IoT Gateways to capture and secure sensor data, SAP SQL Anywhere* database software to enable intelligence at the edge, and the SAP HANA* cloud, will allow retailers to take advantage of real-time analytics to act faster, make smarter decisions, and know more about customers than ever before. To learn more about the Intel and SAP IoT solution, watch this video:

 

 

When we talk about bringing improved analytics to the retail industry, I want to emphasize that this new solution enables two very different types of analytics. The first is conventional big data analytics, the kind of number crunching that typically takes place in corporate offices and allows high-level retail decision-makers to decide where to locate the next store and view sales patterns for the season’s hot products. This kind of analytics has traditionally been processed on its own siloed computing infrastructure, completely apart from the transactional infrastructure that runs the sales and transactional systems for daily retail operations, because you wouldn’t risk running a big analytics query on the same system that held your transactions log, for fear that it might crash the system. But SAP HANA, running on Intel® Xeon® processors, has the power to run both analytical and transactional workloads on the same infrastructure, and in memory. This not only offers the potential to lower infrastructure costs, it means that transactional information can be made available for analysis as soon as it is processed, enabling retailers to act on those insights in near real-time.

 

The Intel and SAP IoT solution doesn’t just enable analytics at the macro level, it also delivers immediate analytical insights at the store level. Using sensor-based tools such as smart-mirrors and –fixtures, traffic flow detectors, interactive visual aids, inventory tags, and personal tracking technologies, Intel IoT Gateways can give retailers instant access to new sources of data that can identify customers and their past purchases and preferences, and to offer personalized offerings and services to differentiate the shopping experience from competitors. These instantaneous analytics, drawn from the store itself, deliver insights not just about customers, but also about how to offer a more engaged customer experience in a store. This can include store layout and traffic flow information, suggestions for strategic pairings of products, identification of inventory not selling well, and alerts about up-to-the-second shopping trends. Collection and real-time analysis of data at the network edge gives a store manager the ability to immediately address dynamics in the store; the IoT gateway also routes the data to the cloud for traditional analytics.

 

Intel products and technologies translate IoT into business value and differentiation for the retail industry. Intel provides the critical, foundational building blocks for building secure IoT networks, with a proven portfolio of compute, storage, network and software components that span from the edge to the cloud. Because the Intel platform is modular, retailers can build their infrastructure as their needs evolve; they can also unlock value from the data and infrastructure they already have.  Essential for retailers, Intel also offers a variety of security technologies built into its processors, including encryption and tokenization, to ensure protection of both customer and retail data. Intel is also at the center of a partner ecosystem that includes not only SAP but retail industry ISVs, and our technology is vendor neutral, standards based, and interoperable.

 

Intel may not be the first company that comes to mind when you think of retail, but if you peer behind the thin veneer that is the storefront, you’ll find that that the technology—and the innovation—that powers retail solutions are similar to the platforms in other industries where Intel is a proven leader.

 

Interested in learning more about Intel’s retail solutions? Visit our website and join us at the National Retail Federation (NRF) annual conference, to be held January 17-20. 2016 in New York City. Stop by the Intel booth #2543 to say hello and let us show you how Intel is revolutionizing retail.

 

Follow me at @TimIntel and #TechTim to keep up with the latest with Intel and SAP.

I recently gave a talk at SNIA’s annual Storage Developer Conference (SDC), filling in for a colleague who was unable to travel to the event.  The talk, “Big Data Analytics on Object Storage—Hadoop Over Ceph Object Storage with SSD Cache,” highlighted work Intel is doing in the open source community to enable end users to fully realize the benefits of non-volatile memory technologies employed in object storage platforms[1].

 

In this post, I will walk through some of the key themes from the SNIA talk, with one caveat: This discussion is inherently technical. It is meant for people who enjoy diving down into the details of storage architectures.

 

Let’s begin with the basics. Object storage systems, such as Amazon Web Service’s Simple Storage Service (S3) or the Hadoop Distributed File System (HDFS), consist of two components: an access model that adheres to a well-specified interface and a non-POSIX-compliant file system.

 

In the context of this latter component, storage tiering has emerged as a required capability. For the purpose of this discussion, I will refer to two tiers: hot and warm. The hot tier is used to service all end-user-initiated I/O requests while the warm tier is responsible for maximizing storage efficiency. This data center storage design is becoming increasingly popular.

 

Storage services in a data center are concerned with the placement of data on the increasingly diverse storage media options available to them. Such data placement is an economic decision based on the performance and capacity requirements of an organization’s workloads. On the $-per-unit-of-performance vector there is often significant advantage to placing data in DRAM or on a storage media such as 3D XPoint that has near-DRAM speed. On the dollar-per-unit-capacity vector there is great motivation to place infrequently accessed data on the least expensive media, typically rotational disk drives but increasingly 3D NAND is an option.

 

With the diversity of media: DRAM, 3D XPoint, NAND, and Rotational, the dynamic of placing frequently accessed, so-called “hot” data on higher performing media while moving less frequently accessed, “warm” data to less expensive media is increasingly important. How is data classified, in terms of its “access frequency?” How are data placement actions carried out based on this information? We’ll look more closely on the “how” in a subsequent post. The focus of this discussion is on data classification. Specifically, we look at data classification within the context of distributed block and file storage systems deployed in a data center.

 

Google applications such as F1, a distributed relational database system built to support their AdWords business and Megastore, a storage system developed to support online services such as Google Application and Compute Engine [2,3]. These applications are built on top of Spanner and BigTable respectively. In turn, Spanner and BigTable store their data, b-tree-like files and write-ahead log to Colossus and Google File Systems respectively [4]. In the case of the Colossus File System (CFS), Google has written about using “Janus” to partition CFS's flash storage tier for workloads that benefit from the use of the higher performing media [5]. The focus of this work is on characterizing workloads to differentiate these based on a “cacheability” metric that measures cache hit rates. More recently, companies such as Cloud Physics and Coho Data have published papers along similar lines. The focus of this work is on characterizations that efficiently produce a [cache] "miss ratio curve (MRC)” [6-8] Like Google, the goal is to keep “hot” data in higher-performing media while moving less frequently accessed data to a lower cost media.

 

What feature of the data-center-wide storage architecture enables such characterization? In both Google’s and Coho’s approaches, the distinction between servicing incoming end-user I/O requests for storage services and accessing backend storage is fundamental. In Google’s case, applications such as F1 and Megastore indirectly layer on top of the distributed storage platform. However, the Janus abstraction is transparently interposed with such applications and the file system. Similarly, Coho presents a storage service, such as NFS mount points or HDFS mount points, to end-user applications via a network virtualization layer. [9,10] This approach allows for the processing pipeline to be inserted between the incoming end-user application I/O requests and Coho’s distributed storage backend.

 

One can imagine incoming I/O operation requests—such as create, delete, open, close, read, write, snapshot, record, and append—encoded in well-specified form. Distinguishing between, or classifying, incoming operations with regard to workload, operation type, etc. becomes analogous to logging in to an HTTP/web farm [11]. And like such web farms, read/access requests are readily directed toward caching facilities while write/mutation operations can be appended to a log.

 

In other words, from the end user’s perspective storage is just another data-center-resident distributed service, one of many running over shared infrastructure.

 

And what about the requisite data management features of the storage platform? While the end user interacts with a storage service, the backend storage platform is no longer burdened with this type processing. It is now free to focus on maximizing storage efficiency and providing stewardship over the life of an organization’s ever-growing stream of data.

 

 

References

 

  1. Zhou et al, “Big Data Analytics on Object Storage - Hadoop Over Ceph Object Storage with SSD Cache,” 2015.
  2. Shute et al, "F1: A Distributed SQL Database That Scales,” 2013 http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41344.pdf
  3. Baker et al, "Megastore: Providing Scalable, Highly Available Storage for Interactive Services,” 2011 http://www.cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf
  4. Corbett et al, "Spanner: Google’s Globally-Distributed Database,” 2012 (see section 2.1 Spanserver Software Stack for a discussion on how Spanner uses Colossus) http://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf
  5. Albrecht et al, “Janus: Optimal Flash Provisioning for Cloud Storage Workloads,” 2013.
  6. Waldspurger et al, "Efficient MRC Construction with SHARDS,” 2015 http://cdn1.virtualirfan.com/wp-content/uploads/2013/12/shards-cloudphysics-fast15.pdf
  7. Warfield, "Some experiences building a medium sized storage system,” 2015 (see specifically slide 21 "Efficient workload characterization”).
  8. Wires et al, “Characterizing Storage Workloads with Counter Stacks,” 2014.
  9. Cully et al, “Strata: High-Performance Scalable Storage on Virtualized Non-volatile Memory,” 2014.
  10. Warfield et al, “Converging Enterprise Storage and Business Intelligence: A Reference Architecture for Data-Centric Infrastructure,” 2015.
  11. Chen et al, "Client-aware Cloud Storage,” 2014 http://www.csc.lsu.edu/~fchen/publications/papers/msst14-cacs.pdf

I just returned from a trip to the future – or rather, I just returned from Autodesk University in Las Vegas, which is kind of the same thing. Autodesk University is one of the most future-focused industry events of the year, where the company, which builds 3D design and modeling software for the engineering, manufacturing, automotive, construction and entertainment industries, provides training, introduces new technologies, and explores coming disruptions in the fields of modeling and design.

 

Autodesk really pulled out the stops for this year’s show (there were even Star Wars storm troopers in the keynote), demonstrating how advanced design software and supercomputer processing power are poised to change the relationship between human designers and their design tools—and improving the safety, efficiency, and performance of our built world while they’re at it.

 

Intel had a large presence at the show as well – Autodesk and Intel work closely together, as the compute-intense workloads associated with cutting edge 3D design and modeling requires a cost-effective, high-performance processing platform such as those powered by Intel® Xeon® processor E3 with Iris Pro Graphics. And, as you’ll see, Intel is also a sponsor of some of Autodesk’s most exciting research and development (R&D) projects.

 

During the opening keynote address, Autodesk CTO Jeff Kowlaski introduced several fascinating R&D projects.

 

Project Dreamcatcher


Project Dreamcatcher is an experimental design platform that brings generative design systems and artificial intelligence (AI) into product design and modeling. The Dreamcatcher  system allows designers to input specific design objectives, such as functional requirements, material type, manufacturing method, performance criteria, and cost restrictions. The research system compares past designer experience and can evaluate a vast number of potential designs—enhanced by machine learning and AI—that satisfy your design goals and constraints. Dreamcatcher then comes back with ideas and designs that we humans by ourselves might never have imagined.  These generative designs are typically stronger, more efficient and potentially less expensive.

 

As an example, Kowlaski showed how the aircraft manufacturer Airbus used the Dreamcatcher system to create a new design for an Airbus A320 interior partition.airbus partition.jpg  After feeding in all the specs, design goals and constraints into Dreamcatcher, the system came back with several potential designs. After a collaborative back-and-forth between the human design team and the high-end system, the optimal design was chosen. Using generative design, additive manufacturing and advanced materials, Airbus engineers manufactured the new “bionic partition,” which is half the weight of the current partition yet even stronger. Airbus estimates that by redesigning the entire cabin of the A320 using these lighter and more optimized materials would cut fuel consumption and save half a million tons of CO2 emissions per plane, per year.

 

Primordial Project


Co-sponsored by Intel, the Primordial Research Project is another Autodesk initiative that Kowalski highlighted in his keynote.  For the Primordial Project, Autodesk teamed with the media company Bandito Brothers to leverage machine learning, generative design, pervasive computing and IoT sensors to prototype the first car ever designed by an AI system. The result? A new vehicle chassis design optimized for maximum driving performance and efficiency. Designed by AI, it could probably never have been devised by humans. We had a prototype of the Primordial Project chassis in the Intel booth at Autodesk University: Check out my Periscope video to have a look.

 

Powering the Graphics Future


What do all these futuristic graphics-heavy designs from Autodesk have in common? They are very CPU and GPU intensive. And that’s where Intel comes in. In the Intel booth at Autodesk University we demo-ed a number of solutions running the Intel® Xeon® processor E3 with Iris™ Pro Graphics technology built directly into the chip.

 

Not so long ago, if you wanted to combine raw computing power and rich visualizations (as required with graphics-heavy applications such as Autodesk’s simulation and animation software), you had to add a separate—and costly—graphics processing unit (GPU) to your workstation.

 

But with the Intel Xeon processor E3-based infrastructure, you get powerful graphics technology integrated directly into the processor, so additional GPUs aren’t required to deliver rich visualization of Autodesk simulation and animation software. This means you can deliver better and richer visualizations at a lower cost and TCO for graphics-intensive workloads. In fact, an Intel Xeon E3 v4 platform with built-in Iris Pro Graphics can deliver up to 90 percent of the performance of a standard GPU plug-in, but at about one-fifth of the cost. That’s cost-savings and performance that you can count on when you’re creating the future with graphics-heavy design and modeling software.


Intel’s presence at that show emphasized the cost effective value of the Intel® Xeon® E3 with Iris Pro Graphics vs. Nvidia Grid GPUs and the emerging Intel® RealSense cameras.  Intel Fellow & CTO of HPC Mark Seager participated in tech trends panel that highlighted high-end design with HPC based platforms.  Autodesk, Citrix, Boxx and HPE helped in booth by teaching 15 Tech talks.

 

Follow me at @TimIntel and #TechTim for the latest news on Intel and Autodesk.

A recent AnandTech post gave me pause. As the leader of Intel’s efforts in Visual Cloud which  include our Intel® Xeon® processor E3 family with integrated graphics, I  thought everyone knew that there were many (and growing number of) options for  getting Intel’s best graphics, Iris™ Pro Graphics in a server system.  Obviously, we’ve not done a good job of getting the word out. I’m using this  post to let you know what systems are in volume production that support our  latest (Intel® microarchitecture codename Broadwell) generation Iris  Pro Graphics and with it, Intel® Quick  Sync Video, Intel’s video processing acceleration technology.

 

A few caveats first: we don’t track all the systems that may  be out there so apologies if I’ve missed any. Also, more are coming from new  suppliers all the time so this blog represents a snapshot of what I know as of  yearend 2015. If you know of ones missed, feel free to post in the comment  space below. We also don’t endorse, promote, or certify any specific platform  or vendor or their capabilities. You’ll have to check directly with that  supplier with any questions about the level of support that they provide for  Intel’s flagship visual cloud technologies: Iris Pro Graphics and Quick Sync  Video.


A quick cheat to figure out if system might be able to  support Intel’s visual cloud technologies. If a server system supports the  Intel Xeon processor E3 v4 family and uses the Intel® C226 Chipset, it has the  basic capability to use our visual cloud technologies. This is not an absolute  answer because sometimes we’ve run into BIOS or other issues that prevent the  technologies from working. The ones listed below have been used for visual  cloud applications by solution providers and service providers. Typically, a  user would populate these servers with the Intel Xeon E3-1285L v4 processor to  get the maximum performance per watt. Please refer to the manufacturer’s web  site for more information on specific systems.


There are three main types of systems that enable Intel  visual cloud technologies.


 

 

  • Finally, in some environments, it’s useful to be  able to land Intel visual cloud technologies in a more general purpose server  system. For that purpose, there are two PCI Express* (PCIe) accelerator cards  available. These plug into standard PCIe slots that meet the specific space and  power requirements for the cards. A number of OEM and distributors provide  these cards pre-integrated into server systems.
    • Artesyn* SharpStreamer*  PCIE-7207 (Contains four Intel® Dual Core processor i7-5650U 2.2 GHz with Intel® HD Graphics 6000) and  SharpStreamer* Mini PCIE-7205 (contains 1 or 2 Intel® Dual Core processor  i5-5350U with Intel® HD Graphics 6000). These cards are known to be compatible  with:

 

I hope that gives you plenty of systems to choose from. If  you want more info out Intel’s efforts in Visual Cloud Computing, please see: http://www.intel.com/visualcloud.

You don’t have to look very hard to see how big data and powerful analytics tools are changing the world. Organizations ranging from leading-edge hospitals and smart cities to modernized farms have leapt ahead of the competition by thinking differently about how to use the business-enabling power of their data centers.

 

A few examples:

 

  • Healthcare providers around the world are now leveraging data analytics to deliver more targeted therapies and personalized medicine—and open up a new market measured in terms of billions of dollars.
  • Disaster management teams are using insights gained from real-time data analytics to know exactly where to target emergency response assistance for those in the hardest hit areas
  • Farm operators are using real-time data analysis to turn agriculture into a more precise science—and a more profitable business—with carefully targeted uses of fertilizers and irrigation water.

 

This list could go on and on, covering examples from industries as diverse as automobile engineering, entertainment, and oil and gas exploration. In virtually any industry, you can find examples of innovation that is taking place today by organizations that have recognized the potential to capitalize on data in new ways.

 

Of course, it takes more than big data to fuel innovation. It also takes what we might call big compute—powerful analytics tools, bigger-and-faster multicore processors, and low-latency fabrics that allow data to move quickly in and out of high-speed storage.

 

This is where Intel enters the innovation picture. Following the lead of computing pioneer Gordon Moore, of Moore’s Law fame, Intel continues to deliver the advances in processors, networking, storage, and management that organizations need to put big data to work in innovative ways.

 

Earlier this the year, for example, we launched the new Intel® Xeon® processor E7 v3 family, which delivers the world’s fastest system for real-time analytics on a trusted, mission-critical, highly scalable platform. And a little later in the year, together with Micron, we announced new 3D XPoint™ technology, which brings non-volatile memory speeds up to 1,000 times faster than NAND.

 

These and other innovations at the data center level fuel countless downstream innovations at the business level. They make it possible for organizations to think about putting data to work in novel ways.  We’re really talking about the data center as the new center of possibility.

 

At Hewlett Packard Enterprise (HPE) Discover 2015 conference in London, participants are received a close-up look at some of the exciting innovations that are taking place today at the intersection of big data and big compute. You can get your own close-up view by exploring The New Center of Possibility site on intel.com and checking out the video below.

 

In legacy IT landscapes, information is frequently siloed horizontally across business units or vertically by function. Critical data may reside in legacy IT systems, a data warehouse, or systems that are outside the corporate data center. This mix of data may be in structured, semi-structured, or unstructured formats.

 

For analytics purposes, operational data is typically replicated, reformatted, merged, and aggregated multiple times before initial analysis can take place. Depending on the data in question, this process might require hours, days, or weeks. This all adds up to a delay in drawing insights from the data.

 

By the time the data gets analyzed, its relevance has decreased and its costs have increased. Any actions taken on the insights gained though analytics will tend to be reactive rather than proactive. This puts the business in a state where it is continually looking back at what happened in the past rather than looking forward to predict what is likely to happen in the future.

 

time-to-insight-graphic.png

 

This high latency from event to insight is an unsustainable state of affairs. For businesses to compete effectively in today’s fast-moving, digitally driven marketplaces, they need to reduce the latency of the environment to just hours, minutes, or even seconds, and they need to focus more on predicting what lies ahead rather than reacting to what happened days, weeks, or months ago.

 

Here’s a case in point. A fast-growing company called FarmLogs is using real-time analytics to help growers leverage data collected by sensors in the field to increase crop production. With the ability to see current soil conditions, precipitation levels, and other field measurements, along with analysis of that data, farmers can adjust resources on any given day or moment – Get the full story at Data Science Technology Seeds Smarter Farming.

 

While this new imperative to shift to predictive analytics is a challenge for CIOs and IT departments, it is also an opportunity. It presents an ideal time for companies to get started on the path of transformation to new, data-driven business models.

 

And this is where we return to the software-defined infrastructure (SDI) story. SDI enables the movement from reactive to predictive analytics by providing a flexible and adaptive environment that allows you to deploy infrastructure on demand, and then deploy analytics on top of the software-defined foundation. This IT agility positions your organization to take advantage of new data as it comes in—which is often your most valuable data. It gives you access to the resources you need, when you need them, and where you need them.

 

SDI is an ideal complement to technologies like Hadoop, which allows you to combine diverse datasets and run queries and reports in real time, and in-memory analytics, which moves data closer to the processors. Capabilities like these accelerate time to insight with big data, and help you make predictive analytics a reality.

 

But, of course, the world won’t stop there. As we move from reactive to predictive analytics, we need to keep our eyes on the road ahead, and the follow-on destination: prescriptive analytics. I will take up that topic in a future post.

 

In the meantime, for a closer look at some of the exciting ways people are using data analytics to accelerate time to insight, visit the Intel New Center of Possibility site.

EMILY BACKUS

SC15 Reflections

Posted by EMILY BACKUS Dec 7, 2015

Fuel your Insights

 

This year’s Supercomputing conference got underway in Austin, Texas with over 12,000 attendees- the largest gathering yet. People from all over the world and from various disciplines came together to reconnect, share, and learn – All to celebrate how high-performance computing (HPC) is fueling insights, discoveries and innovation.

p1-diane-presenting-image.jpg

Intel’s Senior VP Diane Bryant opened the conference with her plenary speech delivered to a full house. The theme, HPC Matters, resonated with this crowd for obvious reasons. Diane discussed how HPC has entered a new era where the technology and the cost of advanced computation have reached a cross point of accessibility.

 

Diane reinforced Intel’s commitment to advancing diversity in the area of HPC by announcing a new scholarship program Intel will fund for the next five years to enable women and minorities with science, technology, engineer and math (STEM) undergraduate degrees to pursue graduate degrees in computational and data science. Dedicating US$300,000.00 per year, the scholarship will be administered in conjunction with SIGHPC, the ACM special interest group on HPC and the Supercomputing diversity committee.

 

Intel announced the launch of Intel Omni-Path Architecture (Intel OPA), an element of Intel Scalable System Framework (Intel SSF).  This new fabric has been eagerly anticipated, with many HPC ecosystem partners already onboard in the Intel® Fabric Builders program to rapidly deploy Intel OPA because of the significant performance and cost benefits. Charlie Wuischpard, VP of Intel’s Data Center Group and Barry Davis, GM HPC Compute and Fabric kicked off the Omni-Path celebration in the booth Monday evening to thank to everyone who worked to develop and deliver Intel Omni-Path Architecture.

 

 

Listen to Dr. Al Gara, Intel Fellow and Chief Architect for Exascale Systems for the Data Center Group talk about how Intel® Scalable System Framework is meeting the extreme challenges and opportunities that researchers and scientists face in HPC today.

 

 

 

Listen to Barry Davis, GM of the High Performance Fabrics Operation at Intel talk about how today’s requirements for HPC fabrics are outstripping the capabilities of traditional fabric technologies in performance, reliability, and economic feasibility

 

 

 

 

It Takes a Village

 

To continue to advance HPC, it takes a community dedicated to cooperation. Intel was honored to announced a number of new strategic alliances at SC15, including Barcelona Supercomputing Center (BSC), Alan Turing Institute (ATI), and Pittsburg Supercomputing Center (PSC) in addition to continued work with many others.

 

Of note was the public unveiling of the OpenHPC Community, a Linux Foundation Collaborative Project. Intel, as one of the 34 founding members of OpenHPC, wants to bring attention to the charter of the organization to provide a new, open source framework for HPC environments with tools, components, and interconnections to enable the software stack.  The community will provide an integrated and validated collection of HPC components that can be used to provide a full-featured reference HPC software stack available to developers, system administrators and users.

 

Listen to Karl Schulz, Principal Engineer of the Enterprise High Performance Computing Group at Intel discuss the OpenHPC community effort.

 

 

 

Go to the Intel Newsroom for more details.

 

 

Looking forward

 

Supercomputing supported a growing number of student events this year, including workshops, sessions, competitions and even recruiting made this an event for the younger generation.

p2-youre-whats-next-photo.jpg

 

Intel and CERN openlab jointly announced the winners of the Intel® Modern Code Developer Challenge on November 14, 2015 at the annual Intel® HPC Developer Conference, held in conjunction with the SC15.  Targeted at student participants, the goal of the contest was to spur advancement in parallel coding and the science it supports, while encouraging students to pursue careers in the field of high performance computing. The students were challenged to improve the runtime performance on code that allowed the simulation of interactive brain cells during the formation of the brain.

 

Out of the roughly 1,700 students, representing 130 universities across 19 countries, Mathieu Gravey, a PhD student from France, submitted the fastest optimized code to win the 9-week internship at CERN openlab. Mathieu optimized the code so the original 45 hour run time was reduced to 8 minutes 24 seconds. Representatives from CERN and Intel were in Austin to award Mathieu the grand prize, plus recognize first place with a guided tour of CERN in Switzerland and second place with a trip to SC16 next year. Congratulations to everyone who participated- it was a great experience working with such a talented group of young HPC enthusiasts.

 

P3-modern-code-winner.jpg

From left to right, Grand Prize winner Mathieu Gravey, Maria Girone (CERN), Michelle Chuaprasert (Intel) & Fons Rademakers (CERN)

 

 

Plans are already underway for SC16 next year in Salt Lake City, Utah…. can’t wait to see what’s next!

Filter Blog

By date:
By tag:
Get Ahead of Innovation
Continue to stay connected to the technologies, trends, and ideas that are shaping the future of the workplace with the Intel IT Center