In the last several blog posts we’ve been focusing on the evolution of data center networks to more virtualization and to software-defined networking (SDN). Another source of the transformation is big data and the emergence of the Hadoop application.
Hadoop is distributed processing on steroids to deal with the incredibly large data sets that are needed to solve business, scientific, law enforcement and other real world problems.
Some data center applications are called on to process data sets on the order of Exabytes of data, which strains the capabilities of traditional relational data base systems. Hadoop emerged as a way to distribute the workload of these jobs to various servers or virtual servers for processing then to compile the responses and to present the result to the user.
From a network perspective, Hadoop needs the ability to scale to up to thousands of processors, in many cases, and that means a layer-three routed network. With this size of data set spread across so many servers, end-to-end latency can become prohibitive unless a low-latency network is in place. One of the reasons Hadoop is popular is that it reduces processing time – without a low-latency network that advantage will disappear.
As a company with the first switch silicon to feature latency of less than 400 ns, the Intel® Ethernet FM6000 switch silicon family, is ready to help networking systems develop products for the new needs of a big data / Hadoop world.