Red Hat’s Data Grid 6 Challenges Hadoop on Big Data

Red Hat’s addition of Data Grid 6 to its JBoss family of products cements the company’s position as a major datacenter player. The new storage management component within Red Hat’s JBoss middleware platform is designed to give large e-commerce and financial transaction customers the ability to handle operations where speed and scalability are essential.

Not Like Hadoop

Red Hat JBoss Data Grid 6 is not like the big-data solutions found in the Hadoop ecosystem: as an in-memory, key-value store, Data Grid is much more optimized to handle the operations that Hadoop simply can’t: transactions like the kind found in e-commerce and financial trading systems.

Because so much read and write activity is needed in a single transaction, a batch processing system like Hadoop or even a relational database can’t keep up with the speed and scaling necessary to make huge systems found at banks, trading houses or mega-merchants like Amazon.

Countering the issues of speed and scalability is the requirement to adhere to the Principles of Distributed Computing formed by computer scientist Eric Brewer. Specifically, Brewer postulated the CAP Theorem, which holds that systems must be consistent with data, all requests must be available, and partition tolerance must be in place in case of system failure.

To get the speeds and uptime it needs, Amazon will use its non-relational Dynamo database to apply an “eventually consistent” approach to their database systems.

Dynamo is part of a class of non-relational databases known as distributed key-value store (DKVS) databases. DKVS is one of five classes that comprise the topology of the NoSQL landscape, each with a different architecture and approach to managing data.

DKVS databases, also known as eventually consistent key-value store databases, are specifically designed to deal with data spread out over a large number of servers. These systems use distributed hash tables for their key-value stores, and because they’re distributed, the database uses peer-to-peer relationships between servers, with no “master” control. Currently most of the databases in this class are Dynamo or Dynamo-based implementations of Dynamo, such as the open source Project Voldemort, Dynomite, and KAI databases.

Key-value store (KVS) databases are similar in architecture to DKVS. But instead of being distributed across servers, data is held on disk or in RAM.

The Big Data Ecosystem

This, then, is the sector of the big data ecosystem in which JBoss Data Grid lives. Based on the open source Infinispan platform, the Data Grid product sits as an addition to the JBoss Enterprise Application platform, Red Hat’s flagship suite of middleware services.

The use of Infinispan makes a lot of sense in this context, since from its inception Infinispan was not just meant to be another data grid storage system, but also one optimized for the cloud. Itself loosely based on JBoss Cache, Infinispan took the clustered-caching libraries of JBoss Cache and bulked it out to a full-fledged platform optimized for the cloud, according to Manik Surtani, Founder & Project Lead at JBoss Cache, Infinispan Data Grid, when he introduced Infinispan in 2009.

Three years later, Surtani’s vision is being realized. The Data Grid software can be accessed by other applications either through Representational State Transfer (REST), the standard memcache API, or the HotRod API. HotRod is another spin-off from Infinispan that facilitates the elasticity of Data Grid – the capability to scale up and down as needed.

Surtani described HotRod’s function as enabling two-way communication between client and server so that as the shape of the data grid changes in the cloud, client nodes will be be aware of their own standing within that ever-changing data grid, thus creating more efficient operations within automated cloud systems.

Becoming a Major Datacenter Player

This additional piece to the JBoss family of products will set Red Hat as a major datacenter player, giving customers a more complete stack to work with for their cloud-based operations, which is exactly where Red Hat wants to position the Enterprise Application Platform.

“JBoss Enterprise Application Platform 6 is really a major milestone for us,” said Craig Muzilla, Red Hat vice president and general manager of the middleware business unit, in a Web broadcast Wednesday. “It was designed from the ground up to be cloud-ready and to support cloud deployments.”