Traditional databases, even when they’re called “relational databases,” tend not handle relationships very well, and the traditional way of processing data – particularly large-scale datasets – can actually mean that some of the relationships between objects are lost or obscured.
Several years ago, Google began encountering these sorts of problems with relational data, particularly as this graph data didn’t really fit into its Map Reduce system for big data processing. So Google developed a product called Pregel, which solved the relational data problem and allowed it to be processed on a massive scale.
While Pregel remains an in-house technology for Google, the data startup Ravel is releasing its Pregel-like, large-scale graph processing technology today.
Sponsor
GoldenOrb, which Ravel is also open sourcing (GitHub link), will solve some of the same types of problems as Pregel, but can be applied to many other areas beyond network analysis and social graph analysis, such as epidemiology and mathematics.
But most importantly, says Zach Richardson, the lead architect of the GoldenOrb project and the CTO of Ravel, it makes the programming that developers have to do far simpler. Rather than worrying about how they can get it to run about thousands of machines, “they can just focus on the algorithm for solving their particular problem.” According to Richardson, this means that large scale data problems are now solvable even by startups.
Richardson says that Ravel opted to open source the technology so that others could work on writing algorithms and solve various problems that, in turn, Ravel hopes to be able to learn from as well. The company has no immediate plans to offer commercial support around GoldenOrb.
0 Responses
Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.