Open-Source Search: Application Centric and a Way to Big Data

Eric Schmidt took the stage today at TechCrunch Disrupt.

What he said seems striking, considering the pace of change we observe in our daily work. Schmidt said Moore’s Law gave us a guide for the speed to expect on processing information. He said what we did not expect is the amount of information we would produce.

What’s resulting in face of this big data are a number of alternatives to the Google way of search that we will learn a lot more about next week in Boston at Lucene Revolution, the first conference dedicated to open-source search.

Sponsor

What’s emerging is in some sense an application-centric approach to search that is built into services using open-source search technology such as Apache Solr and Lucene.

Developers have deeper needs for the applications they are developing. With the advent of big data, they often have more flexibility with open-source search technology as the data can be accessed via an API and indexed in Apache Solr or Lucene.

People from The Guardian will be attending Lucene Revolution. The Guardian is a modern newspaper – arguably one of the most innovative media companies of our time. It realizes it’s role is not to broadcast the news but be a part of the community as leaders in filtering information and providing applications and services.

RedMonk’s James Governor attended a NoSQL event at the Guardian a few months ago. Apache Solr was a focal part of discussion.

The database is part of the challenge. Oracle RAC has worked fine over the years for the Guardian. But now The Guardian has linked data, social networks and tags galore. The database is not designed for that load.

What happens with API access, which drives for example, tag proliferation, which dramatically increases load on the database.

“Apache Solr is like a database, it works like one for us.

Fields can be multi-value. one piece of content with five tags can be stored in one field. Most important is that SOLR offers the ability to facet the content. apply it *like* a tag…

For example: – an editor’s star rating. we can facet on that for free, and just jump to all the three star albums. facets can be combined much more quickly than a relational database.

With Solr we can perform complex queries, filter by facets.”

Open-source search will grow in demand as more applications depend on search technology for providing deeper context and meaning for the services they provide.

This means curating analytics and new ways to pull in, organize and index information.

We will be at the conference on Oct. 7-8. More than 250 people are expected for this new category of search conference. We plan to do some live streaming from the event. We’ll have more on that later this week.

Disclaimer: Lucid Imagination is paying for airfare, hotel and expenses for Alex Williams to attend Lucene Revolution.

Discuss