Twitter by the Petabyte: Using Big Data to Define Market Sentiment

Millions of tweets run through Twitter. It’s the poster child for big data on the Web.

To get data out of Twitter and use it to track sentiment requires tools with considerable processing and computational capabilities.

BigSheets is a tool created by IBM that takes terabytes or even petabytes of Web data and turns it into information that provides business intelligence. The tool can be used for structured or unstructured data from internal or external sources. For instance, it can run streams of Twitter data for days, weeks or even months on particular keywords. That data can then be mashed up with internal information.

Sponsor

To demonstrate its capabilties, IBM Evangeist David Barnes created an excellent demonstration video that shows how Twitter can be mined for sentiment about the iPhone, BlackBerry or Android mobile smartphones.

We took some segments from the video to explain a process that is not nearly as technical as we thought it would be.

Barnes showed how the tool mined Twitter for tweets that mentioned the smartphone terms: iPhone, Android and BlackBerry. He then tracked the tweets for sentiment. Do people like, love or hate the products? Do they want to buy the products?

The tool can also be used to track multiple web sites for hours, days, weeks or months.
During the video, Barnes mined the Twitter stream for three minutes using the smartphone keywords. He had previously ran a query for 36 hours to demonstrate hopw the data can be used. It pulled 305,000 tweets into the application.

BigSheets classified it by the name of the user, the time of the tweet, what the tweet said and other associated metadata.

Making a Tag Cloud

Barnes then applied the sentiment language to the analysis. About 50,000 tweets showed sentiment. He then sought out tweets that expressed a sentiment to buy something.

Barnes then applied visualizaton software to create a tag cloud of the analyzed tweets.

Analyzing the British Parliament

In his last example, Barnes showed how BigSheets can scrape data from Web sites to do analysis. He did a query for people, places and things. That data resulted in data about crime and other matters. He cross referenced that data with the members of pariiament voting records.

What Would Facebook Look Like?

Like any good demo, Barnes leaves us with more questions about big data than we ever had before.

Take Facebook, for example. What would Mark Zuckerberg have created if he had this tool when creating the now famous social network? What will future developers create with tools like this built into what they create?

BigSheets is one of several data mining tools we are seeing that use MapReduce and other techniques to mine data. What’s amazing is what the tools an do but also the accessibility of the tools themselves.

Discuss