How Yahoo’s Latest Acquisition Stole & Broke My Heart

“What do you think about Dapper?” That was the question it felt like everyone asked me for weeks after I wrote up a startup called Dapper.net on TechCrunch in the Summer of 2006. “Create an API for any website!” was the company’s unofficial slogan. Almost no one understood exactly what could be done with this powerful point-and-click tool, but everyone I talked to knew it was exciting.

Last week the company was acquired by Yahoo and brief press coverage of the deal called Dapper simply a semantic advertising platform. It was so much more than that, especially for me. Dapper set my imagination on fire, it powered acts of community management magic and it helped me meet Neil Young in person. We spent many long nights together. Four years after I first wrote about it, I still bring Dapper up in conversation frequently – but for a while now it’s been part of a story of heartbreak and caution.

Sponsor

What Dapper Does

Here’s how I described the core service when it launched, in August 2006:

Here’s how it works. Users identify a web site they are interested in extracting data from and view it through the Dapper virtual browser. [Co-founder Jon] Aizen showed my how to do it using Digg as an example. I clicked on a story headline, on the number of diggs and the URL field. I went to another page on the same site and did the same thing so that Dapper could clearly identify the fields I was interested in.

I then went through the various tools available on the site to set certain conditions and threshholds and ended up with XML feeds I could do all kinds of things with. Like send me an email whenever there’s a TechCrunch story on the front page of digg, or when a search results page shows a TechCrunch story with more than 10 diggs. After I create an end product through the site, other users will be able (after a 24 hour period in which I can edit the project) to use my project either as is, altered to fit their needs or in the future, in combination with other projects.

Below, a 4 minute video demonstrating Dapper that I recorded on New Year’s Eve 2007, after Wired Magazine wrote a post slamming web scraping. I had a sore throat, it was a holiday (on the next New Years I eloped to my living room and got married) but it was important that scraping be defended – a screencast had to be made! It was important.

In February, 2008 the startup held an event called DapperCamp in San Francisco. It was sponsored by IBM and MindTouch, because those and other companies were exploring ways to move data around from static websites into dynamic processes using Dapper.

The event was fabulous. I was the least technical person there, but I flew down my young cousin on my Dad’s side, a developer in training, for his first experience in the Bay area web geek scene. We had a great time and worked late into the night sitting in a little bar brainstorming ideas and scraping feeds from websites.

Our best idea was this: Yahoo’s service MyBlogLog tracked users who navigated to any participating website and upon visiting a site for the 3rd time, a user appeared in a field labled “New Fans” on your site’s MyBlogLog page. We used Dapper to scrape an RSS feed of the usernames of all the new people appearing as fans, people who had just made their 3rd visit to ReadWriteWeb, and we set up a workflow to email those people and welcome them to the community here. It was awesome.

We scraped a feed of the most bookmarked ReadWriteWeb pages in Delicious, a feed of RWW stories submitted to Digg and the number of Diggs they had. We monitored those feeds in a dashboard. We scraped feeds, .csv files, image slideshows and more. It was wonderful.

How Dapper Helped Me Meet Neil Young

I love Neil Young, I always have. In my early twenties I hitchhiked all around the country listening to a tape I’d recorded of Neil Young’s Greatest Hits albums that I’d checked out from the public library, until the tape was worn out and unlistenable. It was my personal soundtrack for years.

Years later, I work on the internet. In my personal consulting practice, I used Dapper a lot. I used it in working with a group of accountants to scrape feeds of news updates posted to old-fashioned government agency websites that had no feeds. I once subcontracted as a consultant to a consultant to a consultancy to an analyst service serving a pharmaceutical company. (I thought that was far enough removed that I wouldn’t get any on me, but none the less at my first meeting an executive said to me ominously “welcome to Big Pharma.”)

It turns out the client at the end of the long pipeline of invoices sold a diet pill, and young women were complaining on MySpace and forums that the pill sometimes caused leakage from their…and I showed the next consultant in line how to use Dapper to scrape the forums for a feed monitoring said customer complaints. The check cleared and I never went back, but I still thank Dapper for making that work possible. If stranger things were ever piped through the service, I don’t know what they were.

And then Dapper helped me meet Neil Young, in person.

Pages: 1 2