What if you were given incredible powers but had such a limited imagination that you only used them to pollute the internet with spam? That’s what’s happening to the powerful distributed labor marketplace of Amazon’s Mechanical Turk, where requesters pay small sums of money for people around the world to perform small tasks that only a human can do. You can use Mechanical Turk to do incredible things – but it turns out that most people don’t. Many just use it to hire an army of spammers.
A new study by NYU academics Professor Panos Ipeirotis, Dahn Tamir and Priya Kanth studied all the new Mechanical Turk requester accounts that have been created over the last two months and found that more than 40% of their requests were for the workers to commit acts of spam. The team used Mechanical Turk itself to evaluate the tasks submitted, but they had to take extra steps after their own requests for work came back filled with spammy, random input from workers who didn’t care. The whole situation is a tragic loss of opportunity – because there are some really fantastic things you can use this service for.
Sponsor
The Good News
What’s the screenshot at the top of this post? That’s the end result of a Mechanical Turk project I did a few months ago. I was blogging about a high-powered technology conference called Techonomy and in preparation for the event, I submitted all of the attendees’ names to Mechanical Turk. I asked the workers to find the Twitter username, web home page URL, US or International designation and gender for each person’s name. I had 3 people look at each name (for quality control) and paid 15 cents for each task.
The people in the system processed hundreds of names in just a few hours for about $50, much faster than I could have done myself. Meanwhile, I was able to spend my time doing other things.
Once they were done, I created all kinds of resources with the information they procured, including a Twitter list of the women attending the Techonomy conference. I then imported that Twitter list into the fabulous service Flipboard, which creates a personalized iPad magazine from the links shared by any list of Twitter users. Thus I now have a daily, dynamic, multi-touch personalized magazine on my iPad made up of the content shared by women who attended the Techonomy conference. That’s pretty awesome. Thanks, Mechanical Turk!
The Bad News
A whole lot of the work being done on Mechanical Turk is downright spammy. That’s a problem for me as a Mechanical Turk work requester because it creates an atmosphere of low quality work and indifference. It’s a problem for me as a web user because I have to deal with the spam – in my search results, in my blog comments and in my email.
The NYU team used the following criteria to determine whether a task was spam:
- SEO: Asks me to give a fake rating, vote, review, comment, or “like” on Facebook, YouTube, DIGG, etc., or to create fake mail or website accounts.
- Fake accounts: Asks me to create an account on Twitter, Facebook, and then perform a likely spam action.
- Lead Gen: Asks me to go to a website and sign up for a trial, complete an offer, fill out a form requesting information, “test” a data-entry form, etc.
- Fake clicks: Asks me to go to a website and click on ads.
- Fake ads: Asks me to post an ad to Craigslist or other marketplace.
- Personal Info: Asks me for my real name, phone number, full mailing address or email.
- You can also use your intuition to classify the HIT
After studying 5842 task groups submitted by 1733 new users, the group drew the following conclusions:
- 40% of the HITs from new requesters are spam.
- 30% of the new requesters are clear spammers.
- The spam HITs have bigger value than the legitimate ones.
That’s bad news, but it could be worse. We wrote about the proliferation of social media spam requests on Mechanical Turk two and a half years ago and though the types of spam have changed (hello, Facebook “Likes” and R.I.P. Delicious) it’s not clear that it’s gotten any worse.
We do have a baseline now, so hopefully these same academics can study the marketplace again in a year and see whether things have grown more or less spammy. They say Amazon is indifferent to the reality of how much hired spam is coming out of Mechanical Turk – and that is in truth a bad thing, for all of us. The culture of work on the site could really be improved, and thus the end product would improve.
It would be a terrible shame if Mechanical Turk was shut down – but it’s also tragic that this is how so many people are using it. Come on everybody, Mechanical Turk is made of people – let’s do more interesting things with it!
0 Responses
Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.