A look back at Guardian crowdsourcing project

I meant to point to this ages ago but somehow it got buried in ‘Drafts’ and time marched on. Earlier this year the Guardian in the UK turned heads when it used crowdsourcing to sift through mountains of documentation about MPs’ expenses and expense claims. The story, about questionable expense claims made by MPs, was broken by The Daily Telegraph and rumbled on for months.

Neiman Labs wrote a great post about ‘four lessons’ learned from the experiment, which was turned around very quickly.  They talked to developer Simon Willison (who visited NZ in 2008 to speak at Webstock) who started coding one week before the project launched. There are a few outtakes from that piece below.

Here’s Simon Willison talking at a News Innovation Unconference about how he and the rest of the team got the crowdsourcing off the ground in a week.





Here’s an excerpt from the Nieman post about four lessons:

Your workers are unpaid, so make it fun. Willison started coding one week before the Thursday launch date, teamed with a designer on Tuesday, a system administrator on Wednesday and leaned on everyone in his 15-person department for ad-hoc help on Thursday. But the bulk of the labor would come from Guardian readers.

How to lure them? By making it feel like a game…The Guardian’s four-panel interface — “interesting,” “not interesting,” “interesting but known,” and “investigate this!” made categorisation easy. And the progress bar on the project’s front page, immediately giving the community a goal to share.

Public attention is fickle, so launch immediately. Before Parliament released its records Thursday, Willison’s team thought they might be able to postpone their launch to Friday if necessary. When they saw Thursday’s newsbroadcasts, they realized they’d been wrong. The country’s imagination was caught. “It became quickly clear on Thursday that it was a huge story, and if we failed to get it out on Thursday, we’d lose a lot of momentum,” Willison said.

Speed is mandatory, so use a framework. Willison’s project was built on Django, the custom Web framework “for perfectionists with deadlines” that he and Adrian Holovaty created for the Lawrence Journal-World. In the world of database programming, a framework is like an offset press: hard to build — Django 1.0 required three years of open-source development — but once it’s set up, there’s no faster way to churn out content. Hand-coding an application like the Guardian’s would have been like publishing a daily newspaper with movable type.

Participation will come in one big burst, so have servers ready. As well as the Guardian’s first Django joint, this was its first project with EC2, the Amazon contract-hosting service beloved by startups for its low capital costs.

Willison’s team knew they would get a huge burst of attention followed by a long, fading tail, so it wouldn’t make sense to prepare the Guardian’s own servers for the task. In any case, there wasn’t time. “The Guardian has lead time of several weeks to get new hardware bought and so forth,” Willison said. “The project was only approved to go ahead less than a week before it launched.” With EC2, the Guardian could order server time as needed, rapidly scaling it up for the launch date and down again afterward.