News has a filter problem: we judge the whole on the little we see

I reckon I read 10 per cent of what a news org produces on a given day; more on some days, less on others, and some days none at all.

So my perception of ‘the news’ as a whole is based on this small amount I see of what the world’s news orgs produce – my daily 10 per cent.

I was thinking about this recently when someone framed the issue of the quality of news in a new way for me.

Rather than asking whether the quality of news was declining, as is commonly discussed among us future-of-news types, he asked whether people’s perception of the quality of news was changing and what effect that was having on society.

What follows is a kind of note to self about that idea (a rather incomplete one).

I’ve lost count of how many times I’ve heard someone say – or said myself in a fit of pique – that ‘the news is rubbish’, ‘we need more investigative journalism’, ‘we need better analysis’, ‘who cares about that celebrity nonsense’ or ‘if I see one more crime story under the heading of National News I’m going to spit.’

These go hand in glove with our belief that if more people were better ‘informed’ our democracies would be healthier, our economies more productive, our crime rates lower, our nation’s health improved and the world a lovelier place altogether.

We tend to talk as though these goals would be more readily achievable if only there were more, better-quality journalism – and I suppose by implication less lower-quality journalism. Then we get mired down in angst over how it’s all going to be paid for.

I’m less sure about the assumptions here.

I find myself wanting to spend more time quantifying what we’ve already got that’s good and being clearer about what we think is missing, before thinking about what to replace it with and how to foot the bill.

And I find myself thinking about how to better curate and distribute what we’re already producing so it reaches more people in a way that works for them. To my mind, if content is king, distribution infrastructure is kingdom.

It’s easy to skate over the fact that there’s already a lot of good journalism around, albeit scattered across multiple outlets and platforms, across days and months and years, and across countries, so that no one of us is aware of it all.

We have a filter problem.

We have a tendency to look at a handful of stories – on a homepage, for example – and pass judgement on the basis of those. I know I do. If the first six stories in my daily news email are all about crime, I get cross and move on. If I turn on a TV current affairs show and see a commentator I don’t respect, or a story I think is lightweight and irrelevant, I get cross and move on. If it happens several times in a  row, I write off the whole show, or the news judgement of the whole news site, at least temporarily.

My criticisms much of the time are justified, I think. There is a lot of lightweight news that I find irrelevant (stories based on Facebook polls or surveys conducted by PR companies, for example) and there are a lot of commentators pronouncing tiredly on dated issues, churning out a column a week regardless of whether they have anything much to say.

But sometimes I come across stories and videos and slide shows and commentary – often well after they were published and often via a link on Twitter or in a blog – that I think are great. Stories that interest me or make me think or make me feel connected with the world or understand it better. And I spend a fortune every year on long-form journalism in the form of paperbacks.

Yet I tend not to think about these great stories when I’m making my snap judgements about how good ‘the news’ is these days.

That’s partly human nature, I guess: a tendency to remember the negative. But it’s also because these stories rarely seem to make their way into my daily news email, and once they are no longer current they’re pushed off the front page and puff boxes and effectively become invisible.

In other words, news orgs aren’t getting these great stories in front of my eyeballs regularly: they’re not showing up in my daily 10 per cent. Or maybe they are, but I don’t get to them that day because I’m busy and I never get round to going back.

I wonder how much our perception of ‘the news’ would change if news stories as a whole were curated, packaged and distributed differently. If there was slightly less obsession with currency and stories were made available for longer and resurfaced from time to time, so readers were no longer punished for being late to a story and forced to go on an archaeological dig to find it. If the news values underlying what goes on the homepage or in the daily email were updated from whenever they were forged (The 1970s?).

With better curation and filtering tools (easier said than done, I know), we’d more successfully tune out the gunk that turns us off and see more in our daily 10 per cent of what switches us on. We’d have less reason to write off the whole lot in a fit of pique. Our perception of ‘the news’ would be more positive, and our faith in it would be stronger.

There’s an uncomfortable truth in here, of course, in that we, the audience, can be hopelessly lazy about ‘staying informed’ and the best filtering tools in the world won’t help us if we don’t take the time to set them up.

The battle to get us to pay for news is as much a battle to get us to want to read much of it in the first place.  Same goes for a more informed citizenry.

We have lots of reasons for not reading as much news as we might, and for being selective about what we do read:

  • We’re busy.
  • We can only take in so much information in one sitting.
  • We like to stick with our own worldview so we focus on the stuff that we find interesting and validating and filter out the rest.
  • Life goes on even if you don’t read the news: the mail’s still delivered whether or not you know who’s running NZ Post, the traffic lights still work, pubs still open, lattes still get poured with foamy little flourishes.
  • Some people have an incentive to read the news because it gives them a competitive edge in their work. Others don’t. Why plod through acres of words when you don’t get a discernable payoff?
  • Some days the news feels like an assault; a slew of problems that we feel powerless to solve. War, starvation, infant mortality, corruption, piracy, sex trafficking and slavery. Must I know of every misery in the world every day?

Imagine if money trees started sprouting in the middle of newsrooms all the world over, and large numbers of clever well-adjusted journalists were employed on cracking salaries and put to work investigating and writing spirited but balanced prose based on meticulous research.

Every day the news sites and blogs and radio and TV bulletins and papers and podcasts would brim with excellently interesting and considered stories.


Now, how many hours a week would you spend reading news? About the same as now? An hour more? Five more? Ten more? Or ten less because you actually preferred the lightweight stuff that took less time and energy to consume?

Would you read more or less news about the economy than before? Sport? Fashion? Astrophysics? Astroturfing? Justin Bieber?

Would the better quality journalism make you twice as well informed? Three times? Ten? Could you measure the difference to your life? To society? Would people who don’t read news now start reading it? Would they stick with it?

Who knows. I may be more reliably informed by what I read but I suspect I wouldn’t read too much more than I do now, and nor would many others. There are only so many hours in the day.

There are no money trees, of course, so we’re not going to answer these questions easily.

Maybe, then, we future-of-news types should focus on how to better get the ‘news we approve of’ out to people in a format, time and place that makes them interested. Maybe we should focus on how to get that daily 10 per cent working better for people now – before we throw buckets of money at funding ‘better journalism’ that may only get lost in the mix.

Maybe we should think more about the idea that there’s just too much news to use. As Josh Benton explores in this lovely post about the role of the journalist – and novelist – in shrinking the world for us, my problem is not that I need more news and information but that I’m drowning in it.

These ideas and many others are more eloquently explored in the Neiman Lab special report: The Digital Landscape: What’s Next for News? which is a highly recommended read.

Younger journalists, self-censorship & other newsroom trends

I 2010 I attended an International Media Conference along with hundreds of journalists from a staggering array of Asia-Pacific countries including Myanmar, Fiji, Papua New Guinea, China, US, Taiwan, Australia, Japan, Singapore and Nepal.

I travelled to the conference with financial support from the Asia NZ Foundation, which aims to strengthen awareness in New Zealand of Asian economic and political issues and to encourage networking. I’m grateful for the support and can attest that my awareness was indeed raised and some interesting connections were made.

The conference, organised by the East-West Center’s Asia Pacific Center for Journalists and The University of Hong Kong’s Journalism and Media Studies Centre, was a big themed event looking not only at media but also at economic issues in Asia and especially China. There were many plenary panels and smaller sessions and it was the kind of event that leaves one a little more knowledgeable but in my case painfully aware of how little one actually knows.

What follows are a few general notes I made during the conference – points or themes that have stayed with me and/or that I managed to get down in my rumpty shorthand and decipher later. I have not, for now anyway, quoted many people directly here but you can see summaries, video highlights and full sessions for yourself here.

Journalists are getting younger

This is something that comes up fairly often in conversation about today’s newsrooms and I was interested to hear it come up a few times at the conference too. It appears commercial pressures, falling wages and other factors are driving the age of journalists down in all parts of the world, at the expense of long-term collective memory, specialist knowledge and the wisdom that comes from experience.

The first reference came from a newspaper editor in Myanmar who noted that as his country heads toward its first election in 20 years, its newsrooms are populated with young journalists who don’t remember anything of the 1990 election and have no experience in political or electoral coverage. That lack of experience is worrying, he said, although it is far from the only concern in a country where all news must be physically sent off for inspection before being published.

Thailand-based Burma activist and editor of Irrawaddy Magazine, Aung Zaw, spoke passionately about human rights abuses and lack of freedom in Burma and what kind of catalyst the election might prove to people frustrated by decades of military rule with no sign of meaningful change.

“Many people believe this election is only to legitimise military rule,” he said, and there are predictions that there will be fraud, intimidation and rigged ballots to that end.

Zaw writes about what conference guest speakers Surin Pitsuwan, secretary general for the Association of Southeast Asian Nations (Asean), and Kurt Campbell, the US Assistant Secretary of State for East Asian and Pacific Affairs, had to say here.

The issue of young journalists was also raised by Dr An Ran, a former Nieman Fellow and senior science editor at China Newsweek, who mentioned the younger age of many health reporters in China and said they would benefit from greater experience and training given the complexities of public health reporting.

Thai elections by the numbers

Kavi Chongkittavorn, senior editor and columnist with The Nation in Thailand and a highly entertaining speaker, gave us a synopsis of Thailand in numbers: Since 1922, Thailand has had 18 constitutions, 27 prime ministers, 56 governments, 17 coups, 15 years of military rule and 25 elections.

He spoke about, among other things, elections and how important they were to Thailand, and about the rural-urban divide and the tendency for politicians to promise voters “free education, free healthcare, village funds, pension funds” and “find the money later”. Sound familiar? You can see him in action in Day One footage here.

Censorship and graft in China

There’s good news here and bad. The good news is that Chinese editors at the conference consistently made the point that the media is changing rapidly in China and while censorship remains there is vastly more diversity of opinion than 20 or 30 years ago and the central government is increasingly pragmatic about opening up the media. One said “I don’t think media in China is still an instrument of the State.”

The bad news is that local and regional government is less pragmatic, more likely to interfere with the media, and more likely to be influenced by big business with money to spend on pumping up some stories and burying others.

Panellists said some journalists were susceptible to bribes –  known as “money to seal up your mouth” –  and it was an issue for the media and other industries, although all the editors present were adamant they were opposed to graft and wouldn’t tolerate it. One audience member asked whether low wages in journalism prepared the ground for graft, but a panellist replied that in fact journalists’ wages weren’t that bad in China, “more middle class”.


Again, an issue that pops up in media world over and one that came up several times in conversation at the conference.

At one end of the scale of self-censorship is a journalist who learns over time what the news editors like and writes to suit, or a sub-editor who instinctively writes a headline that fits with his or her paper’s mindset on the issue at hand.

At the other end are editors in China and elsewhere who must make a judgement call on any number of stories each day as to which may attract negative attention from authorities now or in the future – and figure out how to defend what they run.

Among the problems facing editors that I heard about were that reprisal could come at any time – within hours or months later – and there was no clear rule of thumb as to which stories were going to be problematic. You might run a critical story one day that invites no comeback at all while another which seems innocuous brings a rap months later.

One former editor told me how a staff reporter was once detained and questioned for days as a ‘message’ to the paper about a story run months earlier that displeased someone somewhere. You can never tell exactly what will displease, so it becomes easier to self-censor and steer clear of any potentially difficult material. Displeased authorities can make life difficult for editors in a variety of ways, not least by losing them their jobs.

The main drivers of change in China’s media

There was a strong message that the media is rapidly changing in China. The three main drivers of change identified by panellists were:

  • The internet – harder to control, faster, further reaching, changes people’s expectations. (See Asia NZ’s Charles Mabbett profile of Chinese blogging here.)
  • Private ownership – many new media companies are privately owned and the likes of Tencent, which runs the massively popular QQ messaging service along with blogging platforms and news portal (it has an editorial staff of 600 people), are listed on the Hong Kong Stock Exchange. These companies are driven to do business in new ways and respond to what their customers want. (Charles Mabbett looks more closely at Tencent and other new media companies here.)
  • Pragmatic central government – understands times are changing, wants to retain credibility with populace, wants to open up albeit in a measured way.

Fan Yijin (former publisher of the Southern Media Group in China) said in his early days in the media – in 1970 at the “height of the cultural revolution” – there were just a few national newspapers and magazines and one newspaper per province, each tightly controlled and “of one voice”. In subsequent years as Deng Xiaoping started opening up China to the outside, so did the media start to open up.

Today there are thousands of publications, both state and privately owned, and millions of blogs and social networking accounts. Even with censorship and self-censorship, there are far more voices in the public arena than ever before.

In the intervening years editors saw problems they hadn’t seen before, Mr Fan said. “Social problems, growing gap between rich and poor, farmers who go into the city and sit in front of government buildings to protest… How were we to cover these?”

At first, he said, the government was nervous and banned coverage of these events for fear they would damage China’s image and that one report would lead to more reporters flooding in. But the government realised that “even if old media didn’t report, new media will, even if Chinese media don’t report, foreign media will.”

The government began to realise the value of open reporting and “resolved that all information should be open and go public quickly, in a timely manner”.

That process hasn’t been perfect but “there is progress”.

Earthquakes and seismic waves

The Sichuan earthquake in 2008 was cited several times as an example of a more open media in operation in China. Reporters from all over the country (and world) were allowed into the province and many were blogging and microblogging from the area ahead of filing their reports. This level of access was unprecedented.

On the flipside, Mr Fan acknowledged there was still dissatisfaction and there were still limits placed on journalists. A foreign journalist noted that she’d heard of at least one Chinese staff reporter ordered to leave the province once the story turned from disaster recovery to children dying in the ruins of poorly designed buildings.

On a free day I had wandering about in Guangzhou, a burgeoning city in the export corridor of southern China, I picked up a copy of the China Daily, a State-run English language paper, which carried screeds on the clean-up efforts after the more recent earthquake in Yushu County, Qinghai Province, which killed more than 2,000 people. Like all the other papers the day before, it was printed in black and white (with the exception of a red and gold flag at half mast on the front page) as a gesture of respect. Haven’t seen anything like that for a while.

The conference website has a record of the speakers.

Operation Mincemeat: an elaborately staged deceit

Former CIA director Allen W Dulles’ book about the intelligence business gives an example of how a rigged accident can be used to feed fake information to the enemy.

His example, Operation Mincemeat, as it turns out, is a well-known one told in the 1950s book ‘‘The Man Who Never Was’ and the movie of the same name.

This was the first I’d heard of it though, which just goes to show that all the information in the world won’t do you any good unless you actually see it.

Anyway, it’s a cracking story that left me thinking this: “No matter how far out of my comfort zone my work takes me, at least I’ll never have to find a corpse, dress it in uniform and carefully float it ashore on a Spanish beach before making off in a submarine.”

Dulles explains the operation thus:

An accident was cleverly staged by the British in 1943 before the invasion of Sicily, and it was accepted by the Germans at the time as completely genuine. Early in May of that year the corpse of a British major was found washed up on the southwest coast of Spain near the town of Huelva, between the Portuguese border and Gibraltar.

A courier briefcase was still strapped to his wrist containing copies of correspondence to General Alexander in Tunisia from the Imperial General Staff. These papers clearly hinted at an Allied plan to invade southern Europe via Sardinia and Greece. As we learned after the war, the Germans fully believed these hints. Hitler sent an armoured division to Greece, and the Italian garrison on Sicily was not reinforced.

This was perhaps one of the best cases of deception utilising a single move in recent intelligence history. It was called ‘Operation Mincemeat’, and the story of its execution has been fully told by one of the main planners of the affair, Ewen Montagu.

It was a highly sophisticated feat, made possible by the circumstances of modern warfare and the techniques of modern science. There was nothing illogical about the possibility that a plane on which an officer carrying important documents was a passenger could have come down, or that a body from the crash could have been washed up on the Spanish shore.

Actually, the body of a recently dead civilian was used for this operation. He was dressed in the uniform of a British major; in his pockets were all the identification papers, calling cards and odds and ends necessary to authenticate him as Major Martin. He was floated into Spain from a British submarine, which surfaced close enough to the Spanish coast to make sure that he would reach his target without fail. And he did.

A couple of footnotes.

On how to find a suitable corpse, the BBC’s h2g2 site has this to say:

Because of the sensitive nature of the operation, they could not very well ask all and sundry if they’d come across any bodies or knew anybody who’d died from drowning recently, let alone raise the delicate issue of borrowing a body. They managed to slip some very carefully guarded enquiries to trusty Service medical officers; however, in all of the cases they had the relatives’ consent to worry about, or the possibility that somebody might talk. And all around them, bodies piled up – none of them suitable.

Finally, when the committee was beginning to consider resorting to grave robbery, they heard of a man who had recently succumbed to pneumonia brought on by ingesting rat poison. The pneumonia would help embellish the deception nicely – a man who had died floating around in the sea would be expected to have some liquid in his lungs; the difference between saltwater and freshwater would hardly be noticed in lungs that were decomposing, especially if the post-mortem were to be carried out by someone with the preconceived notion that death was caused by drowning. It would take an expert of Sir Bernard’s calibre to tell that the man had not died at sea; Spain hadn’t any.

And who was Major Martin? Again from BBC’s h2g2 site:

In 1996, 53 years after Mincemeat, a British town planning officer and amateur historian by the name of Roger Morgan uncovered evidence that Major Martin had actually been a homeless Welsh alcoholic named Glyndwr Michael who had died through ingestion of rat poison (whether it was suicide or accidental poisoning was undetermined)24.

But not everyone agrees on the identity of the corpse as this story from earlier this year explains. I’ll leave you to pick up the story from here.

The beauty of checklists

I enjoyed The Checklist Manifesto by Atul Gawande, a surgeon and New Yorker writer who previously wrote a notable piece about finding that more expensive healthcare wasn’t necessarily better healthcare. He’s written a whole lot more since then.

The premise of The Checklist Manifesto is that a simple (but well devised) checklist is perhaps the best tool we have to make us consistently more successful at managing complex situations – whether that be operating on someone, choosing companies to invest in, landing a plane or building a skyscraper.

The airline industry in particular has learned to rely on pilot checklists – and a culture of always using those checklists – to maintain safety, and their evolution makes for interesting reading.

As Gawande notes, the reason no one died when an Airbus A320 was forced to land on the Hudson river in January 2009 was not only because of skill and professionalism on the part of the pilot. It was because the pilot and crew all used the checklists provided for them for precisely that situation.

Gawande’s interest in checklists grew out of wanting to improve outcomes for surgical patients. He points to data showing that by 2004 surgeons around the world were performing 230 million major operations a year and estimates of complication rates ranged from 3 per cent to 17 per cent.

“Worldwide, at least seven million people a year are left disabled and at least one million dead – a level of harm that approaches that of malaria, tuberculosis and other traditional public health concerns.”

The numbers were sufficiently alarming to prompt the World Health Organization (WHO) to ask him to spearhead a project to tackle the issue. The project group discussed all manner of interventions – guidelines, training, incentives – but ruled them out as too easily ignored, expensive or impractical.

Research showed that simple interventions were often the best in public health. Gawande cites as examples how removing the pump from an infected water well was enough to end a famous cholera outbreak in London in 1854, and how child mortality was reduced in poor districts of Karachi, Pakistan, by supplying free soap and instructions on how and when to wash hands – a kind of checklist.

A checklist had also proved useful in reducing infections at a US children’s hospital simply by ensuring antibiotics were administered at the right time.

So the group came up with the WHO Surgical Safety Checklist and trialled it in eight countries, including here in New Zealand, at Auckland Hospital.

I saw it, or something like it, in action when I sat with my Dad at Auckland Hospital ahead of an exploratory surgery recently. The pre-op nurses worked through a set of forms and questions with him before he was prepared for surgery.

The results of the trial were so positive WHO set about rolling the project out worldwide, and New Zealand is one of the countries to have signed up to encourage its use in every hospital.

Not all have taken it up, but let’s hope they do. A ministerial review in NZ last year found that “44 thousand people admitted to hospital suffer an unintended injury caused in the management of their conditions, rather than the underlying disease – this is a similar rate to other countries. Although most of those people had relatively minor adverse events, about 15% resulted in permanent disability or death.”

In testing his theory about the checklist, Gawande visited building sites, the people who write the pilot checklists for Boeing, venture capitalists and top-flight restaurants. He found examples everywhere of simple checklists improving success rates, with the pilot checklists proving particularly instructive.

He learned, however, that not all checklists are created equal and he picked up tips along the way on writing a good one. Here are the main points:

  1. Identify pause points – the point in a process where the team needs to pause and use the checklist. In surgical checklists they came up with three: before the patient has anaesthesia, after anaesthesia but before incision, and after the operation but before the patient is wheeled out of the theatre.
  2. Decide whether it makes more sense to use a Do-Review or a Read-Do list. The first requires people to do their tasks then stop and review them before moving on. The other requires people to read the checklist and do each task in turn, like a recipe.
  3. Use simple, exact wording and use the language of the profession.
  4. Don’t make the checklist too long. If it’s too complex or takes too long people will give up and throw it away. Not everything has to be on the checklist. Just those things that are essential but that people can sometimes miss in the heat of the moment.
  5. Boeing found the best checklists have 6 to 9 points, are on one page that is sparing in colour and free of clutter, uses a mix of upper and lower case letters and a sans serif font. (Boeing use electronic checklists now too.)
  6. Find the right person in a given scenario to give the job of starting or overseeing the checklist. It may not be the most senior person. In some cases the checklist proved a tool for distributing responsibility and power in a way that made people work as a team and unafraid to speak up.
  7. Buy-in is important and so is working as a team. As part of the surgical checklist operating staff are asked to introduce each other so they know one another’s names. Research had shown that people communicate better if they know each other’s names.
  8. Communicating well is as important as checking for potential infection agents.
  9. Checklists have to be road tested in the real world.
  10. Checklists are not static. They need to be adapted to individual situations and improved and updated over time.

Listen to Gawande in Conversation with Tyler on “why Watson will never diagnose your illness, what George Church’s narcolepsy teaches us about CRISPR, what’s missing in medical education, Michael Crichton’s cultural influence, Knausgård versus Ferrante, indie music, and the thing that makes Gawande ‘bawl like a baby’.”

Maps show how transport shrinks the world


New Scientist has published some beautiful maps exploring which are the remotest places on earth – given how much international transport we have available to us.

“The maps are based on a model which calculated how long it would take to travel to the nearest city of 50,000 or more people by land or water. The model combines information on terrain and access to road, rail and river networks. It also considers how factors like altitude, steepness of terrain and hold-ups like border crossings slow travel.

Plotted onto a map, the results throw up surprises. First, less than 10% of the world’s land is more than 48 hours of ground-based travel from the nearest city. What’s more, many areas considered remote and inaccessible are not as far from civilisation as you might think. In the Amazon, for example, extensive river networks and an increasing number of roads mean that only 20% of the land is more than two days from a city – around the same proportion as Canada’s Quebec province.

This one shows roads around the world:

This one shows shipping routes:

And this one shows rivers:

More lovely maps here. H/T to Cliff Kuang at Fast Company.

How future computers were imagined in 1981

From 1981:

“Imagine, if you will, sitting down to your morning coffee, turning on your home computer to read the day’s newspaper. Well, it’s not as far-fetched as it may seem.”

“It takes over two hours to receive the entire text of the newspaper over the phone, and with an hourly use charge of $5 the new tele-paper won’t be much competition for the 20 cent street edition.”

Nice overview of the History of the Internet:

Imagining, in 1969, what the internet might bring us:

How to send an email  – in 1984


Journalists ‘should be interviewed once a year’

John Longhurst on the Canadian Journalism Project suggests journalists should be interviewed themselves from time to time, so they know what it’s like.

And that the interview should be published, so they know what it’s like to have a stranger’s account of themselves on public display.

I agree. I’ve been interviewed a few times over the years and I find it quite disconcerting being on the receiving end of the questions, rather than asking them.

Harder still to not be in control of how the interview notes are written up, which quotes are chosen and what context is given.

Frustrating to see my name misspelled, past job titles inflated or conflated; and short quotes look odd when singled out from a longer conversation.

It’s an eye opener.

No doubt there are countless others who’ve been interviewed and wept when they saw how their 20-minute conversation got condensed into a couple of paragraphs stripped of nuance, context and in some cases rendered insensible.

Journalists are just doing their job, of course. They have to distill and compress information and quotes if they are to fit them inside their 400-word or 60-second story slots.  But it wouldn’t hurt for journalists to get a sensitivity check now and then.

Here’s how John puts it:

It’s hard to explain the sense of vulnerability you feel as you hand your story and comments over to someone who may — or may not — really understand what you are trying to do or say…

Those who do the interviewing likely seldom think of how it must feel to be at the other end of the process. It’s just part of the job — maybe just one of two or three stories that have to be chased down that day.

But for the person being interviewed, it may be one of the most important experiences of their life. For many people — for those who are not professionally involved in work that requires them to deal with the media — it may be the only time in their whole lives that they will be in the newspaper or on the radio or TV. It’s an awesome responsibility for a journalist.

My experience of being interviewed, and doing interviews, makes me think that all reporters should be interviewed at least once a year…

It goes without saying that the subject cannot see the end result before it is published or broadcast. To maximize the anxiety, the final result should be posted on the Web or some other conspicuous place where anyone can see it.”


In praise of open New Zealand

This is an old post but I’m leaving it here for the hell of it. Failed links have been updated (where possible) or removed.

I’m heartened by some ‘open’ developments in New Zealand.

Firstly, kudos to Nat Torkington and Glen Barnes for setting up, a website that promises to catalogue any government data that’s publicly available for download and further use. This is cool. It can become a useful port of call for anyone looking for data to use on a specific project.

There’s quite a lot of publicly available information emanating from the public sector, which is great. I love the images the National Library has made available under a Creative Commons licence on Flickr and I’ve been a regular user this year of Papers Past, which contains countless digitised pages from early New Zealand newspapers.

In recent weeks I’ve been trawling the LINZ website, Companies Office, and many more as I work on a Wintec project to put our National Diploma of Journalism online and make it available as a part-time course of study (a big job, as it turns out, but rewarding to work on and we are on track to open the doors next February).

But public information can be hard to find unless you’ve got a lot of time on your hands or happen to be familiar with all the ministries and their various responsibilities and websites. When you do find it, it’s sometimes ‘eyes only’ –  you can visit the website and read the stuff or maybe download it as a pdf, but there appears to be no practical way to download it in a format that allows you to re-use it or its constituent parts.

The Open Data Catalogue aims to be a centralised list of available data which links to the website that holds it and tells you whether it costs to download, how it can be downloaded (csv, xls, geo, api or other) and what licence applies. Brilliant.

Secondly, kudos to the people inside government who toil away making this data open and available. It can’t be easy making the case for it in the first place given that people have a tendency to want to keep data locked down.

That said, as Jason Ryan notes in a recent NSPC blog post, although there are more than 40 datasets on the Open Data site from central and local government agencies:

The point to bear in mind as you scroll down this list, though, is that these sets represent a fraction of the data the Crown holds. This catalogue could easily extend to hundreds of pages…

So, while we celebrate the initiative of the Ministry for the Environment releasing data under a Creative Commons license, for example, it might be worth pausing and considering how we can accelerate this process.

Later he says:

The critical issue in opening up government data [is]: culture change.

It’s not about the technology. It’s not about data quality. Or privacy. Or commercial sensitivity, or any of that stuff. That should all be dealt to as part of the everyday functioning of any administration. It is about accepting that we, the government, collect and manage this information on behalf of citizens and that it is our fundamental responsibility to make it available to them in a way that supports the creation of public and economic value.

This isn’t an add-on, or a ‘nice to do.’ It’s an integral part of our operating environment now.

I agree completely.

Jason also pointed to an Open Govt Data Bar Camp being planned, possibly next month in Wellington. A Bar Camp is an unconference – a conference where the agenda is set by participants themselves and where ideas can often be put into action on the spot – and represents a good opportunity for people in a position to help open up data to do so collegially.

The ability to get information first-hand, or at least nearer to it, holds considerable appeal for me. I increasingly find myself looking for the source of news stories so I can make up my own mind of their value. By sources I mean the report, speech, press conference, statistics etc that the story was based on or sparked by. I’m fairly sure I’m not alone in this.

So all power to the people who are making that possible. Long may it continue, and thanks.