Archive for the ‘Quality’ Category

I heard a debate on the radio on Tuesday about whether koalas should be classified as an endangered species. There’s an article from ABC news from last month that covers the issue quite well. Oddly enough, I was reminded of it when I had a chat with Gartner analyst Andrea Di Maio that same evening when he pointed out what he called the asymmetry of Gov 2.0. What he was referring to was the fact that many communities have data of their own, and that the standards that we are demanding of government are in no way being reciprocated in terms of what is expected of communities. A question for us and for our customers (typically government agencies) is what should government do with data owned and collected by the community?

How many koalas are there? Are they a threatened species, or endangered? What do we need to do to make sure that Australia retains a diverse, healthy population of koalas? This is a hotly debated topic, with the government accused of siding with property developers at the expense of many hectares of koala habitat. As much as I’m worried about predictions of extinction of koalas within 30 yrs, I’m not trying to push either side of the argument in this post. I’ll leave that to those who are better informed about this. What I do want to do is explore what should be done with “unofficial” statistics.

So here’s the problem: we have official statistics produced by government derived from data that is objectively collected, categorized and disseminated in keeping with scientific survey practices. And then on the flip side, we have passionate communities conducting their own research which increasingly seems to involve collecting data and producing statistics. In terms of quality, I imagine that the output varies a lot. But it is data, and potentially useful data. How should government deal with that, especially where they plan or need to have data that clearly overlaps with what already exists? Could government help turn them into official statistics? I would suspect that in many cases the answer would be a rather emphatic no. However, perhaps there would be cases where there may be some benefit to government to be gained from acknowledging and making some use of community-sourced data.

At the front end of the statistical business process model published by the UNECE, there is a planning phase where existing data sources are considered for inclusion in official collections. Here’s what the UNECE says in step 1.5:

“Check Data Availability: This sub-process checks whether current data sources could meet user requirements, and the conditions under which they would be available, including any restrictions on their use. An assessment of possible alternatives would normally include research into potential administrative data sources and their methodologies, to determine whether they would be suitable for use for statistical purposes. When existing sources have been assessed, a strategy for filling any remaining gaps in the data requirement is prepared…”

I’d take away from that that the authors had absolutely no thought in their minds about community data. So, if I was a community activist, I’d say that means that there is room for it to happen. After all, it doesn’t explicitly preclude a statistician from asking around to see if anyone else is out there counting our cuddly little Aussie icons. Perhaps there would be valid cases where government could collaborate in some way so that either the quality of the output is improved, or at the very least it can be better understood and therefore used appropriately.

There are a couple of issues that spring to mind…

• Biased and/or poor quality evidence. Communities are typically passionate and biased to a particular point of view. With no standards or checks in place to determine data quality, the government would be right to be highly skeptical of any “facts” presented. The CEO of the Australian Koala Foundation (AKF) noted that over 20 years, 2000 field sites have been looked at and over 80,000 trees. Is that enough? For what? Should the government do more? Well, at the very least, they should do some due diligence, or even better, demand a bit of transparency of the AKF, which I suspect they would be quite willing to provide. It’s right that we demand transparency of the government, but it is equally right that community groups offering evidence to support their claims should be held to a similar standard. Maybe government departments need to band together to demand Community 2.0 ?
• Inappropriate use of anecdotal evidence. Let’s face it, right now there are no doubt many policies that are based on little more than personal opinions of government executives rather than any solid evidence. People regularly draw conclusions based on direct experiences, or from stories of those they trust. Here’s a simple case in point from a comment on the ABC’s article: “Last year in the Otways I saw koalas where I had never seen them before. Seems to me that their numbers are increasing and a good thing too.” Let’s hope that’s not from environment minister Peter Garrett. This is a great evolutionary attribute that allows us to form opinions on things that might actually affect us but it doesn’t serve us well when we choose to use it to form a model of complex, widespread populations with many different local influences at play. I hardly need to point out what a huge role data can play when it comes to making informed decisions.
• Real experts in the public ready to make a contribution. There are many informed and passionate members both within the official communities, as well as in the public at large. What if we could give them a little bit more in the way of facts and figures to work with? As it is, there is a fair bit of scientific knowledge introduced by commenters. One knowledgeable commenter had a fascinating insight into the problem: “Koala populations are notoriously difficult to monitor. They are such a specialized animal that a minor change in habitat can lead to local extinctions in one area while they pop up somewhere else where they haven’t been seen in living memory.” Well, it sounds like he knows about it. It would be helpful if there was a way for him to easily reference credible evidence to back that up.

As one commenter noted: “At least it’s a positive step to have some dialogue over koala population density, and clearly there are big differences in estimates.” Yep, that pretty much sums it up. The question is, how do we get to the next step? Personally, I don’t know. I think that Andrea got it right when he said that government should acknowledge the existence of the community data sources. What they do next is an open question. Having surveys with tens of thousands of data points may still be unreliable depending on the use, but it may be better than making conclusions based on what Fred saw on his weekend trip to the Otways. I’d love to hear what other community vs government data debates people have had, and what the outcome was.