There is a new wave of open government data scheduled to crash over the US on January 22 resulting from the government’s Open Government Directive. Is the government paying enough attention to data privacy issues that this deluge could trigger, and how aware are agencies of the well-established fact that anonymizing data is often an inadequate means of protecting privacy in public sector information, and that in many cases more “scrubbing” of the data is needed before any part of it can be safely released for public use?
Until recently, many government agencies have not been motivated to provide data transparency. Compared with the work that directly aligns with their mission and funding being a visionary supporter of the principles of transparent government is not really high on the agenda. In fact, in many cases, the message from up high hasn’t really reached them at all (one senior US government official’s take on Gov 2.0 was “oh, that’s a subset of Web 2.0 isn’t it?”). If you add to this reluctance the quite significant disincentives such as the risks of being too transparent, inadvertent privacy breaches, and plain and simple costs, then it’s not surprising that the average department hasn’t been as enthusiastic as the Gov 2.0 activist community might like them to be. And if the ROI on the whole deal is often external, why bother?
Well, there’s nothing like a directive straight from the top to get things moving. As of December 8, U.S. federal agencies had 45 days to get three “high-value datasets” published online and available through data.gov. Wow! Having worked with national statistics agencies for many years, I have some grasp of how long they typically take to publish data and it’s often longer than this, especially when you are dealing with data that has not previously been published. Of course, the data in some cases might be basic lists of non-sensitive material, in which case perhaps it is not too much extra work to make it suitable for public access. What I’m interested in examining is what it will take for agencies that don’t have it that easy, who will need to derive statistics from their data, or reduce it in some way to make it “safe” for public consumption.
Firstly, why bother publishing statistics if the raw data is available? Isn’t the open data community interested in getting “raw data now”, so that it’s quick for the agency and promises maximum flexibility for users? The reality in many cases — and one that seems to still be ignored by some who work in Information Management — is that even after you “de-identify” data by stripping obviously identifying attributes from it such as names, addresses, SSNs, etc, it does not necessarily protect privacy. It can still be a fairly trivial exercise for an ill-meaning data analyst, or even a non-technical person in many cases, to re-identify many of the people in the list. That is why in many cases we’ll see statistics being released about the data, rather than the raw data itself.
Associate Professor of Law Paul Ohm from the University of Colorado released a paper about the “Surprising Failure of Anonymization” last year, citing some prominent cases where anonymized data was re-identified and pointing out that there are many laws and regulations that are based on the false assumption of anonymization being a panacea for data privacy protection. In one example he describes, a researcher demonstrated how 87.1% of people in the U.S. were uniquely identified by their combined ZIP code, birth date, and sex. He also covers the AOL search data scandal, where individuals were identified from vast volumes of data by their unique search habits, uncovering some embarrassing personal information along the way.
While the individual agencies may not all have a clear understanding of all the potential privacy issues related to open data, at least the federal administration does have a focus on this. The directive itself states that data can only be made available “subject to valid privacy, confidentiality ….. restrictions”. In addition, the “Concept of Operations” paper for data.gov does have privacy in its sights, stating that there will be working groups looking into privacy issues arising from how data is mashed up and/or used in applications. I would point out that these groups could make an early head start simply by reading Paul Ohm’s paper, and not wait until after this round of data has been released. It seems that for the moment at least, the idea of what constitutes adequate privacy protection for open data is really up to each agency to decide.
While the working groups deliberate how privacy issues that result from data mashups and the like should be addressed, many datasets will be posted to data.gov and despite the proven limits of the effectiveness of anonymization, the experience that my colleagues and I have gained from talking with people who work in Information Management in government is that key staff in at least some agencies are not sufficiently aware of this, and that in their view, anonymization is essentially all you need to do to make data safe for release. I’d be interested to know if this agrees with others’ observations.
My observation regarding government’s understanding of data privacy issues is based largely on anecdotal evidence collected by myself and my colleagues. Perhaps I am overstating things and agencies do have the required skills and knowledge to release data safely. It would be good to hear about how different agencies are dealing with the Open Data Directive and what you think about the challenges of releasing useful data without unduly compromising privacy.
Note: Ohm’s paper is fairly lengthy. For a very interesting summary of the paper, you can check out this post on ars technica, which sparked a lot of debate regarding the importance of privacy.
