Spreadsheets Datapaloozas and the Liberation of US Government Data

A hashtag reporting the location of downed power lines. Medication error reports from the US Food and Drug Administration available through APIs. A crowdsourcing tool to geotagged photos of disaster areas to help first responders. These varied technologies are part of a major open data initiative underway at White House — and they're being propelled by a series of data-fests called Datapaloozas.

What's a Datapalooza? Think of a party that attracts a curious blend of data geeks, entrepreneurs, industry hotshots, politicians and bureaucrats. They come together in the hope of turning vast amounts of government data into apps and computer programs that anyone or any business can use.

Opening Government Data Vaults

Brian Forde, senior advisor to the US Chief Technology Officer, described the public-private partnership events as opportunities to use open data as the foundation of useful products, services and innovations. Like similar efforts on a smaller scale in New York and other cities, the initiative encourages the development of apps that utilize public data to benefit the public — a confluence of resources that could only happen at this time in the history of apps, big data, inexpensive and powerful devices, and high speed connections.

“For the first time in history, we’ve opened up huge amounts of government data to the American people and put it on the Internet for free,” Forde told CMSWire. There are more than 75,000 data sets available at Data.gov, ranging from what various hospitals charge for the same procedures to measurements of weather and climate.

But there’s still much more data to mine. Forde said “many more government datasets are still hard to find or are locked up in unusable formats.” It's the reason why the Obama Administration is now requiring governmental agencies to release newly generated data in common machine-readable formats.

It symbolizes what Forde described as “the liberation of government data.”

In the past few years, there have been a series of Datapaloozas that brought together government officials and key stakeholders. The goal is to showcase solutions that utilize these newly freed rivers of information and encourage the development of new ones. The events, including ones centered around health, safety, educationenergy and emergency preparedness, have resulted in multiple new apps, tools and services.

Making data more "liquid" — that is, open, widely available and sharable in machine-readable formats — has the potential to unlock large amounts of economic value, a 2013 McKinsey report maintains. Think census data, utilities benchmarking, real-time train movements and even social media entries.

How much value? Try $3 trillion in additional value annually. McKinsey researchers said that potential value would be divided roughly between the United States ($1.1 trillion), Europe ($900 billion) and the rest of the world ($1.7 trillion).

Emergencies, Farmers

While the newly accessible governmental data is driving many of these solutions, some involve using new tools for data capture and coordination.

Last week — at a Datapalooza held by the Federal Emergency Management Administration (FEMA), the US Department of Energy and the Weather Channel — the White House’s Office of Science and Technology Policy announced a new effort to use Twitter hashtags to track and report emergency information during disasters.

There’s also the example of a private online crowdfunding platform called Crowdtilt, which could make it easier for businesses to raise funds for disaster recovery.

Given the range and scope of projects thus far, one can imagine this data harvesting is well on the way to becoming a distinct industry. One company,  Climate Corporation, is using federal data sets to help farmers plan, manage and protect their crops. It's using open data sources from the National Weather Service, including historical records and forecasts, as well as terrain and soil maps from the US Geological Survey, crop yield records from the US Dept. of Agriculture and earth and weather data from NASA. The firm has raised more than $100 million in venture capital and was recently sold for nearly a billion dollars.

Data Coming Back

Another company, BrightScope, leverages Dept. of Labor data to generate independent ratings for 401(k) retirement plans.

Using government data on energy usage, weather and appliance energy efficiency, a firm called OPower provides personalized advice about reducing residential energy costs.

Popular real estate sites Zillow and Trulia are employing data from the Bureau of Labor Statistics, Federal Housing Finance Agency and the Census Bureau to provide insight on the housing market, including whether renting is a more economical option.

An optimist might say this all points to the day when governments keep crowdsourced, online punch lists of potholes and their fix date, or when patients can enter a medical procedure into a search engine to find easy-to-understand comparative pricing. We’re not there yet. But at a time when the US government is being criticized for taking too much data from its citizens, it’s heartening to see useful data flowing the other way.

(Title image from a recent Safety Datapalooza.)