storage units

The fourth principle of Digital Earth Experience design is to delete 90%. The illusion of cheap storage has encouraged by far the worst hoarding habits in human history. We are drowning in digital crap, and it’s going to get much, much worse. Most organizations haven’t even begun to think how to properly manage the data. In fact, they’re not even at a stage where they think it’s a problem.

In an age of the Big Bang of data, where more data has been produced in the last two years than in all of previous history, data management skills have gone backwards. Twenty years ago I saw more emphasis in organizations I was dealing with on architecting and managing data better. We have become drunk on Moore’s Law and cheap storage.

  • 90% of data never accessed three months after it’s stored (Tech Target)
  • 80% of digital data never accessed (Active Archive Alliance)
  • 90% of data not analyzed (Lucidworks)
  • 90% of data never analyzed (IDC)
  • 90% of IoT data never used (IBM)

Let me bring you on a journey to see what this data crap might look like. Over the years, we have done hundreds of Top Tasks projects, and when I looked at the data folder I saw it contained 13.7 GB.

I started a cleanup. Every time we did a Top Tasks survey we would end up with a processed file. The software is always being improved so if a client asks us to resend data we will always process a fresh file. So, there’s no need to store the old processed files. I also deleted lots of folders that were direct copies. The base survey files themselves often had copies. I deleted those.

Many years ago we needed an Excel and a CSV folder of the base results. Can’t remember why. I deleted all the unnecessary CSV folders. There were HTML files hanging around for some reason. We’ve never needed those so I deleted them. There were TEMP files. I deleted them. Sometimes, we had multiple languages in a survey. At the end of the survey we’d merge them into a master survey file. I deleted the individual language folders.

Everything I got rid of was junk, crap. It had absolutely no useful value. At the end of it all I still had everything I needed to reprocess a set of survey results for a client should they request it. I started with a folder that was 13.7 GB and I ended with a folder that was 1.07 GB. I deleted 92% crap.

Here’s an even more important question. Why am I keeping any of this stuff? I can count on one hand that is missing two fingers the number of requests I’ve had over the years from clients that required reprocessing. At most, I should only hold on to the last two years’ worth of survey data.

Digital brings out the hoarder in us. Digital feeds our insecurities. We think: “Why not store everything then I don’t have to think?” The “what if” dragon roars in our ears. Better safe than sorry. It’s so cheap. But it’s not. The most dangerous cost is the cost closest to zero. The Web is junkland. Over 90% of pages never even get found in Google.

Digital culture is the worst possible culture to address global warming. We must break our addiction to Moore’s Law and cheap storage.