A little while ago, we reported the CIA was pouring money into the open source Apache Lucene, Solr search. And (shocker!) the CIA (yet again, through their technology VC arm In-Q-Tel) continues to invest into search and crawling-related technologies, according to Wired magazine.
Visible Technologies Make it All Very Visible
In our previous coverage of Visible Technologies, we told you about their suite of on-demand enterprise-level social media monitoring tools. truVOICE is one of them. It’s their flagship product for data collection and tracking of social media conversations. We’re thinking that this product is the main attraction for In-Q-Tel and the undisclosed amount of investment.
truSEARCH, another module of the suite family, is also an important component in data mining with key-phrase drilldown, topic discovery and general search functionality. Visible’s technology can also score what they scrape and assign a positive/negative/neutral rating to it -- all displayed in a nice dashboard view in truVOICE.
Wired says “Visible crawls over half a million web 2.0 sites a day, scraping more than a million posts…” That would include all those comments you make on Amazon, Flickr, forums and discussion boards.
What’s in it for the CIA?
It’s not a conspiracy theory. It’s a normal government reaction to have granular access to the sprawling social media and social networking spaces. Wired says that the investment is “part of a larger movement within the spy services to get better at using “open source intelligence.” By “open source” they mean data in public channels that may be on the surface, or hidden in the volumes of social content generated every second. Geographically, this is, of course, not limited to the United States.
Looking at it from the privacy perspective (yup, privacy is still dead), it’s only expected that something like this would happen sooner or later. Everything that you put out in the web universe can be found, dissected, analyzed and acted upon (be it by the HR departments and headhunters, or the government).