control valve on a pipe
PHOTO: Alan Levine

Will there ever be one enterprise information aggregator? That was the question Jed Cawthorne posed last month. I suggested on Twitter that search could be the integrator, and Jed challenged me to set out how this would work. I like a challenge! To make this a stand-alone column I’ll start with the penultimate paragraph from his column:

“Might dissatisfaction with the whole remote working experience, the constant context switching between disparate interfaces, the lack of familiarity with a multiplicity of video conferencing, shared white board, social collaboration and other tools push the industry towards some form of interoperability? Will I ever get my one aggregator to rule them all, or will we go further than that and drive towards standards-based interconnection between different vendors products using a truly 21st century equivalent to SMTP (but with security built in)?”

Information Aggregation Shouldn't Be a Problem

The problem Jed raised shouldn't be a problem in the first place and the remedy should not be IT. The issue isn't new, as a report published in 1966 makes clear. The fundamental issue is we are not managing information as an asset and most organizations do not have an information management (IM) strategy, IM policies and IM training.

The mess many companies find themselves in is because no one person owns information assets and information distribution channels who can work with IT to see where technology can support information content architectures. It is worth noting at this point that "unstructured" information actually has a very strong structure based on grammar and document layouts.

Organizations should have an IM strategy in place around the information lifecycle of Create, Store, Discover, Use, Share, Review and then either Archive, Delete or Revise. (Discover is not just about search but also alerting and browsing.) Within the Share section, there should be policies about collaboration which itself consists of the elements of Connection, Conversation, Coordination, Collaboration and Consultation. Discussions tend to be around "social media" and "collaboration" as generic terms, when in practice, the devil is in the detail. This stack approach may seem over-the-top but is the model used in the OSI Seven-Layer Model for network management, which remains relevant some four decades after its introduction. 

Related Article: Virtual Workspaces: How to Do More Than Just Meetings Online

Using Search for Information Integration

Search is often seen only in terms of queries and lists of responses. That undersells its value. Search technology has been used in media monitoring for many years as an integration platform across potentially hundreds of different sources. These sources are monitored in near real-time speed using a combination of technologies. This was the basis for the IBM Streamz service and it is well worth reading the paper by Ido Guy and his colleagues at IBM Haifa. This paper dates from 2013. Given it is IBM, you can guess at how much more work has been done on this problem and potential solutions. Currently the role of knowledge graphs in these systems is of interest, and a recent paper by Microsoft gives some insights into how these can be used in media monitoring.

One element of these services is the use of sentiment analysis. In principle, this means you could be alerted only to social media items which raise concerns or challenges to the status quo and have them highlighted on your screen. Search applications are very good at security management so that is not an issue. 

This use of search-based applications is big business. Take a look at Meltwater and Signal AI as just two examples. The business is big enough and competitive enough for these and other companies to have large research capabilities. To get a sense of scale read this case study on Bloomberg.

Related Article: What Do We Mean By 'Search'?

This Is Why You Need a Search Team

These solutions could have been up and running in your organization for several years had you invested in search expertise. But as long as the epitome of search is seen as PnP additions to Office 365 because the core application is not good enough, the chances of bringing these aggregation technologies into play are zero. Many search companies are already offering this degree of integration even if not at real-time speeds, and using novel approaches to the user interface to present information in a way that allows the user to customize not only the content they see, but also the UI itself. That is why starting with the UI is so important in enterprise search.

Related Article: Enterprise Search Development: Start With the User Interface

Maintaining Competitive Advantage

As Jed rightly noted, information integration is a huge challenge in social media and other information channels. For a standard to emerge the benefit of adopting it has to be significantly greater than going it alone. Very few examples of standards exist in IT other than Ethernet and CD-ROM. Can you imagine all the social media vendors sitting around a table and agreeing on a common standard?

Using search-based applications to provide integration is entirely possible and in effect is off-the-shelf. They will enable the social media vendors to optimize the functionality of their own channel to meet what they see as specific market requirements. However, the statistics from the search application may tell a different story.