sending out an SOS
AI can help us in many areas of the enterprise, but enterprise search isn't (yet) one of them PHOTO: Nico Caiser

You can't swing a cat around a room without hitting at least two experts who expect artificial intelligence (AI) to take over almost every job people perform today. 

This leads to images of HAL 9000s taking over our lives, leaving us mere mortals to dream of new goals for the AIs to achieve.

While this train of thought may lead "Dune" fans to questions what a real Butlerian Jihad may look like, the reality of the impact, and capabilities, of AI in the information space will likely be much less invasive in the short term. 

We’ve been working towards incorporating AI into business solutions for years. And finally, we've reached the point with the algorithms where search may just be smart enough.

Define 'Intelligence'

The first thing to understand is that when we talk about AI, we aren’t talking about real intelligence. At least not yet. We are talking advanced decision trees and statistical analysis of large amounts of data.

For decades, AI algorithms became more complex, seeking to eke out every bit of improvement. After the advent of big data, we realized simpler decision trees and other streamlined algorithms could deliver better results when fed more data. This led to Watson winning in Jeopardy as the proper context for each question became possible to discern.

In some ways, this is intelligence. The human brain processes large amounts of data all the time. We are often unaware of how much we are processing. 

That hunch you had last week? It was likely your brain processing large amounts of information you weren’t consciously aware of. When someone mentioned Star Wars in the '80s, you instinctively knew if it was a reference to a movie or a weapons system. 

Once again, your brain quickly assessed all the variables and determined the proper context.

Translating this into search within the enterprise has been a case of realizing just how much our brains do for us.

The Problem With Enterprise Search? It's Complicated

How do we leverage this in our enterprises? It's tricky. 

Many organizations don’t have all the necessary data to replicate the power big data brings to the table. Still, systems are getting smarter at determining the meaning of words and paragraphs. E-Discovery tools have made a sizable jump in categorizing data, but they require a lot of training to discern which pattern sets matches which business reality. 

Over 15 years ago, I presented at a conference where people were excited about the next generation of search tools. Vendors were talking about how soon search would become effective. Google made the same promise shortly thereafter with its search appliance.

These solutions worked adequately on large datasets or in well-defined scenarios. 

The problem is enterprise information is usually poorly defined and datasets are dispersed across multiple systems. 

Do You Speak Search Engine? Me Neither

We now face a two-fold problem. The first is we over-tag everything. Monica Crocker told the audience at the recent AIIM conference we needed to tag content enough to be able to find it again — and no more. 

When you have too many fields it leads to a lot of invalid tagging. For example, to find a contract we need to know what type of contract it is, who it is with and when it was signed. The value of the contract is useful information, but it won’t help us find the contract.

The second problem is when people search for things, they try to remember those tags. They want to find the content for a specific purpose. 

When you stray into content that is less well defined than contracts, knowing what you need is a feeling, an instinct. You may remember characteristics of the content but that may not be how it was tagged or categorized. You struggle to take the advanced query and properties your brain knows and forcing them into the search dialog. Full text only works when the query you are thinking of is fully understood by the search engine, which is rare on the enterprise level.

10 More Years to Solve the Search Problem

We are finally at the point where we can auto-classify content into the necessary groups to apply business rules. Fine tuning that down into smaller distinctions is more challenging as the precision increases. Subtle differences add up. Finding a specific document that wasn’t explicitly tagged in the same mindset as your query is still a challenge.

When I gave that presentation about search 15 years ago, I said my goal was that the issue of making content findable would be resolved before my kids had to deal with it.

We have about 10 more years.

I think we will make that goal. But I don’t think this wave of AI is going to solve the problem. It will make some of our business processes operate more smoothly and allow us to increase the level of automation. This should allow us to switch our focus from the more repetitive aspects of our jobs to creating more value.

However, it's not real AI, not yet at least. The AI we currently have is not quite ready to take my query, properly understand the full context and deliver the correct result. 

Until then, we can continue to use AI technology to make our lives at work and home easier. Just don’t expect it to take all our jobs quite yet.