Search Is Getting Interesting Again

2015-22-April-Searching-Eye.jpg

The thing about generalizations is that you can always find an exception.

Earlier this year I wrote an article, ”Search Isn’t Dead: It’s Gone Incognito.” In it, I argued that search has become so embedded in applications and such a part of everyday life that we no longer think of it on its own. It lives on in content management systems, databases and app repositories.

There are at least three or four exceptions to this premise. And it’s the exceptions that are the most interesting.

The Golden Age of Search, Revisited

Ten years ago, Autonomy surprised the market by acquiring the larger and more established Verity. Just a few years after that, Microsoft sent a shock through the enterprise search business by acquiring FAST Search -- and the consolidation race was on.

Soon, only a handful of serious enterprise-scale search platforms were available that could handle real-world enterprise-scale search problems.

Consider the innovation and excitement going on within the Apache Lucene-Solr project. Lucene has been around in various forms for nearly 15 years. At that point in time, the "real" search platforms included Verity, Autonomy, Fulcrum, Excalibur and others -- a veritable golden age of search.

Yonik Seeley created Solr as a website search platform, built on top of the Lucene kernel in 2004. It's been generally available for over eight years. 

Lucene and Solr became perhaps the most used search technology in the world -- in many cases making their way into companies under the radar

Then a funny thing happened: Tools built on Lucene and Solr became serious contenders. Consulting companies used both to provide solutions for their customers, and the products became increasingly capable and enterprise ready. Lucid Imagination, now Lucidworks, was founded just before Microsoft acquired FAST to help companies implement Solr solutions at scale.

Within a few years, Lucid had created a set of tools and a user interface for enterprise use of Solr, including a crawler, console and rudimentary reporting and it started to enjoy success as an enterprise search platform.

And more recently, Attivio and Elasticsearch were founded, providing search technology powered by Apache Lucene. Like Lucid, both built enterprise tools to make the platform even more powerful.

Today, Lucidworks has Fusion, an enterprise class search platform based on Solr. Fusion includes one of the most powerful crawlers you’ll find today, easily as good as the former champion, the FAST ESP crawler. It understands document level security, index pipelines, a number of connectors and is multithreaded for great scalability.

At the same time, Elasticsearch has taken its flagship product, extended with support from open source projects Logstash and Kibana, to create what at first seems like a graphics-capable log indexer, that is growing to an enterprise application with monitoring and repository security.

The Reborn Old Guard

The long established enterprise products are also enjoying a renaissance. Microsoft put the FAST technology and teams from the acquisition to good use, producing a SharePoint product with "power search" that offers innovative capabilities. Coveo also offers powerful enterprise search for Windows server, and counts hundreds of Fortune 500 companies among its customers.

And the good times spread to the niche players as well. Recommind fills the void left by Autonomy’s exit to serve the legal market. And MarkLogic, once positioned as an XML search platform (with Ultraseek roots), provides strong NoSQL database capabilities with search built in as part of the bargain.

Many of these platforms -- notably Lucene and Solr -- drive a growing number of hosted search services that provide great search service without the need to license any software or hardware, simply pay a monthly fee. These hosted services often lead the way in search and reporting.

The New Search

Like so many technologies that appeared to be on the way out, search is everywhere, and enjoying more innovation and excitement than it has had for years.

To think, just a short while ago the enterprise search market seemed like a desert. It’s in bloom again. 

Creative Commons Creative Commons Attribution 2.0 Generic License Title image by  kenny barker