blazing fire in a fireplace
What's hot in search and big data analytics this year? Read more. PHOTO: alex Lázaro

Search and big data analytics have evolved significantly over the last few years, and organizations are increasingly using these technologies to meet their mission-critical needs.  

At the beginning of 2016, we were talking a lot about machine learning and semantic search and how they would be key developments in this space. 

Those have certainly been hot topics and continue to be areas that companies are seeking to exploit for their data-driven applications. 

But what will we be talking about in 2017 as it pertains to this space?

Here’s a look at five areas you can expect to hear more of this year and beyond.

Open Source Rises to the Top

Open source technologies are becoming more prominent in a wide range of use cases, from traditional enterprise search to log analytics, e-commerce search, and even government document search.

In fact, current data shows open source search engines have gained significant popularity because of flexibility, cost and features. According to DB-Engines, Elasticsearch and Solr — two open source search engines based on Lucene — top the list of leading commercial and open source search engines.

Just last month I discussed the features and limitations of Elasticsearch and Solr in this article.

With its growing use in the commercial and government space, the move to open source will continue to be a hot topic as organizations seek greater features, costs savings, and more flexibility with their search and big data analytics solutions.

Life Without Google Search Appliance

In early 2016, Google announced its end of support for the Google Search Appliance (by March 2019) as part of its strategic move to a cloud-based platform. Since then, many have begged the question: What now?

In my article last summer, I provided some tips on moving from the GSA and looked at some of the replacement options available at the time.

As we enter 2017, there are still no concrete details from Google about a new cloud-based search solution, so users are forging ahead with seeking out and comparing their existing alternatives.  With the March 2019 deadline getting closer, we’ll be hearing a lot more about the alternatives, and maybe even get word on Google’s cloud-based plans. 

Analytics Powered by Enterprise Data Lakes

Enterprises have a lot of data but how well they use it to derive insights is key to success. Over the last year, we’ve been hearing a lot of hype around enterprise data lakes (or enterprise data hubs) to bring together data silos and make the right data available to the right users at the right time. 

There is a wide variety of structured and unstructured data in enterprise data lakes. That said, search engines are the ideal tool for storing, processing, accessing, and presenting this data because they are schema-free and can scale to billions of records.

Data lakes’ search and analytics capabilities are nearly endless when we combine search engines, big data techniques, and visualization dashboards in groundbreaking use cases such as bioinformatics, precision agriculture, and precision medicine.

As data lakes continue to gain popularity as a way to store massive amounts of data and analytics, we’ll see organizations continuing to have the conversation in 2017 about how to best exploit this.

Search Engines Become 'Insight Engines'

Just like Google, Cortana and Siri, search is becoming much more than just keyword matching.

We’re now heading into the age of search results personalization. Search engines are becoming personal digital assistants or as Gartner calls it, Insight Engines. This was made possible with big data analytics techniques like machine learning and predictive analytics.

It’s pervading the modern business world in a multitude of use cases from intranet search, to e-commerce search, recruiting, medical research, media and publishing, and many others. Organizations are just beginning to skim the surface on how real-time personalization is significantly enhancing their operations so we expect a lot of talk about how to implement this in the coming year.

Search Engine Scoring

We know that analyzing statistically valid scores helps increase search engine relevancy over time. But, while this method can significantly increase business value and the bottom line, not many organizations have started engine scoring or are implementing it effectively.

With that said, we are observing proven success with newer, better algorithms used in the scoring process, some of which are discussed in this article.  This is, without a doubt, a solid technique that will continue to grow and be discussed in the coming years as organizations seek to improve their user’s search experience.

In conclusion, with the rise of open source, massive volumes of structured and unstructured data, and the need to do complex analytics, we will continue to hear a lot on these topics throughout 2017 as search and big data continue to converge.