Today was the second and final day of Lucid Imagination’s annual Lucene love fest, Apache Lucene EuroCon, in Barcelona, Spain. The information exchange, networking and new ideas continued to flow in sessions, halls and over the free wine at lunch.
Open Source Search -- It’s Not All Rainbows and Puppies
The attendees, speakers and vendors at Eurocon are clearly enthusiastic about search. It’s not often that you hear 10 conversations about computational linguistics and relevancy scoring on your way to get a cup of espresso, but that’s the norm in Barcelona this week. All of the major players for Lucene/Solr, including Lucid Imagination’s own heavily involved staff, have turned out for the festivities.
With all of the enthusiasm, you could easily get the impression that Lucene/Solr is a magical solution that has somehow managed to avoid the challenges and pain points associated with other enterprise search platforms. It has not. This is a fact that Grant Ingersoll, Chief Scientist for Lucid Imagination, readily admits.
Enterprise data is growing rapidly in volume and diversity which makes implementing search increasingly challenging. There is no platform that will allow you to deliver a solution perfectly suited for your organization, unless you have very simplistic needs, with just a few button clicks -- no matter what the sales guy says (ahem Google).
Day Two Festivities
Day two’s sessions heavily focused on the challenges organizations and vendors are encountering implementing search using Lucene/Solr and how they solved them. Even in cases where the solutions weren’t directly applicable to the audience, you could almost see the ideas forming above the attendees’ heads as they considered how the last sentence could reshape their approach.
The morning began with a well-received speech by Michael Busch, Tech Lead for Search Infrastructure at Twitter. Busch discussed the modifications Twitter made to Lucene to support more than 1.5 billion queries -- yes with a ‘b’ -- and 200 million tweets of additional data per day. The problem became even more complex with the recent addition of image and video search on the same engine. Busch was clearly a rock star at the conference and could easily pick up all of the hot Lucene chicks if any of those mythical creatures existed.
The day continued with explanations of
- faceting, the technique of providing categories to aid users with navigation that Endeca and web search popularized
- troubleshooting Solr problems using explain plan information, which included a demo new, free visualization software that makes the data easier to understand
- improving search using analyzers
and of course what conference would be complete without at least a couple of sessions on the cloud. Jon Gifford discussed how Loggly, a logging software-as-a-service platform, is using SolrCloud to deliver its solution and how they resolved performance issues.
Conference Take Away
It is easy to get enamored with technology, especially when its star is rising. However, no technology, no matter what problem it is solving is perfect. I think Lucene EuroCon provided attendees a good mix of hope for the future of the Lucene/Solr platform and the reality that you WILL encounter problems implementing search.