New Relic Offers Lucene and Solr Performance Monitoring

5 minute read
Dee-Ann LeBlanc avatar

Two technologies that many sites and services rely on these days are web applications and search. Those relying heavily on web apps plus Apache Lucene (news, site) and Apache Solr (news, site) now have a new tool available to help them track down and identify performance problems within their sites, thanks to a company called New Relic.

Who's New Relic?

New Relic offers SaaS web application profiling and management. Their product consists of a collection of user agents that can be installed into web applications. Once installed, they collect performance information while transactions work their way through the applications. This data is sent once a minute to New Relic's servers, which aggregate the data for the customer, who can then view the incoming metrics on a variety of devices.


New Relic's SaaS product showing web application troubleshooting information.

In aggregating statistics across all of their customers, about six months ago the company noticed something interesting happening with both Ruby on Rails and Java deployments. The number of Solr instances to handle search for applications was rising and displacing Google Search Appliance, often for e-commerce and social networking.

Deconstructing Solr Problems

Search is a critical site feature, especially in the world of Web Content Management. If your search is slow or poor, people will quickly give up and move on to your competitors. Since such a large portion of their customer base is using Apache Solr, New Relic offers specialized dashboards that appear as a user drills down to view the search application's performance.

Rather than displaying generic performance information, these dashboards show Solr cache wait times, how much memory and CPU is used for each search, what the longest searches are, which the best and worst performing are, and more. The ability to drill down into individual searches lets developers determine what's causing the problems, such as inefficiently-built queries.

New Relic's SaaS product changes automatically to show Solr troubleshootinginformation in a different format.

Giving Back

New Relic's product is not open source, though Mike Malloy, their VP of Marketing, stated that they're looking at going in this direction in the future. Perhaps that's because they've noticed that their customers are almost entirely using completely open source stacks, with the occasional proprietary components here and there such as IBM's WebSphere or Oracle's WebLogic.

The State of the Stack

One thing that New Relic's customers have found is that their issues aren't always with their own code. Sometimes the problems stem from one of the open source components they're using. In many cases, the projects behind web application components don't hear back from the users. According to Malloy, receiving real performance data from production servers has brought enthusiastic response from the projects.

Last year, New Relic started publishing "The State of the Stack" newsletter, in which they include a statistical summary of which Ruby on Rails components people have deployed along with information such as:

  • What version(s) of Ruby people are using
  • What version(s) of Rails people are using
  • What plugins, gems, switches, framework extension packages and other open source components are in use

Doing so gives each project insight into how much their work is being used and in what context. In a few months, Malloy says they intend to do the same with the Java stack.

Connecting Projects and Their Clients

Another issue they're exploring is how to help better connect the two communities: their clients who are using the software, and the projects making the software. The idea is to allow users to request features and to share even more production performance data.

Learning Opportunities

Such sharing would be the choice of each individual customer. For example, they currently have an opt-in system allowing their Ruby on Rails customers to share performance data with the core project team. Since basic performance metrics don't contain any end user consumer data, so far there haven't been any privacy issues.

Malloy stated that about 20% to 25% of their current Ruby on Rails customers have opted to share data. New Relic in turn gets occasional information requests from the core development teams, typically when they're working on the requirements for the next version of Rails or Ruby.

The company also checks in code to a variety of open source projects, often the virtual machines to help ensure that there are hooks for instrumentation and gathering performance data. They're working with the Ruby team to develop hooks for the dynamic instrumentation they can get out of the Java virtual machine.

Their engineers also are individually involved in other projects. For example, one of their engineers is involved with projects that handle payment gateways and checks in code to the Jetty team.

Using New Relic's Tools

You can install New Relic's agents within your own data centers, or you can use their tools through partners in the cloud, platform and hosting infrastructure spaces (for example, Amazon Web Services). The company offers five levels of product, from free monitors to monitoring with additional troubleshooting dashboards, all the way up to their Enterprise offering which includes product training, unlimited summary metric data storage and more:


New Relic offers five levels of pricing and support

They use a subscription payment model based on the number of hosts (not CPUs, hosts). While the company's agents are not open source licensed, their agent code is published on github, where people fork the code to create (unsupported) extended versions.

For those whose Web CMS lives in the Ruby, Ruby on Rails, or Java worlds, New Relic's offerings could be worth checking out. And if you handle your search with Solr, trying them certainly couldn't hurt.