Two technologies that many sites and services rely on these days are web applications and search. Those relying heavily on web apps plus Apache Lucene (news, site) and Apache Solr (news, site) now have a new tool available to help them track down and identify performance problems within their sites, thanks to a company called New Relic.
Who's New Relic?
New Relic offers SaaS web application profiling and management. Their product consists of a collection of user agents that can be installed into web applications. Once installed, they collect performance information while transactions work their way through the applications. This data is sent once a minute to New Relic's servers, which aggregate the data for the customer, who can then view the incoming metrics on a variety of devices.
New Relic's SaaS product showing web application troubleshooting information.
In aggregating statistics across all of their customers, about six months ago the company noticed something interesting happening with both Ruby on Rails and Java deployments. The number of Solr instances to handle search for applications was rising and displacing Google Search Appliance, often for e-commerce and social networking.
Deconstructing Solr Problems
Search is a critical site feature, especially in the world of Web Content Management. If your search is slow or poor, people will quickly give up and move on to your competitors. Since such a large portion of their customer base is using Apache Solr, New Relic offers specialized dashboards that appear as a user drills down to view the search application's performance.
Rather than displaying generic performance information, these dashboards show Solr cache wait times, how much memory and CPU is used for each search, what the longest searches are, which the best and worst performing are, and more. The ability to drill down into individual searches lets developers determine what's causing the problems, such as inefficiently-built queries.
New Relic's SaaS product changes automatically to show Solr troubleshooting information in a different format.
New Relic's product is not open source, though Mike Malloy, their VP of Marketing, stated that they're looking at going in this direction in the future. Perhaps that's because they've noticed that their customers are almost entirely using completely open source stacks, with the occasional proprietary components here and there such as IBM's WebSphere or Oracle's WebLogic.
The State of the Stack
One thing that New Relic's customers have found is that their issues aren't always with their own code. Sometimes the problems stem from one of the open source components they're using. In many cases, the projects behind web application components don't hear back from the users. According to Malloy, receiving real performance data from production servers has brought enthusiastic response from the projects.
Last year, New Relic started publishing "The State of the Stack" newsletter, in which they include a statistical summary of which Ruby on Rails components people have deployed along with information such as:
- What version(s) of Ruby people are using
- What version(s) of Rails people are using
- What plugins, gems, switches, framework extension packages and other open source components are in use
Doing so gives each project insight into how much their work is being used and in what context. In a few months, Malloy says they intend to do the same with the Java stack.
Connecting Projects and Their Clients
Another issue they're exploring is how to help better connect the two communities: their clients who are using the software, and the projects making the software. The idea is to allow users to request features and to share even more production performance data.
Such sharing would be the choice of each individual customer. For example, they currently have an opt-in system allowing their Ruby on Rails customers to share performance data with the core project team. Since basic performance metrics don't contain any end user consumer data, so far there haven't been any privacy issues.