Varnish Software has added optional cache persistence into the latest iteration of its Massive Storage Engine (MSE), which it first released in 2014.

MSE was designed and implemented to suit the requirements of anyone who needed to combine a large cache with high performance. The new persistence capability allows websites to retain data across restarts and reboots and ensures that, in the case of a system crash, websites will not lose their cache content.

"Persistent storage has been in high demand, mostly because re-populating a cache after a restart takes considerable time, in particular when large objects are involved, and time is important when high performance is a hard requirement," the company noted in a blog post.

Both MSE 1.0 and MSE 2.0 are part of Varnish Software's Varnish Plus portfolio, which offers multiple modules to optimize web performance and scalability.

As a module of Varnish Plus, MSE was designed specifically for the high-performance needs of video distribution, Content Delivery Networks (CDNs) and large cache use cases that handle multi-terabyte data sets. A CDN is a distributed network of proxy servers deployed in multiple data centers that serve content to end-users with high availability and high performance

“The Massive Storage Engine (MSE) is one of our most popular modules in Varnish Plus. Many of our customers who are currently using MSE store up to 100 terabytes of data in their caches,” Hildur Smaradottir, international marketing manager with Varnish Software told CMSWire.

“As the need to store that much data in cache grows, as it has with the proliferation of streaming, video on demand, it has becomes imperative to have a way to persist or retain the data across both planned and unplanned Cache restarts. To respond to this need we've made persistence an optional feature within version 2.0 of the Massive Storage Engine.”

What's Varnish?

Varnish Software is the company behind Varnish Cache, the open source HTTP accelerator.

It is used by high-profile, high-traffic websites including Wikipedia, online newspaper sites such as The New York Times and the Guardian in the UK, as well as social media and content sites such as Facebook, Twitter,Vimeo and Tumblr.

Varnish claims its Cache boosts the performance of more than 2.2 million websites worldwide, serving up web content faster and decreasing server load to handle significantly more traffic.

The development of Cache goes back to 2005 and the Norwegian tabloid newspaper Verdens Gang. At that point in time, the newspaper was reaching over 45 million page views every week, half of these on the front page. Twelve servers were needed to handle the request.

Anders Berg, a system administrator at Varnish wrote a spec to try and reduced the number of servers needed and on which Varnish Cache is based. It was open sourced, available under the BSD license.

Version 1.0 of Varnish was released in 2006, Varnish 4.0 in 2014. “MSE as an addition to this is designed for customers delivering their digital content over CDNs and for customers delivering large volumes of data,” Smaradottir said.

“For instance, if a customer has terabytes of content in their cache and needs to restart their Varnish Cache instance refetching their content from the backend will have a direct cost for them (charged to their CDN provider). Now that Varnish can persist data across restarts the cost saving potential is significant."

Over the next two years, apart from developing the entire Varnish Plus suite of modules as a Software-as-a-Service offering, Varnish will continue to develop its API Engine, an API management platform that can handle 23,000 API calls per second.

“Due to the proliferation of APIs we predict that in 18-24 months most companies who have already or have plans to open up their APIs will see the need for that type of performance,” Smaradottir said.

However, the really big push in the coming months will be the development of a blueprint that allows enterprises to build their own CDNs using Varnish.