Salesforce reported an outage with its Pardot B2B marketing automation platform that began Friday, May 17, lasted into the next week and left users unable to perform their marketing tasks — but more than able to complain about it on Twitter.
Salesforce Is Down! Salesforce Is Down!
The internet erupted when Salesforce Pardot users began to experience performance issues on May 17. Customers across several Salesforce North America and European instances experienced the service disruption. The cloud outage happened because of a faulty database script deployment, according to Salesforce.
Salesforce acknowledged the problem at 10:20 a.m. EDT on May 17 and was still providing updates as of 4:10 p.m. EDT on May 21. It first reported restoration of access and permissions May 18 at 6:24 a.m. EDT. However, Salesforce CTO and co-founder Parker Harris was tweeting about the outage on May 20.
While most affected customers have had their permissions restored, we're still working to resolve permissions access for a subset of users in affected orgs on NA53, NA57, NA59. We're all hands on deck to remediate the issue for all customers. https://t.co/mjgZrP3JdI for status.— Parker Harris (@parkerharris) May 20, 2019
In other words, this was a long, hard fight for Salesforce and its affected customers. This particular outage, though, is about more than Salesforce going down and customers complaining on Twitter. It reinforces the delicate state of operating major business functions on cloud software. While the promises and outcomes of the Software-as-a-Service (SaaS) world can be awesome, it can all go downhill in an instant.
There are lessons to be learned here from those who experienced the Salesforce outage and those who observed. How did those users respond? What’s it like to be inside a major outage that runs a large chunk of your business and marketing operations? What do you do next? Can you prepare?
Related Article: How Your SaaS Company Can Survive a Cloud Outage
Lost Productivity, More Game of Thrones Chatter
Jarin Chu, senior director of sales and marketing operations at OpFocus, told CMSWire the Salesforce outage resulted in lost time, lost productivity, more water cooler chats about Game of Thrones and “lost trust in Salesforce transparency and security.”
“We did what work we could without our data, but access wasn’t recovered until Sunday night (May 19) and granting correct permissions to all the different standard and custom profiles still took hours,” she said. “Sales couldn’t close deals, couldn’t send out contracts through Salesforce.”
In the early hours of the outage, Chu and her colleagues found themselves scouring forums, Twitter and Reddit to get a sense of what was happening. “I ran a team meeting using other integrated tools, and most worked out of Gmail,” she said. “If both Gmail and Salesforce went down ….”
From her vantage point, it just was not the best start to a Friday morning. Team members received an email from the Salesforce administrator that only admins had access to Salesforce objects. “We could log in, but all we’d see is insufficient privileges on records,” Chu said.
No Easy Way to Prepare
What can organizations do to prepare for such a disruption? Companies that depend on systems like Salesforce need to have a business continuity plan to continue operations if Salesforce and other hosted systems go down, according to Robert Reeves, Datical co-founder and CTO. “Salesforce users can enact business continuity plans that include syncing your Salesforce tasks and calendar with Office365, Exchange or Google Apps,” he said. “However, since Salesforce has become the system of record of most companies using them, having a backup of the data will be very difficult. After all, avoiding hosting your own data was the point of using Salesforce in the first place.”
Paurush Saxena, senior vice president of technology at Incentive Technology Group (iTG), said one of his company’s customers who use Pardot as a reach-out tool for the elderly population was not able to do so during the outage. “In this specific case, while every business case and their usage of Pardot is different,” Saxena said, “It should be a wake-up call for all of us to dust off contingency plans and ensure that you actually execute them on a regular basis."
Related Article: Salesforce Customers Lose CRM Data in 20 Hour Outage
Detailed Outage Plan Created
Tom Burton, COO of LeadSmart Technologies, was part of a team that experienced Salesforce outages. His team proactively reached out to clients and users during the outage.
The big lesson for them? “We also realized we were certainly not as prepared as we should have been and have subsequently created a detailed ‘outage plan’ that will be deployed immediately if outages were to occur in the future,” Burton said. “Kind of like having an earthquake or disaster plan with your family, we realized we needed a well-thought plan and communication strategy for our clients when the unexpected occurs.”
Questions to Ask Potential New Vendors
Should you jump ship if your cloud software provider frustrates you with its handling of a major outage? Michael Biggerstaff, chief inspiration officer for Nxtbook Media, said his team will be looking for opportunities outside of Salesforce.
The stoppage last week for his company happened at 9:03 a.m. EDT. Team members spent the next 45 minutes trying to uncover something they may have done or a hardware issue, network-down possibility. "During this 45 minutes we looked internally, along with looking at the Salesforce website that had nothing listed, and social media," Biggerstaff told CMSWire. "Social media, Twitter specifically, was the first place we realized this was a Salesforce issue due to other customers talking about it."
During his software search, Biggerstaff said he'll be asking more questions about how his potential provider handles problems: notifying clients, how it is done, how often it happens in its history. What they do about downtime specifically. How is it handled in general? Are there any options or workarounds when the cloud is down?
"This might be something that is just the down side of SaaS," Biggerstaff said. "To be clear I don’t have an expectation that software is always up and running; life happens. To me it is all about how effective you are in handling the problems when it does go down. Your customers need to know right away. It is not possible to over communicate when a problem occurs."
Related Article: Blame People for Cloud Downtime Woes
Realities of the SaaS World
Is this Salesforce outage just another sign of the inevitable risks of marrying critical business outcomes to the performance of a SaaS provider? Is it just something we have to, well, live with? “Typically Salesforce upgrades are unperceivable to end-users,” said Shell Black, president and Salesforce MVP Hall of Fame member who runs his own Salesforce consultancy. “In some ways the risks are greater in a multitenant environment because a mistake can impact multiple customers — as with this outage.”
The counter argument? “Most companies,” Black said, “are ill equipped in talent, experience, funding and infrastructure to be managing critical business systems and are better off using SaaS alternatives.”