No matter what industry you’re in, there's likely a digital component that is mission critical to your business — a simple website, perhaps, or possibly an extensive mobile app. Whatever form this component may take, these systems are expressions of the business and are part of the customer experience.
The same is true for the internal employee experience. Digital tools are a part of a normal work environment, and employees are much less productive when the tools aren’t working.
Test System Resilience With Chaos Engineering
Given how important digital assets are, it is incumbent on IT to maintain hardware and software infrastructure that is reliable and resilient. Otherwise, systems may experience disastrous failures at inopportune times and take digital assets offline. (Are there any opportune times for widespread system failure?) Most modern IT architectures attempt to create systems that embrace resilience and reliability. But do they succeed? There’s only one way to know — test the system.
Chaos engineering tries to do just this. It’s an approach to testing system resilience by using a set of random failures in a production environment to ensure the failovers and redundancies built into the system work as designed. It starts with the idea of a steady state, which is defined as the normal behavior of a system. Test engineers then make changes in the system that mimic real-world failures. It could be something as catastrophic as a major hardware failure or something simpler, like a container with a microservice shutting down abruptly.
Once the failure is introduced, the outcome is immediately measured and compared to the steady state. If the system does not respond as expected, the “failure” can be reversed and the system brought to normal.
Related Article: How Software Can Aid Testing in Agile Environments
Set Loose the Chaos Monkey
One major tenet of chaos engineering is to “minimize the blast radius.” To do that, test engineers need to plan the failure so as to minimize impact on customers or internal company processes. For example, running failure tests on accounting systems at the end of the month is probably not a good idea; doing that would certainly maximize damage.
Learning Opportunities
Webinar
Mar
29
The CX Commute vs. the CX Joy Ride
Is your CX traffic holding up your customer's journey?
Webinar
Mar
29
Is Your Microsoft 365 Data Recoverable? How to Survive the IT Hotseat When Data is Lost
See why MIcrosoft recommends third party backup
Webinar
Mar
30
An Optimized Customer Contact Strategy Combines Transparency and Intelligence: The State of Outbound Communications
Learn from the Neustar-commissioned Forrester Consulting study on challenges and solutions for contact centers
Webinar
Apr
5
Adopt a Modern Digital-First Marketing Strategy
Learn how to use a composable DXP to your advantage
One of the pioneers in chaos engineering is Netflix, which employs chaos engineering principles to ensure its infrastructure is capable of supporting its high-performance, always-on service. As part of that endeavor, it created a software product called Chaos Monkey. Available under Apache License 2.0, Chaos Monkey can terminate containers and virtual machines randomly to simulate failures of those common software components.
Related Article: Six Sigma Letting You Down? Rediscover Concurrent Engineering
Preserve Your System's Integrity
Like CI/CD and infrastructure as code, chaos engineering is an attempt to manage the new levels of scale and complexity in modern IT systems. In a way, it’s similar to using a controlled burn in forest management — creating a small failure before a big failure happens. Better a controlled failure than an unexpected and uncontrolled disaster.
You should consider adopting chaos engineering as a way to preserve your mission-critical systems, especially those that deliver vital digital assets to customers and employees.
Learn how you can join our contributor community.
