{"id":13387,"date":"2020-12-11T08:32:42","date_gmt":"2020-12-11T13:32:42","guid":{"rendered":"https:\/\/www.eginnovations.com\/blog\/?p=13387"},"modified":"2022-08-23T00:15:34","modified_gmt":"2022-08-23T04:15:34","slug":"what-is-chaos-engineering","status":"publish","type":"post","link":"https:\/\/www.eginnovations.com\/blog\/what-is-chaos-engineering\/","title":{"rendered":"What is Chaos Engineering and Why is it Important?"},"content":{"rendered":"
Robust, resilient IT systems are crucial to data-driven operations. Whether these systems drive internal processes or deliver customer-facing services, the need for reliability and availability remains the same.So, why would you deliberately try to break your services?<\/p>\n

What is \u2018Chaos Monkey\u2019?<\/h2>\n

\"\"Chaos engineering does just that – deliberately terminating instances in your production environment. Online video streaming service Netflix was one of the first organizations to popularize the concept with their Chaos Monkey engine<\/a>.<\/p>\n

When Netflix began migrating to the cloud in 2010, they found a potential problem with hosted infrastructure – hosts could be terminated and replaced at any moment, potentially affecting quality of service. To ensure a smooth streaming experience, their systems needed to be able to manage these terminations seamlessly.<\/p>\n

To assist with testing, Netflix developers created \u2018Chaos Monkey\u2019. This application runs in the background of Netflix operations, terminating services randomly.<\/p>\n

The Netflix Chaos Monkey is perhaps the best-known example<\/a> of chaos engineering. And as cloud services mature, this chaos engineering methodology will gain in popularity.<\/p>\n

Why would you deliberately break your IT systems?<\/h3>\n

At the heart of the chaos engineering model is the concept of deliberately breaking things in your production environment<\/em>. But why would you do that? Why not restrict testing to the dev environment?<\/p>\n

\n

\u201cNo amount of testing can prove software right; a single test can prove software wrong.\u201d<\/p>\n

\u2013 Amir Ghahrai<\/div>\n<\/div>\n

\"\"Using chaos engineering principles, you introduce an important element of randomness into testing and accelerate the process of identifying single points of failure. System failures are rarely predictable, and the chaos monkey can surface issues that have not been previously considered. If you only ever test for what you think may break, other important issues may be overlooked.<\/p>\n

In this way, random outages help to keep testing honest. The testing scripts cannot be skewed, shortened, or cheated, and every fault identified is real – you can literally see the problem and its effects.<\/p>\n

Benefits of Chaos Engineering<\/h3>\n

Conducting tests on the production system is quite a high risk. You will probably need a relatively robust, mature platform before you unleash the chaos monkey. However, there are also some benefits.<\/p>\n

First, you probably do not have to replicate your entire production environment for testing, which helps to reduce costs<\/a>. It is also almost impossible to properly the simulate effects at scale in a development environment.<\/p>\n

\n

\u201cThe impact of an extended outage would depend on the scale of the cloud provider: an incident that takes a top-three cloud provider offline in the US for three to six days would result in losses of between $6.9bn and $14.7bn, and between $1.5bn and $2.8bn in industry insured losses. A cyber-incident that takes a 10th to 15th placed cloud provider offline in the US for three to six days would result in losses of between $1.1bn to $2.1bn and between $220m and $450 million in industry insured losses.\u201d<\/p>\n

\u2013 ZDNet<\/div>\n<\/div>\n

Second, there is an added incentive to address issues quickly. Any breakages caused by the chaos monkey need to be fixed as fast as possible to maintain an adequate level of service for customers. It\u2019s also worth remembering that building fixes in the production environment will dramatically reduce time to deployment.<\/p>\n

Breaking things the correct way<\/h3>\n

Developing meaningful fixes after a chaos monkey breakage is often a two step-process: a quick \u2018patch\u2019 to restore operations followed by a more in-depth code update.<\/p>\n

Chaos tests are best performed in four cases:<\/p>\n

    \n
  1. When deploying new code<\/li>\n
  2. When adding dependencies<\/li>\n
  3. As usage patterns change<\/li>\n
  4. When mitigating problems<\/li>\n<\/ol>\n

    Although random, chaos tests should not be completely uncontrolled. In many cases, the monkey should only be unleashed on a sub-section of the system to test a specific hypothesis. Only if the test is passed, you should widen the scope of the test to assess other parts of the system.<\/p>\n

    Along with randomly crashing services, chaos engineering also requires an effective monitoring system. This will help you assess the impact and severity of an outage and its effect on the user experience, for instance. Application tracing is absolutely critical for identifying the source of any failure and the modules that require work.<\/p>\n

    Wherever the chaos monkey exposes blind spots in your system design, application monitoring<\/a> can help you understand them better. This allows you to formulate robust fixes and updates. Monitoring will also allow you to assess the efficacy of each fix, verify that future outages can be prevented, and that the system continues to meet your performance requirements.<\/p>\n

    From a customer\/user-facing perspective, monitoring also allows you to assess the impact on digital user experience<\/a>:<\/p>\n