Resilience as a service

• US 9,747,153 B2
• Filed: 06/22/2015
• Issued: 08/29/2017
• Est. Priority Date: 06/22/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Raghavendra, Chetan Pentam, Klementiev, Dmitri A., Jain, Varun, Zervos, Michail, Gakkhar, Dhruv, Kolluru, Satya Sudha, Thangavel, Lenina Devi
• Primary Examiner(s): Iqbal, Nadeem
• Application Number:
  US14/745,756
• Publication Number:
  US 20160371134A1
• Time in Patent Office: 799 Days
• CPC Class Codes: G06F 11/0721 within a central processing unit [CPU]
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 11/3676 for coverage analysis
• CPC Class Codes: H04L 41/5058 Service discovery by the service manager
• CPC Class Codes: H04L 43/55 Testing of service level quality, e.g. simulating service usage

Abstract

Systems and methods disclosed herein are directed to creating a service directory of dependencies for services running on a system, wherein instances of a first service are dependent upon instances of a second service. The directory of dependencies comprises metadata associated with connections between the services. The system injects faults targeting all levels of the dependencies. The system is monitored to detect failures created by the faults. The injected faults are selected from transport layer faults, memory pressure, processor pressure, storage pressure, virtual machine restart, and virtual machine shut down. A domain name service is monitored to identify names that are resolved for the services. The service directory is then updated continuously with additional dependencies using information about the resolved names. The faults may be injected in a guided manner, wherein the scope of the faults is increased in steps over time to identify a failure point in the system.

Claims

1. A system, comprising:

at least one processor running instances of two or more services, wherein instances of a first service are dependent upon instances of a second service;

a fault-injection manager that introduces a fault-inducing condition in the system;

a health monitoring manager that detects failures in the system; and

wherein the fault-injection manager identifies the first service as vulnerable to the fault-inducing condition when the fault-inducing condition causes an instance of the second service to fail,

wherein the fault-injection manager introduces the fault-inducing condition in a guided manner, wherein a scope of the fault-inducing condition is increased in steps over time.

2. The system of claim 1, further comprising:

a dependency graph manager that identifies dependencies between service instances.

3. The system of claim 2, wherein the dependency graph manager identifies the dependencies among service instances using a directory name service, IP sniffing, packet sniffing, code instrumentation, or code tracing.

4. The system of claim 2, wherein names that have been resolved in a directory name service are stored to a directory name service log.

5. The system of claim 2, wherein the dependency graph manager continuously updates the service instance dependencies by analyzing names that have been resolved in a directory name service in real-time.

6. The system of claim 1, wherein the fault-inducing condition includes one or more of:

transport layer faults, memory pressure, processor pressure, storage pressure, virtual machine restart, or virtual machine shut down.

7. The system of claim 1, wherein the fault-injection manager introduces the fault-inducing condition in a guided manner by increasing the fault-inducing condition until failure occurs.

8. A method, comprising:

creating a service directory of dependencies for services running on a system, wherein instances of a first service are dependent upon instances of a second service, the directory of dependencies further comprising metadata associated with connections between the services;

continuously updating the service directory with additional dependencies, based on results of monitoring of a domain name service resolving names for the services;

injecting faults in the system, the faults targeting a plurality of levels of the dependencies; and

monitoring the system to detect failures created by the faults.

9. The method of claim 8, wherein the injected faults are selected from a limited set of faults.

10. The method of claim 9, wherein the limited set of faults comprise:

transport layer faults, memory pressure, processor pressure, storage pressure, virtual machine restart, and virtual machine shut down.

11. The method of claim 8, wherein:

monitoring of a domain name service includes monitoring the domain name service to identify names that are resolved for the services, by the domain name service; and

the service directory is updated with additional dependencies using information about the names that have been resolved.

12. The method of claim 8, further comprising:

injecting the faults in a guided manner, wherein the scope of the faults is increased in steps over time to identify a failure point in the system.

13. The method of claim 8, wherein the faults are injected at a plurality of levels of the dependencies.

14. The method of claim 8, further comprising:

identifying a system failure;

stopping the fault injection; and

identifying services associated with an injected fault.

15. The method of claim 8, wherein the injected faults are associated with communications between dependent services.

16. The method of claim 8, wherein the injected faults are directed to instances associated with roles.

17. A system, comprising:

a processor; and

a memory containing a program, which, when executed on the processor, performs an operation for identifying an impact of system fault conditions, the operation comprising;

creating a service directory of dependencies for services running on the system, wherein instances of a first service are dependent upon instances of a second service, the directory of dependencies further comprising metadata associated with connections between the services;

continuously updating the service directory with additional dependencies, based on results of monitoring of a domain name service resolving names for the services;

injecting faults in the system, the faults targeting all levels of the dependencies; and

monitoring the system to detect failures created by the faults.

18. The system of claim 17, wherein the injected faults include one or more of:

transport layer faults, memory pressure, processor pressure, storage pressure, virtual machine restart, or virtual machine shut down.

19. The system of claim 17, wherein:

monitoring of the domain name service includes monitoring the domain name service to identify names that are resolved for the services, by the domain name service; and

the service directory is updated with additional dependencies using information about the names that have been resolved.

20. The system of claim 17, wherein the operation further comprises:

injecting the faults in a guided manner, wherein the scope of the faults is increased in steps over time to identify a failure point in the system.