Service directory and fault injection management systems and methods

• US 10,474,521 B2
• Filed: 08/09/2017
• Issued: 11/12/2019
• Est. Priority Date: 06/22/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Raghavendra, Chetan Pentam, Klementiev, Dmitri A., Jain, Varun, Zervos, Michail, Gakkhar, Dhruv, Kolluru, Satya Sudha, Thangavel, Lenina Devi
• Primary Examiner(s): Iqbal, Nadeem
• Application Number:
  US15/672,711
• Publication Number:
  US 20170337099A1
• Time in Patent Office: 825 Days
• CPC Class Codes: G06F 11/0721 within a central processing unit [CPU]
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 11/3676 for coverage analysis
• CPC Class Codes: H04L 41/5058 Service discovery by the service manager
• CPC Class Codes: H04L 43/55 Testing of service level quality, e.g. simulating service usage

Abstract

A system includes: at least one processor running instances of two or more services. Instances of a first service are dependent upon instances of a second service. A fault-injection manager, based on data in a service directory indicative of dependencies of services of the system, determines a fault-inducing condition to inject into the system and injects the fault-inducing condition into the system. A health monitoring manager detects failures in the system and identifies the first service as vulnerable to the fault-inducing condition when the fault-inducing condition causes an instance of the second service to fail.

Claims

1. A system, comprising:

at least one processor running instances of two or more services,

wherein instances of a first service are dependent upon instances of a second service, and wherein the dependency relationship between the first service and the second service is based at least on a dynamic naming service (DNS) log and is stored in a service directory;

a fault-injection manager that, based on data in the service directory indicative of dependencies of services of the system, determines a fault-inducing condition to inject into the system and injects the fault-inducing condition into the system; and

a health monitoring manager that detects failures in the system and that identifies the first service as vulnerable to the fault-inducing condition when the fault-inducing condition causes an instance of the second service to fail.

2. The system of claim 1, wherein the fault-injection manager selects the fault-inducing condition to inject into the system from a group consisting of:

transport layer faults;

memory pressure;

processor pressure;

storage pressure;

virtual machine restart; and

virtual machine shut down.

3. The system of claim 1, wherein the service directory includes metadata associated with connections between services of the system.

4. The system of claim 1, further comprising a dependency graph creator/updater that updates the service directory based on changes in a domain name system (DNS) log.

5. The system of claim 4, wherein the dependency graph creator/updater updates the service directory each time the DNS log is updated.

6. The system of claim 1, wherein the fault-injection manager determines the fault-inducing condition to inject into the system for a type of role instance.

7. The system of claim 1, wherein the fault-injection manager determines the fault-inducing condition to inject into the system between a role and one or more endpoints.

8. The system of claim 1, wherein the fault-injection manager increases a scope of the fault-inducing condition over time until a failure is detected.

9. The system of claim 1, wherein the fault-injection manager, until a failure is detected, increases, over time, at least one of:

a number of affected roles;

a number of affected machines;

an amount of pressure; and

an amount of delay.

10. A method, comprising:

creating a service directory including data indicative of dependencies of services of a system based on developer input,

wherein instances of a first service are dependent upon instances of a second service and wherein the dependency relationship between the first service and the second service is based at least on a dynamic naming service (DNS) log and is stored in a service directory;

updating the service directory to include data indicative of additional dependencies of services of the system based on a domain name system (DNS) log updated by a DNS server;

based on data in the service directory, determining faults to inject into the system;

injecting the faults into the system; and

monitoring the system for failures created by the injection of the faults.

11. The method of claim 10, wherein determining faults to inject into the system includes selecting a fault to inject into the system from a group consisting of:

transport layer faults;

memory pressure;

processor pressure;

storage pressure;

virtual machine restart; and

virtual machine shut down.

12. The method of claim 10, wherein the service directory includes metadata associated with connections between services of the system.

13. The method of claim 10, further comprising updating the service directory each time that the DNS log is updated.

14. The method of claim 10, further comprising:

injecting a fault into the system; and

increasing a scope of the fault over time until a failure is detected.

15. The method of claim 10, further comprising:

injecting a fault into the system; and

until a failure is detected, increasing, over time, at least one of;

a number of affected roles;

a number of affected machines;

an amount of pressure; and

an amount of delay.

16. A system, comprising:

one or more processors; and

memory including a program that, when executed by the one or more processors, performs functions comprising;

creating a service directory including data indicative of dependencies of services of a system based on developer input, wherein instances of a first service are dependent upon instances of a second service, and wherein the dependency relationship between the first service and the second service is based at least on a dynamic naming service (DNS) log and is stored in a service directory;

updating the service directory to include data indicative of additional dependencies of services of the system based on a domain name system (DNS) log updated by a DNS server;

based on data in the service directory, determining faults to inject into the system;

injecting the faults into the system; and

monitoring the system for failures created by the injection of the faults.

17. The system of claim 16, wherein the function of determining faults to inject into the system includes selecting a fault to inject into the system from a group consisting of:

transport layer faults;

memory pressure;

processor pressure;

storage pressure;

virtual machine restart; and

virtual machine shut down.

18. The system of claim 16, wherein, when executed by the one or more processors, the program performs functions further comprising updating the service directory each time that the DNS log is updated.

19. The system of claim 16, wherein, when executed by the one or more processors, the program performs functions further comprising:

injecting a fault into the system; and

increasing a scope of the fault over time until a failure is detected.

20. The system of claim 16, wherein, when executed by the one or more processors, the program performs functions further comprising:

injecting a fault into the system; and

until a failure is detected, increasing, over time, at least one of;

a number of affected roles;

a number of affected machines;

an amount of pressure; and

an amount of delay.