Injecting Faults at Select Execution Points of Distributed Applications

• US 20150161025A1
• Filed: 12/05/2013
• Published: 06/11/2015
• Est. Priority Date: 12/05/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Baset, Salman A., Ramasamy, Harigovind V., Tang, Chunqiang, Wang, Long, Pham, Cuong M., Singh, Manas, Tak, Byung Chul
• Granted Patent: US 9,483,383 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3644 by instrumenting at runtime
• CPC Class Codes: H04L 43/0805 by checking availability

Abstract

Methods, systems, and articles of manufacture for injecting faults at select execution points of distributed applications are provided herein. A method includes monitoring a run-time state of each of multiple components of a distributed application to determine one or more sequence of events that triggers a fault injection point at one of the multiple components; defining a fault injection scenario in a specification based on said monitoring, wherein said fault injection scenario comprises a description of one or more sequence of events during which an intended fault is to be injected to a target component of the multiple components at one selected event; and executing the fault injection defined in the specification to perform injection of the intended fault during run-time of the distributed application.

Claims

1. A method comprising:

monitoring a run-time state of each of multiple components of a distributed application to determine one or more sequence of events that triggers a fault injection point at one of the multiple components;

defining a fault injection scenario in a specification based on said monitoring, wherein said fault injection scenario comprises a description of one or more sequence of events during which an intended fault is to be injected to a target component of the multiple components at one selected event; and

executing the fault injection defined in the specification to perform injection of the intended fault during run-time of the distributed application;

wherein at least one of said monitoring, said defining, and said executing is carried out by a computing device.

2. The method of claim 1, wherein said one or more sequence of events comprises a sequence of log events.

3. The method of claim 1, wherein said monitoring comprises monitoring the run-time state of each of the multiple components based on collected execution traces of the distributed application.

4. The method of claim 1, wherein said monitoring comprises monitoring the run-time state of each of the multiple components of the distributed application under multiple workloads.

5. The method of claim 1, wherein said defining comprises defining a fault injection scenario based on one or more event dependencies discovered from said monitoring.

6. The method of claim 1, wherein said intended fault comprises at least one of an application crash, network congestion, a lack of memory and/or disk space, a memory, disk or network corruption and a configuration error.

7. The method of claim 1, wherein said target component comprises at least one of a given function, a given process, a virtual machine, and a physical machine.

8. The method of claim 1, comprising:

analyzing results from said injection of the intended fault.

9. The method of claim 8, wherein said analyzing comprises collecting multiple logs derived from said injection of the intended fault.

10. The method of claim 9, comprising:

displaying said multiple logs to a user.

11. The method of claim 8, wherein said analyzing comprises generating statistics pertaining to said injection of the intended fault.

12. The method of claim 1, comprising:

executing injection of multiple intended faults concurrently.

13. An article of manufacture comprising a computer readable storage medium having computer readable instructions tangibly embodied thereon which, when implemented, cause a computer to carry out a plurality of method steps comprising:

monitoring a run-time state of each of multiple components of a distributed application to determine one or more sequence of events that triggers a fault injection point at one of the multiple components;

defining a fault injection scenario in a specification based on said monitoring, wherein said fault injection scenario comprises a description of one or more sequence of events during which an intended fault is to be injected to a target component of the multiple components at one selected event; and

executing the fault injection defined in the specification to perform injection of the intended fault during run-time of the distributed application.

14. The article of manufacture of claim 13, wherein said monitoring comprises monitoring the run-time state of each of the multiple components based on collected execution traces of the distributed application.

15. The article of manufacture of claim 13, wherein said monitoring comprises monitoring the run-time state of each of the multiple components of the distributed application under multiple workloads.

16. The article of manufacture of claim 13, wherein said defining comprises defining a fault injection scenario based on one or more event dependencies discovered from said monitoring.

17. The article of manufacture of claim 13, wherein the method steps comprise:

executing injection of multiple intended faults concurrently.

18. A system comprising:

a memory; and

at least one processor coupled to the memory and configured for;

monitoring a run-time state of each of multiple components of a distributed application to determine one or more sequence of events that triggers a fault injection point at one of the multiple components;

defining a fault injection scenario in a specification based on said monitoring, wherein said fault injection scenario comprises a description of one or more sequence of events during which an intended fault is to be injected to a target component of the multiple components at one selected event; and

executing the fault injection defined in the specification to perform injection of the intended fault during run-time of the distributed application.

19. The system of claim 18, wherein said defining comprises defining a fault injection scenario based on one or more event dependencies discovered from said monitoring.

20. The system of claim 18, wherein the at least one processor is further configured for:

executing injection of multiple intended faults concurrently.