Injecting faults at select execution points of distributed applications

• US 9,483,383 B2
• Filed: 12/05/2013
• Issued: 11/01/2016
• Est. Priority Date: 12/05/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Baset, Salman A., Pham, Cuong M., Ramasamy, Harigovind V., Singh, Manas, Tak, Byung Chul, Tang, Chunqiang, Wang, Long
• Primary Examiner(s): Decady, Albert
• Assistant Examiner(s): CONTINO, PAUL F
• Application Number:
  US14/097,713
• Publication Number:
  US 20150161025A1
• Time in Patent Office: 1,062 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3644 by instrumenting at runtime
• CPC Class Codes: H04L 43/0805 by checking availability

Abstract

Methods, systems, and articles of manufacture for injecting faults at select execution points of distributed applications are provided herein. A method includes monitoring a run-time state of each of multiple components of a distributed application to determine one or more sequence of events that triggers a fault injection point at one of the multiple components; defining a fault injection scenario in a specification based on said monitoring, wherein said fault injection scenario comprises a description of one or more sequence of events during which an intended fault is to be injected to a target component of the multiple components at one selected event; and executing the fault injection defined in the specification to perform injection of the intended fault during run-time of the distributed application.

Claims

1. A method comprising:

monitoring a run-time state of each of multiple distributed components of a distributed application that comprises a variable number of multiple states, based on collected execution traces of the distributed application, to determine one or more sequence of events that triggers a fault injection point at each of the multiple components, wherein said one or more sequence of events comprises a sequence of log events;

defining a scenario of multiple fault injections in a specification based on said monitoring, wherein said scenario comprises a description of one or more sequence of events during which each of the multiple faults is to be injected across the multiple distributed components at one selected event, and wherein said defining comprises defining the scenario based on one or more event dependencies discovered from said monitoring; and

executing the multiple fault injections defined in the specification concurrently across the multiple distributed components of the distributed application during run-time of the distributed application;

wherein said monitoring, said defining, and said executing are carried out by at least one computing device.

2. The method of claim 1, wherein said monitoring comprises monitoring the run-time state of each of the multiple distributed components of the distributed application under multiple workloads.

3. The method of claim 1, wherein each of the multiple faults comprises at least one of an application crash, network congestion, a lack of memory and/or disk space, a memory, disk or network corruption and a configuration error.

4. The method of claim 1, wherein each of the multiple distributed components comprises at least one of a given function, a given process, a virtual machine, and a physical machine.

5. The method of claim 1, comprising:

analyzing results from said multiple fault injections.

6. The method of claim 5, wherein said analyzing comprises collecting multiple logs derived from said multiple fault injections.

7. The method of claim 6, comprising:

displaying said multiple logs to a user.

8. The method of claim 5, wherein said analyzing comprises generating statistics pertaining to said multiple fault injections.

9. An article of manufacture comprising a non-transitory computer readable storage medium having computer readable instructions tangibly embodied thereon which, when implemented, cause a computer to carry out a plurality of method steps comprising:

monitoring a run-time state of each of multiple distributed components of a distributed application that comprises a variable number of multiple states, based on collected execution traces of the distributed application, to determine one or more sequence of events that triggers a fault injection point at each of the multiple components, wherein said one or more sequence of events comprises a sequence of log events;

defining a scenario of multiple fault injections in a specification based on said monitoring, wherein said scenario comprises a description of one or more sequence of events during which each of the multiple faults is to be injected across the multiple distributed components at one selected event, and wherein said defining comprises defining the scenario based on one or more event dependencies discovered from said monitoring; and

executing the multiple fault injections defined in the specification concurrently across the multiple distributed components of the distributed application during run-time of the distributed application.

10. The article of manufacture of claim 9, wherein said monitoring comprises monitoring the run-time state of each of the multiple distributed components of the distributed application under multiple workloads.

11. A system comprising:

a memory; and

at least one processor coupled to the memory and configured for;

monitoring a run-time state of each of multiple distributed components of a distributed application that comprises a variable number of multiple states, based on collected execution traces of the distributed application, to determine one or more sequence of events that triggers a fault injection point at each of the multiple components, wherein said one or more sequence of events comprises a sequence of log events;

defining a scenario of multiple fault injections in a specification based on said monitoring, wherein said scenario comprises a description of one or more sequence of events during which each of the multiple faults is to be injected across the multiple distributed components at one selected event, and wherein said defining comprises defining the scenario based on one or more event dependencies discovered from said monitoring; and

executing the multiple fault injections defined in the specification concurrently across the multiple distributed components of the distributed application during run-time of the distributed application.