Repair-policy refinement in distributed systems

• US 8,504,874 B2
• Filed: 09/21/2010
• Issued: 08/06/2013
• Est. Priority Date: 09/21/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Goldszmidt, Moises, Budiu, Mihai, Zhang, Yue, Pechuk, Michael
• Primary Examiner(s): Manoskey, Joseph D
• Application Number:
  US12/886,566
• Publication Number:
  US 20120072769A1
• Time in Patent Office: 1,050 Days
• Field of Search: 714/2, 714/25, 714/26, 714/30, 714/33, 714/37, 714/39, 714/47.1
• US Class Current: 714/25
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16

Abstract

In a distributed system a plurality of devices (including computing units, storage and communication units) are monitored by an automated repair service that uses sensors and performs one or more repair actions on computing devices that are found to fail according to repair policies. The repair actions include automated repair actions and non-automated repair actions. The health of the computing devices is recorded in the form of states along with the repair actions that were performed on the computing devices and the times at which the repair actions were performed, and events generated by both sensors and the devices themselves. After some period of the time, the history of states of each device, the events, and the repair actions performed on the computing devices are analyzed to determine the effectiveness of the repair actions. A statistical analysis is performed based on the cost of each repair action and the determined effectiveness of each repair action, and one or more of the policies may be adjusted, as well as determining from the signals and events from the sensors whether the sensors themselves require adjustment.

Claims

1. A method comprising:

monitoring a plurality of devices using a plurality of sensors, wherein the plurality of sensors generate a plurality of signals and events for each device based on the monitoring;

determining a state of each of the plurality of devices based on the generated plurality of signals and events by a repair service;

performing one or more repair actions on one or more of the plurality of devices according to the determined states by the repair service;

recording, for each device, the state of the device, a time the device entered the state, any repair actions performed on the device by the repair service, and the generated plurality of signals and events for the device;

determining an effectiveness of the one or more repair actions using the recorded states, the performed repair actions, and the time the devices entered each state; and

determining the effectiveness of the one or more sensors using the determined effectiveness of the one or more repair actions and the generated plurality of signals and events of the one or more sensors using a formal language to represent a sequence of the signals and events.

2. The method of claim 1, wherein determining the effectiveness of the one or more repair actions comprises determining the effectiveness of each repair action based on an amount of time a device that the repair action is performed on remains in a healthy state.

3. The method of claim 2, wherein determining the effectiveness of each repair action uses statistical survival analysis.

4. The method of claim 1, wherein the formal language comprises a regular expression and further comprising extracting subsegments of interest from the sequence of signals and events for the analysis of the effectiveness of the one or more sensors.

5. The method of claim 1, further comprising adjusting one or more of the sensors according to the determined effectiveness of the one or more sensors.

6. The method of claim 1, further comprising determining the effectiveness of one or more policies using the determined effectiveness of the one or more repair actions and the generated plurality of signals and events using the formal language to represent a sequence of the signals and events.

7. The method of claim 6, wherein determining the effectiveness of the one or more policies comprises determining the effectiveness of the one or more policies using one of machine learning and statistical techniques, wherein the statistical techniques comprise a pattern classifier.

8. The method of claim 1, wherein the devices each comprise at least one of a computing unit, a storage unit, a communication unit, a process, or a software application.

9. The method of claim 1, wherein the state comprises a healthy state or a failed state, and the repair actions include at least one of restarting the device, reinstalling a software application associated with the device, or reimaging the device.

10. A method comprising:

receiving a sequence of states for each of a plurality of devices by a repair service, wherein the sequence of states for a device comprises an indicator of each state the device entered into and an associated time, an indicator of each repair action performed on the device, and the events and signals generated by one or more sensors or devices that led to the repair action;

determining the effectiveness of the repair actions based on the sequence of states and an amount of time each state that had a repair action performed remained in a healthy state by the repair service;

receiving one or more policies by the repair service, wherein the policies indicate repair actions to perform on the devices based on the states of the devices and the events and signals generated by the one or more sensors; and

determining the effectiveness of the one or more policies by the repair service based on the determined effectiveness of the repair actions and the events and signals generated by one or more sensors that led to the repair actions from the sequence of states using a formal language to represent a sequence of the signals and events.

11. The method of claim 10, wherein determining the effectiveness of the repair actions comprises determining the effectiveness of the repair actions using statistical survival analysis.

12. The method of claim 10, further comprising refining the one or more policies based on the effectiveness of the one or more policies.

13. The method of claim 10, wherein the devices each comprise at least one of a computing unit, a communications unit, a network unit, a process, or a software application.

14. A system comprising:

a plurality of devices;

a plurality of sensors monitoring the plurality of devices and generating a plurality of signals and events for each device based on the monitoring; and

a repair service adapted to;

determine a state of each of the devices based on the plurality of signals and events;

perform one or more repair actions on one or more of the devices according to the determined states;

record, for each device, the state of the device, a time the device entered the state, any repair actions performed on the device by the repair service, and the generated plurality of signals and events for the device;

determine the effectiveness of one or more repair actions using the recorded states, the performed repair actions, and the time the devices entered each state; and

determine the effectiveness of the one or more sensors using the determined effectiveness of the one or more repair actions and the generated plurality of signals and events of the plurality of sensors using a formal language to represent a sequence of the plurality of signals and events.

15. The system of claim 14, wherein the repair service is adapted to determine the effectiveness of each repair action based on an amount of time a device that the repair action is performed on remains in a healthy state.

16. The system of claim 14, wherein the repair service is further adapted to adjust one or more of the plurality of sensors according to the determined effectiveness of the plurality of sensors.

17. The system of claim 14, wherein the devices each comprise at least one of a computing unit, a storage unit, a communication unit, a process, or a software application.

18. The system of claim 14, wherein the state comprises a healthy state or a failed state, and the repair actions include at least one of restarting the device, reinstalling a software application associated with the device, or reimaging the device.

19. The system of claim 14, wherein determining the effectiveness of each repair action uses statistical survival analysis.

20. The system of claim 14, wherein the repair service is further adapted to determine the effectiveness of one or more policies using the determined effectiveness of the one or more repair actions and the generated plurality of signals and events using the formal language to represent a sequence of the signals and events.