Proactive failure handling in data processing systems

• US 9,594,620 B2
• Filed: 04/01/2016
• Issued: 03/14/2017
• Est. Priority Date: 04/04/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Xia, Hao, Pfleiger, Todd F., Benvenuto, Mark C., Kalhan, Ajay
• Primary Examiner(s): Bonzo, Bryce
• Assistant Examiner(s): Arcos, Jeison C
• Application Number:
  US15/088,377
• Publication Number:
  US 20160217025A1
• Time in Patent Office: 347 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/004 Error avoidance G06F11/07 and subgroups take precedence
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/0769 Readable error formats, e.g. cross-platform generic formats, human understandable formats
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/2257 using expert systems
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 11/3065 Monitoring arrangements determined by the means or processing involved in reporting the monitored data error or fault reporting or logging G06F11/0766
• CPC Class Codes: G06F 11/324 Display of status information
• CPC Class Codes: G06F 2201/865 Monitoring of software
• CPC Class Codes: H04L 43/065 related to network devices

Abstract

Embodiments are directed to predicting the health of a computer node using health report data and to proactively handling failures in computer network nodes. In an embodiment, a computer system monitors various health indicators for multiple nodes in a computer network. The computer system accesses stored health indicators that provide a health history for the computer network nodes. The computer system then generates a health status based on the monitored health factors and the health history. The generated health status indicates the likelihood that the node will be healthy within a specified future time period. The computer system then leverages the generated health status to handle current or predicted failures. The computer system also presents the generated health status to a user or other entity.

Claims

1. A computer system for predicting the health of a plurality of data processing systems by using health report data, the computer system comprising one or more processors executing computer executable instructions which cause the computer system to perform the following:

monitors one or more health indicators for a plurality of data processing systems;

accesses one or more stored health indicators that provide a health history for one or more of the monitored plurality of data processing systems;

based on both the monitored health indicators and the stored health history, predicts a health status, wherein the predicted health status indicates for at least one of the monitored plurality of data processing systems a probability that the at least one monitored data processing system will be healthy or unhealthy in the future; and

presents the predicted health status to a specified entity.

2. The computer system of claim 1, wherein the computer system further performs the following:

makes a determination that the probability that the at least one monitored data processing system will be healthy is below a threshold level; and

transfers one or more portions of data stored on the at least one monitored data processing system to one or more other data processing systems in the database system.

3. The computer system of claim 2, wherein the computer system further performs the following:

prevents the at least one monitored data processing system from storing new data.

4. The computer system of claim 2, wherein making the determination that the probability that the at least one monitored data processing system will be healthy is below a threshold level comprises determining that the at least one monitored data processing system is in a critical state.

5. The computer system of claim 2, wherein making the determination that the probability that the at least one monitored data processing system will be healthy is below a threshold level comprises determining that the at least one monitored data processing system has experienced one or more failures within a specified time period.

6. The computer system of claim 5, wherein the computer system further performs the following:

assigns an error level to each of the failures.

7. The computer system of claim 6, wherein the computer system further performs the following:

determines that a threshold number of the failures are beyond a specified error level, such that the at least one monitored data processing system is blacklisted.

8. The computer system of claim 7, wherein monitored data processing systems that have a threshold number of the failures beyond a specified error level are blacklisted, regardless of a monitored data processing system'"'"'s health history.

9. The computer system of claim 7, wherein blacklisted data processing systems are put on probation for a specified amount of time to determine whether errors occur during probation.

10. The computer system of claim 9, wherein the computer system further performs the following:

upon determining for a monitored data processing system that was blacklisted that the probationary period is complete and that no further errors have occurred, allowing the monitored data processing system that was blacklisted to continue storing new data and removing the monitored data processing system from the blacklist.

11. The computer system of claim 9, wherein the computer system further performs the following:

upon determining for a monitored data processing system that was blacklisted that the probationary period is complete and that one or more further errors have occurred, preventing the monitored data processing system that was blacklisted from storing new data.

12. The computer system of claim 11, wherein the computer system further performs the following:

relocates data portions that are hosted on the monitored data processing system that is prevented from storing new data.

24. The computer system of claim 1, wherein the predicted health status indicates for at least one of the monitored plurality of data processing systems a probability that the at least one monitored data processing system will be healthy in the future.

25. The computer system of claim 1, wherein the predicted health status indicates for at least one of the monitored plurality of data processing systems a probability that the at least one monitored data processing system will be healthy within a specified future time period.

26. The computer system of claim 1, wherein the predicted health status indicates for at least one of the monitored plurality of data processing systems a probability that the at least one monitored data processing system will be unhealthy in the future.

13. A computer-implemented method for proactively handling failures in a database system comprising a plurality of data processing systems each of which corresponds to a node of the database system, the computer-implemented method being performed by one or more processors executing computer executable instructions for the computer-implemented method, and the computer-implemented method comprising:

monitoring one or more health indicators for a plurality of data processing systems each of which corresponds to a node of a database system;

accessing at the first data processing system one or more stored health indicators that provide a health history for the one or more of the monitored data processing systems;

predicting a health status based on the monitored factors indicators and the health history, wherein the predicted health status indicates the probability that the one or more monitored data processing systems will be healthy or unhealthy in the future;

determining, for at least one of the monitored data processing systems, that a threshold number of failures have occurred that are beyond a specified error level;

based on the determination, blacklisting the monitored data processing system for which the determination was made;

transferring one or more portions of data stored from the monitored data processing system that is blacklisted to one or more of other data processing systems; and

preventing the data processing system that was blacklisted from storing new data.

14. The computer-implemented method of claim 13, wherein the data processing system that is blacklisted is categorized as up and blacklisted, such that the blacklisted data processing system remains used for storing data while the data is transferred to other data processing systems, and no new data is stored on the data processing system that is up and blacklisted.

15. The computer-implemented method of claim 13, wherein the data processing system that is blacklisted is categorized as down and blacklisted, such that the blacklisted data processing system is no longer used for storing data, data is transferred from the blacklisted data processing system to other data processing systems, and no new data is stored on the data processing system this is down and blacklisted.

16. The computer-implemented method of claim 15, wherein the data is transferred without waiting for a probationary period.

27. The computer-implemented method of claim 13, wherein the predicted health status indicates for at least one of the monitored plurality of data processing systems a probability that the at least one monitored data processing system will be healthy in the future.

28. The computer-implemented method of claim 13, wherein the predicted health status indicates for at least one of the monitored plurality of data processing systems a probability that the at least one monitored data processing system will be healthy within a specified future time period.

29. The computer-implemented method of claim 13, wherein the predicted health status indicates for at least one of the monitored plurality of data processing systems a probability that the at least one monitored data processing system will be unhealthy in the future.

17. A database system comprising:

a plurality of data processing systems;

one or more computer-readable storage hardware media, excluding transmission media, and having stored thereon computer-executable instructions that, when executed by one or more processors, cause the database system to be configured with an architecture that proactively handles failures in the plurality of data processing systems by using health report data, and wherein the architecture is configured to perform the following;

monitor one or more health indicators for a plurality of data processing systems of the database system;

access one or more stored health indicators that provide a health history for one or more of the monitored plurality of data processing systems;

based on both the monitored health indicators and the stored health history, predict a health status, wherein the predicted health status indicates for at least one of the monitored plurality of data processing systems a probability that the at least one monitored data processing system will be healthy or unhealthy in the future; and

present the predicted health status to a specified entity.

18. The database system of claim 17, wherein the architecture is further configured to blacklist at least one of the monitored data processing systems upon determining that a threshold number of failures have occurred that are beyond a specified error level.

19. The database system of claim 18, wherein the architecture is further configured to transfer one or more portions of data stored on the monitored data processing system that is blacklisted to one or more other data processing systems.

20. The database system of claim 19, wherein the data processing system that is blacklisted is categorized as up and blacklisted, such that the categorized data processing system remains used for storing data while the data is transferred to other data processing systems, and no new data is stored on the data processing system that is up and blacklisted.

21. The database system of claim 18, wherein the predicted health status indicates for at least one of the monitored plurality of data processing systems a probability that the at least one monitored data processing system will be healthy in the future.

22. The database system of claim 18, wherein the predicted health status indicates for at least one of the monitored plurality of data processing systems a probability that the at least one monitored data processing system will be healthy within a specified future time period.

23. The database system of claim 18, wherein the predicted health status indicates for at least one of the monitored plurality of data processing systems a probability that the at least one monitored data processing system will be unhealthy in the future.