Systems and methods for prioritizing error notification

• US 7,689,873 B1
• Filed: 09/19/2005
• Issued: 03/30/2010
• Est. Priority Date: 09/19/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Bennett, Victor
• Primary Examiner(s): Baderman; Scott T.
• Assistant Examiner(s): Truong; Loan
• Application Number:
  US11/230,713
• Time in Patent Office: 1,653 Days
• Field of Search: 714/48, 705/5, 705/415
• US Class Current: 714/48
• CPC Class Codes: G06F 11/0748 in a remote unit communicating with a single-box computer node experiencing an error/fault remote testing G06F11/2294
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06Q 10/02 Reservations, e.g. for tickets, services or events
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling

Abstract

Errors occurring in computing clusters and other computing systems can impact system performance. Each error has an error type and each error type has a base cost estimating importance of correcting the error. Each error type also has a confidence indicating the level of agreement between those who fix the errors and those who assigned the base cost. An error type'"'"'s actual cost is produced using the base cost and confidence. An error cascade map contains estimates that one error will cause another. An error type that causes other error types has a cascade cost. Upon detecting an error type, a repair order can be generated, depending on the cost involved. Repairs are then performed. Feedback mechanisms and correlations can be used to update the confidences and the error cascade map.

Claims

1. A method for prioritizing repairs of a computing system of networked computers comprising:

associating a base cost and a confidence with each one of at least one error type that can impact the performance of the computing system such that an actual cost for each one of the at least one error type can be produced with at least one computer using the associated base cost and the associated confidence, wherein the associated confidence is a value indicating the degree to which the base cost associated with a respective error type is accurate;

detecting an occurrence of any one of the at least one error type with at least one computer, creating a detected error associated with the error type that occurred, and using the detected error to update a detected error collection wherein the detected error collection comprises one or more detected errors;

using the actual cost of the error type associated with each detected error in the detected error collection to create one or more repair orders with at least one computer; and

detecting one or more repairs with at least one computer and updating the detected error collection accordingly.

2. The method of claim 1 further comprising updating the confidence associated with the error type associated with the one or more repair orders based on a time to repair metric.

3. The method of claim 1 wherein the actual cost is the base cost multiplied by the confidence.

4. The method of claim 2 further comprising detecting an absence of a previously detected error, inferring that the previously detected error has been repaired and thereby detecting at least one of the one or more repairs by inference.

5. The method of claim 1 wherein the detected error collection is an ordered queue of the detected errors and further comprising ordering the detected errors in the detected error collection based on the actual cost of the detected errors.

21. The method of claim 1, wherein the computing system is a computer cluster.

6. A method for prioritizing repairs of a computing system of networked computers comprising:

associating a base cost and a confidence with each one of at least one error type that can impact the performance of the computing system such that an actual cost for each one of the at least one error type can be produced with at least one computer using the associated base cost and the confidence, wherein the associated confidence is a value indicating the degree to which the base cost associated with a respective error type is accurate;

producing with at least one computer an error cascade map associating at least one causing error with at least one resulting error to produce at least one cascade association and associating a cascade probability with each one of the at least one cascade association wherein the causing error is one of the at least one error type, the resulting error is one of the at least one error type, and the cascade probability indicates the likelihood that the causing error will cause the resulting error to occur and wherein a cascade cost for each one of the at least one error type can be produced with at least one computer from the error cascade map and the actual cost of each of the at least one error type;

detecting an occurrence of any one of the at least one error type with at least one computer, creating a detected error associated with the error type that occurred, and using the detected error to update a detected error collection wherein the detected error collection comprises one or more detected errors;

using the cascade cost of the error type associated with each detected error in the detected error collection to create one or more repair orders with at least one computer; and

detecting one or more repairs with at least one computer and updating the detected error collection accordingly.

7. The method of claim 6 wherein the cascade cost for each one of the at least one error type is produced by multiplying the cascade probability from the error cascade map by the actual cost of each of the at least one error type.

8. The method of claim 6 wherein the cascade probability is a function of elapsed time from the occurrence of the causing error.

9. The method of claim 6 further comprising updating the confidence associated with the error type associated with the one or more repair orders based on the elapsed time between repair orders and repair performance.

10. The method of claim 6, wherein the cascade cost of the error type is calculated by combining the actual costs of all of the cascaded error types.

11. The method of claim 9 further comprising detecting an absence of a previously detected error, inferring that the previously detected error has been repaired and thereby detecting one or more of the one or more repairs by inference.

12. The method of claim 6 wherein the detected error collection is an ordered queue and further comprising ordering the detected error collection based on the cascade cost of the detected errors.

22. The method of claim 6, wherein a cascade probability is assigned to each one of two or more cascade associations.

13. A system for prioritizing repairs of a computing system comprising:

at least two computers in the computing system wherein each of the at least two computers can communicate with any of the at least two computers via a communications network;

at least one error type wherein an error type identifies one of at least one error that impacts the performance of the computing system and an error price association wherein each of the at least one error type is associated with a base cost and a confidence such that an actual cost can be produced for each one of the at least one error type, wherein the associated confidence is a value indicating the degree to which the base cost associated with a respective error type is accurate;

a detected error collection comprising one or more detected errors wherein each of the at least one detected errors has an error type;

at least one error detection module that detects an occurrence of at least one error and updates the detected error collection based on the at least one error;

a repair assignment module that examines the detected error collection and creates one or more repair orders that cause one or more repairs wherein each one of the one or more repairs is associated with one or more of the one or more repair orders; and

a repair detection module that detects an occurrence of at least one repair and updates the detected error collection.

14. The system of claim 13 further comprising at least one actual cost wherein each one of the at least one actual cost is associated with one of the at least one error type, wherein each one of the at least one actual cost is produced from the error price association, wherein each one of the one or more detected errors has an actual cost, and wherein the repair assignment module prioritizes the one or more repair orders based on the actual cost of each one of the one or more detected errors.

15. The system of claim 13 further comprising:

an error cascade map comprising one or more cascade elements comprising a causing error, a resulting error, and a cascade probability wherein the causing error is one of the at least one error type, the resulting error is one of the at least one error type, and the cascade probability is the probability that the resulting error will follow the causing error; and

at least one cascade cost wherein each one of the at least one cascade cost is associated with one of the at least one error type, wherein each one of the at least one cascade cost is produced from the error price association and the error cascade map, wherein each one of the one or more detected errors has a cascade cost, and wherein the repair assignment module prioritizes the one or more repair orders based on the cascade cost of each one of the one or more detected errors.

16. The system of claim 15 further comprising error correlations wherein the error correlations are produced from the detected error collection and wherein the error correlations are used to update the error cascade map.

17. The system of claim 15 wherein the cascade probability is a function of elapsed time from the occurrence of the causing error.

18. The system of claim 13 further comprising a confidence module that tracks the one or more repair orders and the one or more repairs and updates the error price association.

19. A system for prioritizing repairs of a computing system comprising:

a means of computing comprising at least two computers that can communicate over a communications network;

a means of detecting at least one error in the means of computing and assigning a cost to each one of the at least one error based upon a base cost and a confidence associated with each at least one error, wherein the associated confidence is a value indicating the degree to which the base cost associated with a respective error is accurate;

a means of tracking the at least one error;

a means of prioritizing and assigning at least one repair wherein each one of the at least one repair removes at least one of the at least one error; and

a means of detecting the completion of each one of the at least one repair.

20. The system of claim 19 further comprising a means of determining a probability that at least one cascade error will result from each one of the at least one error and a means of assigning a cascade cost to each one of the at least one error based upon the probability.