Invariants-based learning method and system for failure diagnosis in large scale computing systems

• US 8,185,781 B2
• Filed: 04/05/2010
• Issued: 05/22/2012
• Est. Priority Date: 04/09/2009
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Laboratories America Inc (NEC Corporation)
• Inventors: Chen, Haifeng, Jiang, Guofei, Yoshihira, Kenji, Saxena, Akhilesh
• Primary Examiner(s): Baderman, Scott
• Assistant Examiner(s): Patel, Kamini
• Application Number:
  US12/754,572
• Publication Number:
  US 20100262858A1
• Time in Patent Office: 778 Days
• Field of Search: 714/26
• US Class Current: 714/26
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/2257 using expert systems
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

A method system for diagnosing a detected failure in a computer system, compares a failure signature of the detected failure to an archived failure signature contained in a database to determine if the archived failure signature matches the failure signature of the detected failure. If the archived failure signature matches the failure signature of the detected failure, an archived solution is applied to the computer system that resolves the detected failure, the archived solution corresponding to a solution used to resolve a previously detected computer system failure corresponding to the archived failure signature in the database that matches the detected failure.

Claims

1. A method for diagnosing a detected failure in a computer system, the method comprising:

comparing, in a computer process, a failure signature of the detected failure to an archived failure signature contained in a database to determine if the archived failure signature matches the failure signature of the detected failure;

if the archived failure signature matches the failure signature of the detected failure, applying, in a computer process, an archived solution to the computer system that resolves the detected failure, the archived solution corresponding to a solution used to resolve a previously detected computer system failure corresponding to the archived failure signature in the database that matches the detected failure;

wherein the archived failure signature is based on a set of broken computer system invariants, the set of broken computer system invariants corresponding to the previously detected computer system failure;

further comprising constructing the database in a computer process prior to comparing the failure signature of the detected failure to the archived failure signature;

wherein the constructing the database includes extracting invariants from the computer system;

wherein the extracting the invariants includes;

modeling invariants of the computer system;

evaluating each of the invariants to determine whether it is broken;

counting the broken invariants to determine whether the number of the broken invariants meets a predetermined threshold number;

if the number of the broken invariants meets the predetermined threshold number deeming this result the previously detected computer system failure; and

combining the broken invariants into the set of broken invariants forming the archived failure signature of the previously detected computer system failure.

2. The method of claim 1, wherein if the archived failure signature does not match the failure signature of the detected failure, further comprising resolving the detected failure using an alternative solution implemented manually or in a computer process.

3. The method of claim 2, further comprising:

storing the alternative solution in the database or another database as another archived solution; and

storing the failure signature of the detected failure in the database as another archived failure signature.

4. The method of claim 3, wherein the another archived solution is indexed to the another archived failure signature.

5. The method of claim 1 wherein the archived solution is contained in the database or another database.

6. The method of claim 5, wherein the archived solution is indexed to the archived failure signature.

7. The method of claim 1, wherein the failure signature of the detected failure is determined by extracting invariants from the computer system prior to the comparing of the failure signature of the detected failure to the archived failure signature contained in a database.

8. The method of claim 7, wherein the extracting the invariants includes:

modeling invariants of the computer system;

evaluating each of the invariants to determine whether it is broken;

counting the broken invariants to determine whether the number of the broken invariants meets a predetermined threshold number;

if the number of the broken invariants meets the predetermined threshold number deeming this result the detected failure in the a computer system; and

combining the broken invariants into a set of broken computer system invariants, the set of broken invariants forming the failure signature of the detected failure in the computer system.

9. A system for diagnosing a detected failure in a computer system, the system comprising:

a database containing an archived failure signature; and

a processor associated with the database, the processor executing instructions for;

comparing a failure signature of the detected failure to the archived failure signature contained in the database to determine if the archived failure signature matches the failure signature of the detected failure;

if the archived failure signature matches the failure signature of the detected failure, applying an archived solution to the computer system that

resolves the detected failure, the archived solution corresponding to a solution used to resolve a previously detected computer system failure corresponding to the archived failure signature in the database that matches the detected failure;

wherein the processor executes further instructions for extracting invariants from the computer system prior to the comparing of the failure signature of the detected failure to the archived failure signature contained in the database, in order to determine the failure signature of the detected failure

wherein the extracting the invariants includes;

modeling invariants of the computer system;

evaluating each of the invariants to determine whether it is broken;

counting the broken invariants to determine whether the number of the broken invariants meets a predetermined threshold number;

if the number of the broken invariants meets the predetermined threshold number deeming this result the detected failure in the a computer system; and

combining the broken invariants into a set of broken computer system invariants, the set of broken invariants forming the failure signature of the detected failure in the computer system.

10. The system of claim 9, wherein the archived failure signature is based on a set of broken computer system invariants, the set of broken computer system invariants corresponding to the previously detected computer system failure.

11. The system of claim 9, wherein if the archived failure signature does not match the failure signature of the detected failure, the processor executing further instructions for resolving the detected failure using an alternative solution implemented manually or in a computer process.

12. The system of claim 11, wherein the processor executes further instructions for:

storing the alternative solution in the database or another database as another archived solution; and

storing the failure signature of the detected failure in the database as another archived failure signature.

13. The system of claim 12, wherein the another archived solution is indexed to the another archived failure signature.

14. The system of claim 9, wherein the archived solution is contained in the database or another database.

15. The system of claim 14, wherein the archived solution is indexed to the archived failure signature.

16. A method for diagnosing a detected failure in a computer system, the method comprising:

comparing, in a computer process, a failure signature of the detected failure to an archived failure signature contained in a database to determine if the archived failure signature matches the failure signature of the detected failure;

if the archived failure signature matches the failure signature of the detected failure, applying, in a computer process, an archived solution to the computer system that resolves the detected failure, the archived solution corresponding to a solution used to resolve a previously detected computer system failure corresponding to the archived failure signature in the database that matches the detected failure;

wherein the failure signature of the detected failure is determined by extracting invariants from the computer system prior to the comparing of the failure signature of the detected failure to the archived failure signature contained in a database;

wherein the extracting the invariants includes;

modeling invariants of the computer system;

evaluating each of the invariants to determine whether it is broken;

counting the broken invariants to determine whether the number of the broken invariants meets a predetermined threshold number;

if the number of the broken invariants meets the predetermined threshold number deeming this result the detected failure in the a computer system; and

combining the broken invariants into a set of broken computer system invariants, the set of broken invariants forming the failure signature of the detected failure in the computer system.