Management of a fault condition in a computing system

• US 9,804,913 B2
• Filed: 09/25/2015
• Issued: 10/31/2017
• Est. Priority Date: 09/25/2015
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Dasari, Annapurna
• Primary Examiner(s): LE, DIEU MINH T
• Application Number:
  US14/865,872
• Publication Number:
  US 20170091007A1
• Time in Patent Office: 767 Days
• Field of Search: 714 57, 714 2, 714 16, 714 25, 714 31, 714 48
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0721 within a central processing unit [CPU]
• CPC Class Codes: G06F 11/0748 in a remote unit communicating with a single-box computer node experiencing an error/fault remote testing G06F11/2294
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0772 Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50

Abstract

Systems, apparatuses, and/or methods may manage a fault condition in a computer system. An apparatus may dynamically publish a message over a publisher-subscriber system and dynamically subscribe to a message over the publisher-subscriber system, wherein at least one message may be used to address a fault condition in the computing system. The apparatus may predict a fault condition in a high performance computing (HPC) system, communicate fault information to a user, monitor health of the HPC system, respond to the fault condition in the HPC system, recover from the fault condition in the HPC system, maintain a rule for a fault management component, and/or communicate the fault information over the publisher-subscriber system in real-time. Messages may also be aggregated to minimize fault information traffic. The publisher-subscriber system may facilitate dynamic and/or real-time coordinated, integrated (e.g., system-wide), and/or scalable fault management.

Claims

1. A system comprising:

a sensor to collect data in a high performance computing (HPC) system; and

a plurality of message determiners, wherein each of the message determiners are to dynamically publish a message over a publisher-subscriber system and are to dynamically subscribe to a message over the publisher-subscriber system, wherein at least one message is to correspond to the data from the sensor and is to be used to coordinate actions to manage a fault condition in the HPC system, and wherein a message determiner is to include;

a data determiner to dynamically determine a need for data of interest to the message determiner; and

a message generator to generate a request message to one or more of request subscription to a message that is to include the data of interest or prompt subscription to the request message to cause the data of interest to be published over the publication-subscription system.

2. The system of claim 1, further including:

a fault predictor to predict a fault condition in the HPC system;

a fault reporter to communicate fault information to a user;

a distributed fault monitor to monitor health of the HPC system;

a distributed fault processor to one or more of respond to the fault condition in the HPC system or recover from the fault condition in the HPC system;

a fault policy maintainer to maintain a rule for a fault management component; and

a system interface to communicate the fault information over the publisher-subscriber system in real-time to one or more interested message determiners.

3. The system of claim 1, wherein at least one of the message determiners is to receive information from a broker, and wherein the information is to include one or more of a data message including data of interest or subscriber contact information.

4. The system of claim 1, wherein at least one of the message determiners is to provide a message to an aggregator at a higher level of a fault management system hierarchy, and wherein the aggregator is to collect messages from a lower level of the fault management system hierarchy and is to filter the messages to provide a sub-set at least of all available fault information to a higher level of the fault management system hierarchy.

5. The system of claim 1, wherein a message determiner is to include:

a data determiner to dynamically determine an ability to provide data of interest to at least one other message determiner;

a subscription approver to determine whether to permit subscription to provide a data message including the data of interest;

a capability adjuster to determine whether to adjust a management capability of a fault management component associated with the message determiner based on one or more of a description of the data of interest or an updated policy for the message determiner; and

a message generator to generate the data message to be published over the publication-subscription system.

6. The system of claim 1, wherein at least one message is to include a fault monitor message, a fault response message, a fault recovery message, a fault report message, a fault policy message, or a fault prediction message.

7. An apparatus comprising:

a message determiner to dynamically publish a message over a publisher-subscriber system and to dynamically subscribe to a message over the publisher-subscriber system, wherein at least one message is to be used to coordinate actions to manage a fault condition in a computer system, and wherein the message determiner is to include;

a data determiner to dynamically determine a need for data of interest to the message determiner; and

a message generator to generate a request message to one or more of request subscription to a message that is to include the data of interest or prompt subscription to the request message to cause the data of interest to be published over the publication-subscription system.

8. The apparatus of claim 7, further including one or more of:

a fault predictor to predict a fault condition in a high performance computing (HPC) system;

a fault reporter to communicate fault information to a user;

a fault monitor to monitor health of the HPC system;

a fault processor to one or more of respond to the fault condition in the HPC system or recover from the fault condition in the HPC system;

a fault policy maintainer to maintain a rule for a fault management component;

or

a system interface to communicate the fault information over the publisher-subscriber system in real-time to one or more interested message determiners.

9. The apparatus of claim 7, wherein the message determiner is to one or more of:

receive information from a broker, wherein the information is to include one or more of a data message including data of interest or subscriber contact information;

or

provide a message to an aggregator at a higher level of a fault management system hierarchy.

10. The apparatus of claim 7, further including one or more of:

a data determiner to dynamically determine an ability to provide data of interest to at least one other message determiner;

a subscription approver to determine whether to permit subscription to provide a data message including the data of interest;

a capability adjuster to determine whether to adjust a management capability of a fault management component associated with the message determiner based on one or more of a description of the data of interest or an updated policy for the message determiner;

or

a message generator to generate the data message to be published over the publication-subscription system.

11. The apparatus of claim 7, wherein at least one message is to include a fault monitor message, a fault response message, a fault recovery message, a fault report message, a fault policy message, or a fault prediction message.

12. A method comprising:

dynamically publishing a message over a publisher-subscriber system by a message determiner;

dynamically subscribing to a message over the publisher-subscriber system by the message determiner, wherein at least one message is used to coordinate actions to manage a fault condition in a computer system, the method further including;

determining a need for data of interest to the message determiner; and

generating a request message to one or more of request subscription to a message including the data of interest or prompt subscription to the request message to cause the data of interest to be published over the publication-subscription system.

13. The method of claim 12, further including one or more of:

predicting a fault condition in a high performance computing (HPC) system;

communicating fault information to a user;

monitoring health of the HPC system;

one or more of responding to the fault condition in the HPC system or recovering from the fault condition in the HPC system;

maintaining a rule for a fault management component;

or

communicating the fault information over the publisher-subscriber system in real-time to one or more message determiners.

14. The method of claim 12, further including one or more of:

receiving information from a broker, wherein the information includes one or more of a data message including data of interest or subscriber contact information;

or

providing a message to an aggregator at a higher level of a fault management system hierarchy.

15. The method of claim 12, further including one or more of:

determining an ability to provide data of interest to at least one other message determiner that has published a publication request message including a description of the data of interest;

determining whether to permit subscription to provide a data message including the data of interest;

determining whether to adjust a management capability of a fault management component associated with the message determiner based on one or more of a description of the data of interest or an updated policy for the message determiner;

or

generating the data message to be published over the publication-subscription system.

16. The method of claim 12, wherein at least one message is to include a fault monitor message, a fault response message, a fault report message, a fault policy message, or a fault prediction message.

17. At least one non-transitory computer readable storage medium comprising a set of instructions which, when executed by a device, cause the device to:

dynamically publish a message over a publisher-subscriber by a message determiner; and

dynamically subscribe to a message over the publisher-subscriber system by the message determiner, wherein at least one message is to be used to coordinate actions to manage a fault condition in a computer system, and wherein the instructions, when executed, cause a device to;

determine a need for data of interest to the message determiner; and

generate a request message to one or more of request subscription to a message including the data of interest or prompt subscription to the request message to cause the data of interest to be published over the publication-subscription system.

18. The at least one computer readable storage medium of claim 17, wherein the instructions, when executed, cause a device to one or more of:

predict a fault condition in a high performance computing (HPC) system;

communicate fault information to a user;

monitor health of the HPC system;

one or more of respond to the fault condition in the HPC system or recover from the fault condition in the HPC system;

maintain a rule for a fault management component;

or

communicate the fault information over the publisher-subscriber system in real-time to one or more message determiners.

19. The at least one computer readable storage medium of claim 17, wherein the instructions, when executed, cause a device to one or more of:

receive information from a broker, wherein the information is to include one or more of a data message including data of interest or subscriber contact information;

or

provide a message to an aggregator at a higher level of a fault management system hierarchy.

20. The at least one computer readable storage medium of claim 17, wherein the instructions, when executed, cause a device to one or more of:

determine an ability to provide data of interest to at least one other message determiner;

determine whether to permit subscription to provide a data message including data of interest;

determine whether to adjust a management capability of a fault management component that is to be associated with the message determiner based on one or more of a description of the data of interest or an updated policy for the message determiner;

or

generate the data message to be published over the publication-subscription system.

21. The at least one computer readable storage medium of claim 17, wherein at least one message is to include a fault monitor message, a fault response message, a fault report message, a fault policy message, or a fault prediction message.