INTER-PROCESSOR FAILURE DETECTION AND RECOVERY

• US 20120089861A1
• Filed: 10/12/2010
• Published: 04/12/2012
• Est. Priority Date: 10/12/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cardinell, Charles S., Hathorn, Roger G., Laubli, Bernhard, Van Patten, Timothy J.
• Granted Patent: US 8,850,262 B2
• Time in Patent Office: Days
• US Class Current: 714/2
• CPC Class Codes: G06F 11/0724 in a multiprocessor or a multi-core unit multiprocessors per se G06F15/80
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs

Abstract

An approach to detecting processor failure in a multi-processor environment is disclosed. The approach may include having each CPU in the system responsible for monitoring another CPU in the system. A CPU_n reads a timestamp_n+1 created by CPU_n+1 which CPU_n is monitoring from a shared memory location. The CPU_n reads its own timestamp_n and compares the two timestamps to calculate a delta value. If the delta value is above a threshold, the CPU_n determines that CPU_n+1 has failed and initiates error handling for the CPUs in the system. One CPU may be designated a master CPU, and be responsible for beginning the error handling process. In such embodiments, the CPU_n may initiate error handling by notifying the master CPU that CPU_n+1 has failed. If CPU_n+1 is the master CPU, the CPU_n may take additional steps to initiate error handling, and may broadcast a non-critical interrupt to all CPUs, triggering error handling.

Claims

1. An apparatus for detecting processor failure in a multi-processor device, the apparatus comprising:

a retrieval module configured to retrieve a timestamp_n+1 from a shared memory that is shared by a plurality of central processing units (CPUs), wherein the timestamp_n+1 is written to the shared memory by a CPU_n+1;

a comparison module configured to compare the timestamp_n+1 to a timestamp_n generated by a CPU_n checking the CPU_n+1 for failure and determine a delta value;

the comparison module further configured to compare the delta value with a threshold value and determine whether the CPU_n+1 has failed; and

a detection module configured to, in response to the comparison module determining that the CPU_n+1 has failed, initiate error handling for the plurality of CPUs.

2. The apparatus of claim 1, further comprising a timestamp module to read the timestamp_n from hardware and write the timestamp_n to the shared memory.

3. The apparatus of claim 1, wherein the threshold value is set lower than a system threshold value for the system in which the multi-processor device operates.

4. The apparatus of claim 1, wherein the CPU_n+1 is not a master CPU and the CPU_n is not the master CPU, initiating error handling comprising the CPU_n notifying the master CPU of the failure on CPU_n+1, and wherein the master CPU causes the plurality of CPUs to perform error handling.

5. The apparatus of claim 1, wherein the CPU_n+1 is a master CPU, the detection module further configured to:

send a non-critical interrupt to CPU_n+1;

send a critical interrupt to CPU_n+1 in response to the CPU_n+1 failing to respond to the non-critical interrupt; and

broadcast a group non-critical interrupt to all CPUs in response the CPU_n+1 failing to respond to the critical interrupt, wherein the group non-critical interrupt causes the CPUs to perform error handling.

6. The apparatus of claim 1, wherein each of the plurality of CPUs has its own cache line in the shared memory for writing timestamps.

7. The apparatus of claim 1, the comparison module further configured to add additional time to the timestamp_n prior to comparing the timestamp_n+1 to the timestamp_n.

8. The apparatus of claim 7, wherein the additional time accounts for time to move the timestamp_n+1 from CPU_n+1 to CPU_n.

9. The apparatus of claim 1, wherein the shared memory stores one or more timestamps generated by the plurality of CPUs in a global array.

10. A system for detecting processor failure in a multi-processor device, the system comprising:

a shared memory that is shared by a plurality of central processing units (CPUs);

a CPU_n+1 of the plurality of CPUs configured to write a timestamp_n+1 to a global array in the shared memory;

a CPU_n of the plurality of CPUs configured to detect a failure in CPU_n+1, detecting a failure comprising the steps of;

retrieving the timestamp_n+1 from the shared memory;

comparing the timestamp_n+1 to a timestamp_n generated by the CPU_n and determining a delta value;

comparing the delta value with a threshold value and determining whether the CPU_n+1 has failed; and

in response to determining that the CPU_n+1 has failed, initiating error handling for the plurality of CPUs.

11. The system of claim 10, wherein the shared memory and the plurality of CPUs are components of a Fibre Channel Storage Host Adapter.

13. The system of claim 10, wherein the CPU_n is a first core in a multi-core processor and the CPU_n+1 is a second core in a multi-core processor, the multi-core processor comprising a plurality of cores.

14. The system of claim 10, wherein the CPU_n is configured to read the timestamp_n from hardware and write the timestamp_n to the global array.

12. The system of claim 12, wherein the threshold value is set lower than a system threshold value for the Fibre Channel Storage Host Adapter.

15. A method for detecting processor failure in a multi-processor device, the method comprising:

retrieving a timestamp_n+1 from a shared memory that is shared by a plurality of central processing units (CPUs), wherein the timestamp_n+1 is written to the shared memory by a CPU_n+1;

comparing the timestamp_n+1 to a timestamp_n generated by a CPU_n checking the CPU_n+1 for failure; and

in response to the difference between timestamp_n+1 and timestamp_n being larger than a threshold value, determining that there is a failure on CPU_n+1 and initiating error handling for the plurality of CPUs.

16. The method of claim 15, further comprising reading the timestamp_n from hardware and writing the timestamp_n to the shared memory.

17. The method of claim 15, wherein the CPU_n+1 is not a master CPU and the CPU_n is not the master CPU, and wherein initiating error handling comprises the CPU_n notifying the master CPU of the failure on CPU_n+1, and wherein the master CPU causes the plurality of CPUs to perform error handling.

18. The method of claim 15, wherein the CPU_n+1 is a master CPU, the method further comprising:

sending a non-critical interrupt to CPU_n+1;

sending a critical interrupt to CPU_n+1 in response to the CPU_n+1 failing to respond to the non-critical interrupt; and

broadcasting a group non-critical interrupt to all CPUs in response the CPU_n+1 failing to respond to the critical interrupt, wherein the group non-critical interrupt causes the CPUs to perform error handling.

19. The method of claim 15, wherein each of the plurality of CPUs has its own cache line in the shared memory for writing timestamps.

20. The method of claim 15, further comprising adding additional time to the timestamp_n prior to comparing the timestamp_n+1 to the timestamp_n.