System and method of implementing heartbeats in a multicore system

• US 10,459,782 B2
• Filed: 08/31/2017
• Issued: 10/29/2019
• Est. Priority Date: 08/31/2017
• Status: Active Grant

• Current Assignee: NXP USA, Inc. (NXP Semiconductors NV)
• Original Assignee: NXP USA, Inc. (NXP Semiconductors NV)
• Inventors: Baca, David
• Primary Examiner(s): Wilson, Yolanda L
• Application Number:
  US15/693,252
• Publication Number:
  US 20190065296A1
• Time in Patent Office: 789 Days
• CPC Class Codes: G06F 11/0724 in a multiprocessor or a multi-core unit multiprocessors per se G06F15/80
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/182 based on mutual exchange of the output between redundant processing components
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064

Abstract

Counter based heartbeat messaging is implemented by storing heartbeat count vectors and health vectors of each core in a shared memory. Each core implements its heartbeat operation by storing the heartbeat count and health vectors from shared to local memory. A core uses its locally stored vectors to detect fault conditions at the other cores, and to achieve interactive consistency. Any core can initiate a system reaction to a core having a failing health status when a defined number of cores agree with that status.

Claims

1. A method comprising:

storing to local memory of a first core, from shared memory of a system, during a first heartbeat cycle, a plurality of local heartbeat vectors, including a first local heartbeat vector (HBV) of a first core, a second local HBV of a second core, and a third local HBV of a third core, wherein, each cores'"'"' local HBV includes a first HeartBeat Count (HBC) for the first core, a second HBC for the second core, and a third HBC for the third core;

updating, by the first core during the first heartbeat cycle, the first HBC of the first local HBV, wherein no other core'"'"'s current HBC is updated by the first core during the first heartbeat cycle;

determining, by the first core during the first heartbeat cycle, a health status of the second core based solely upon the plurality of local HBVs stored at the first core after the first HBC has been updated;

determining, by the first core during the first heartbeat cycle, a health status of the third core based solely upon the plurality of local HBVs stored at the first core after the first HBC has been updated; and

updating, by the first core during the first heartbeat cycle after the first HBC has been updated, a shared HBV of the first core in shared memory with an updated HBV based upon the first HBC of the first local HBV, the second HBC of the second local HBV, and the third HBC of the third local HBV.

2. The method of claim 1, wherein determining the health status of the second core is based upon a comparison of the second HBC of the first local HBV and the second HBC of the second local HBV.

3. The method of claim 2, wherein determining the health status of the second core further comprises:

determining the health status of the second core is OK if a difference between the second HBC of the first local HBV and the second HBC of the second local HBV does not meet a fail threshold, otherwise determining the health status of the core is NOT OK.

4. The method of claim 3 further comprising:

in response to determining the health status of the second core is NOT OK, determining if the health status of the second core as determined by the third core prior to storing the shared HBV of the third core was also NOT OK, and if so providing, by the first core, a fail indicator to indicate a failure of the second core has been detected, otherwise not providing a fail indicator.

5. The method of claim 4 further comprising:

storing, during the first heartbeat cycle prior to determining if the health status of the second core as determined by the third core was also NOT OK, shared health vectors for each one of the first core, the second core, and the third core, as a plurality of local health vectors including a first local health vector, a second local health vector, and a third local health vector, respectively, wherein the third local health vector includes the health status of the second core as determined by the third core.

6. The method of claim 4 further comprising:

implementing a remedial action by the system in response to the first core providing the fail indicator.

7. The method of claim 3, wherein determining the health status of the second core is NOT OK includes determining that the second core is experiencing on of a high-frequency failure or a low-frequency failure.

8. The method of claim 1 wherein updating the first HBC of the first local HBV includes incrementing the first HBC.

9. The method of claim 1 further comprising:

storing to local memory of the first core, from the shared memory, during a second heartbeat cycle, the plurality of local heartbeat vectors;

updating, by the first core during the second heartbeat cycle, the first HBC of the first core in the first local HBV, wherein no other core'"'"'s current HBC is updated by the first core during the second heartbeat cycle;

determining, by the first core during the second heartbeat cycle, a health status of the second core based solely upon the plurality of local HBVs after the first HBC of the first local HBV has been updated;

determining, by the first core during the second heartbeat cycle, a health status of the third core based solely upon the plurality of local HBVs after the first HBC of the first local HBV has been updated; and

updating, by the first core during the second heartbeat cycle after the first HBC has been updated, the shared HBV of the first core in shared memory with the first HBC of the first local HBV, the second HBC of the second local HBV, and the third HBC of the third local HBV.

10. The method of claim 1 further comprising:

storing, during the first heartbeat cycle and after the first core has updated its shared HBV, the shared heartbeat vector (HBV) of the first core, the shared HBV of the second core, and the shared HBV of the third core from the shared memory to local memory of the core as a plurality of local HBVs including a first local HBV, a second local HBV, and a third local HBV, respectively;

during the first heartbeat cycle, storing to local memory of a second core from the shared memory after the first core has updated its shared HBV, the plurality of local heartbeat vectors;

updating, by the second core during the first heartbeat cycle, the second HBC of the second local HBV, wherein no other HBC of the plurality of local HBVs is updated in the local memory of the second core during the first heartbeat cycle;

determining, by the second core during the first heartbeat cycle, a health status of the first core based solely upon the plurality of local HBVs stored at the second core after the second HBC has been updated;

determining, by the second core during the first heartbeat cycle, a health status of the first core based solely upon the plurality of local HBVs stored at the second core after the second HBC has been updated; and

updating, by the second core during the first heartbeat cycle after the first HBC has been updated, a shared HBV of the second core in shared memory with an updated HBV based upon the first HBC of the first local HBV, the second HBC of the second local HBV, and the third HBC of the third local HBV.

11. The method of claim 10, wherein the storing, updating, determinings, and updating of the first core and of the second core are atomic to each other.

12. The method of claim 10, wherein one or more of the storing, updating, determinings, and updating of the first core can overlap with the storing of the second core.

13. A device comprising:

a shared memory configured to store a plurality of shared HeartBeat Vectors (HBVs), including a first shared HBV of a first core, a second shared HBV of a second core, and a third shared HBV of a third core, each HBV includes a HeartBeat Count (HBC) of each core, including a first HBC of the first core, a second HBC of the second core, and a third HBC of the third core;

the first core coupled to the shared memory and includes;

local memory,

a bus controller coupled to the local memory and to the shared memory,

a heartbeat controller coupled to the bus controller to implement heartbeat operations, during each heartbeat operation, the heartbeat controller is configured to;

enable the bus controller to store the plurality of shared HBVs at the local memory as plurality of local HBVs, including a first local HBV, a second local HBV, and a third local HBV;

enable heartbeat update circuitry that is configured to update the first HBC of the first local HBV;

enable health calculator circuitry that is configured to calculate a health of the second core based on the first and second HBCs of the first and second local HBVs, and configured to calculate a health of the third core based on the first and third HBCs of the first and third local HBVs; and

enable the bus controller to update the first shared HBV based upon the first HBC of the first local HBV, the second HBC of the second local HBV, and the third HBC of the third local HBV.

14. The device of claim 13, wherein the heartbeat controller further enables the bus controller to store a plurality of share health vectors from the shared memory to the local memory as a plurality of local health vectors, including a first local health vector of the first core, a second local health vector of the second core, and a third local health vector of the third core, and

the first heartbeat controller is further configured to enable failure detect circuitry to provide a fail indicator if calculated heath of the second core is NOT OK and a stored health of the second core in the third local health vector is the same as the calculated heath of the second core.

15. The device of claim 14, wherein the heartbeat controller further enables the bus controller to update a first shared health vector based upon the calculated health of the first second core and of the third core.

16. The device of claim 14 further comprising reaction circuitry coupled to the first core, the reaction circuitry configured to implement a remedial action in response to receiving the fail indicator.

17. The device of claim 13, wherein

the second core is coupled to the shared memory and includes:

local memory,

a bus controller coupled to the local memory and to the shared memory,

a heartbeat controller coupled to the bus controller to implement heartbeat operations, during each heartbeat operation, the heartbeat controller is configured to;

enable the bus controller to store the plurality of shared HBVs at the local memory as the plurality of local HBVs;

enable heartbeat update circuitry that is configured to update the second HBC of the second local HBV;

enable health calculator circuitry that is configured to calculate a health of the first core based on the first and second HBCs of the first and second local HBVs, and configured to calculate a health of the third core based on the second and third HBCs of the second and third local HBVs; and

enable the bus controller to update the second shared HBV based upon the first HBC of the first local HBV, the second HBC of the second local HBV, and the third HBC of the third local HBV.

18. A method comprising:

storing to local memory of a first core, from shared memory of a system, during a first heartbeat cycle, a plurality of local heartbeat vectors and a plurality of local health vectors,

the plurality of local heartbeat vectors include a first local heartbeat vector (HBV) of a first core, a second local HBV of a second core, and a third local HBV of a third core, wherein, each cores'"'"' local HBV includes a first HBC of the first core, a second HBC of the second core, and a third HBC of the third core,

the plurality of local health vectors include a first local health vector of the first core, a second local health vector of a second core, and a third local health vector of a third core, wherein, each cores'"'"' local health includes a health status of each other core, and

determining, by the first core during the first heartbeat cycle, a first health status of the second core based solely upon the plurality of local health vectors, wherein if the first health status of the second core is determined to be NOT OK, based upon the plurality of local health vectors, updating the health status of the second core in the first local health vector to be NOT OK, otherwise, determining, a second health status of the second core based solely upon the plurality of local HBVs stored at the first core.

19. The method of claim 18 further comprising:

updating, by the first core during the first heartbeat cycle, the health vector of the second core stored in shared memory based upon the first health status if the first health status is NOT OK, otherwise, updating, by the first core during the first heartbeat cycle, the health vector of the second core stored in shared memory based upon the second health status.

20. The method of claim 19, wherein the health status of NOT OK indicates one of a high-frequency failure and a low-frequency failure of the second core.