Memory management in fault tolerant computer systems utilizing a first and second recording mechanism and a reintegration mechanism

• US 5,953,742 A
• Filed: 07/01/1996
• Issued: 09/14/1999
• Est. Priority Date: 07/01/1996
• Status: Expired due to Term

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Williams, Emrys John
• Primary Examiner(s): Swann, Tod R.
• Assistant Examiner(s): Thai, Tuan V.
• Application Number:
  US08/675,265
• Time in Patent Office: 1,170 Days
• Field of Search: 395/575, 395/182.01, 395/182.03, 395/182.09, 395/182.1, 395/182.13, 395/182.17, 395/182.18, 395/182.11, 711/163, 711/167, 711/154
• US Class Current: 711/154
• CPC Class Codes: G06F 11/141 for bus or memory accesses
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/181 Eliminating the failing redundant component

Abstract

A memory management system for a fault tolerant computer system. The memory management system includes a first recording mechanism which can be activated to record memory update events; a second recording mechanism which records at least a limited number of memory update events; a fault input for a fault signal to activate the first recording mechanism in the event of a fault event; and a memory reintegration mechanism to reintegrate at least parts of memory identified in the first and second recording mechanisms. Preferably, the recording of memory updates (writes) is not based on recording each address accessed, but rather on memory segments (pages) updated (written to). Further, a fault tolerant computer system includes a plurality of synchronous processing sets, each having a processor with internal memory and operating in lockstep, and an out of sync detector for detecting an out-of-sync-event and for generating an out-of-sync signal.

Claims

1. A memory management system for a fault tolerant computer system, said memory management system comprising:

a first recording mechanism which can be activated to record memory update events;

a fault input for a fault signal to activate said first recording mechanism in the event of a fault event;

a second recording mechanism having a capacity to record at least a limited number of memory update events, and to maintain a record of recent memory update events up to a number sufficient to cover the time to activate said first recording mechanism following said fault event, said second recording mechanism comprises a first-in-first-out buffer, an output of said first-in-first-out buffer is connected to said first recording mechanism, and further said first-in-first-out buffer stores up to a predetermined number of update addresses, an address decoder is connected to said output of said first-in-first-out buffer to generate a page signal representative of a memory update address output from said first-in-first-out buffer and said address decoder is responsive to said fault signal to pass said page signal to said first recording mechanism; and

a memory reintegration mechanism to reintegrate at least parts of memory identified in said first and second recording mechanisms.

2. A memory management system according to claim 1, wherein said first recording mechanism is a memory management unit comprising storage having an entry for each of a plurality of memory pages, a code being written to a page entry each time that page is written to when said first recording mechanism has been activated.

3. A memory management system according to claim 2, wherein said first recording mechanism has an enable input connected to receive the fault signal for activating said first recording mechanism.

4. A memory management system according to claim 1, wherein said second recording mechanism comprises a logic analyzer.

5. A memory management system according to claim 1, wherein said second recording mechanism maintains a record of recent memory update events up to a number sufficient to cover the time to activate said first recording mechanism following a fault event, the operation of said second recording means being inhibited in response to said fault signal.

6. A memory management system according to claim 5, wherein said first recording mechanism comprises a software generated list of update events.

7. A memory management system according to claim 1, wherein said second recording mechanism comprises a table look-aside buffer and a memory access table maintained in main memory.

8. A memory management system according to claim 1, wherein said memory reintegration mechanism is operative to reintegrate memory pages identified in said first and second recording mechanisms.

9. A fault tolerant computer system comprising a plurality of synchronous processing sets, each including a processor and internal memory and operating in lockstep, and an out-of-sync detector for detecting an out-of-sync event and for generating an out-of-sync signal, wherein each processing set includes:

a first recording mechanism which can be activated to record memory update events;

a fault input for a fault signal to activate said first recording mechanism in the event of a fault event;

a second recording mechanism having a capacity to record at least a limited number of memory update events, and to maintain a record of recent memory update events up to a number sufficient to cover the time to activate said first recording mechanism following said fault event, said second recording mechanism comprises a first-in-first-out buffer, an output of said first-in-first-out buffer is connected to said first recording mechanism, and further said first-in-first-out buffer stores up to a predetermined number of update addresses, an address decoder is connected to said output of said first-in-first-out buffer to generate a page signal representative of a memory update address output from said first-in-first-out buffer and said address decoder is responsive to said fault signal to pass said page signal to said first recording mechanism; and

a memory reintegration mechanism to reintegrate at least parts of memory identified in said first and second recording mechanisms.

10. A fault tolerant computer system comprising a plurality of synchronous processing sets, each comprising a processor and internal memory and operating in lockstep, and an out-of-sync detector for detecting an out-of-sync event and for generating an out-of-sync signal, wherein each processing set also comprises:

a first recording mechanism which can be activated to record memory write events;

a fault input for receiving said out-of-sync signal to activate said first recording mechanism in the event of said out-of-sync event;

a second recording mechanism having a capacity to record at least a limited number of memory write events, and to maintain a record of recent memory update events up to a number sufficient to cover the time to activate said first recording mechanism following said out-of-sync event, said second recording mechanism comprises a first-in-first-out buffer, an output of said first-in-first-out buffer is connected to said first recording mechanism, and a further said first-in-first-out buffer stores up to a predetermined number of update addresses, an address decoder is connected to said output of said first-in-first out buffer to generate a page signal representative of a memory update address output from said first-in-first-out buffer and said address decoder is responsive to said out-of-sync signal to pass said page signal to said first recording mechanism; and

a memory reintegration mechanism to reintegrate in an out-of-sync processing set at least parts of memory identified in said first and second recording mechanisms.

11. A fault tolerant computer system according to claim 10, wherein said first recording mechanism is a memory management unit comprising a RAM with an entry for each of a plurality of memory pages, a code being written to a page entry each time that page is written to when said first recording mechanism has been activated.

12. A fault tolerant computer system according to claim 11, wherein said first recording mechanism has an enable input connected to receive the out-of-sync signal for activating said first recording mechanism.

13. A fault tolerant computer system according to claim 10, wherein said second recording mechanism comprises a logic analyzer.

14. A fault tolerant computer system according to claim 10, wherein said second recording mechanism maintains a record of recent memory update events up to a number sufficient to cover the time to activate said first recording mechanism following the out-of-sync event, the operation of said second recording means being inhibited in response to said out-of-sync signal.

15. A fault tolerant computer system according to claim 14, wherein said first recording mechanism comprises a software generated list of update events.

16. A fault tolerant computer system according to claim 10, wherein said second recording mechanism comprises a table look-aside buffer and a memory access table maintained in main memory.

17. A fault tolerant computer system according to claim 10, wherein said memory reintegration mechanism is operative to reintegrate memory pages identified in said first and second recording mechanisms.

18. A fault tolerant computer system according to claim 10, comprising three synchronous processing sets operating in lockstep, wherein said out-of-sync detector determines an out-of-sync processing set by majority voting.

19. A fault tolerant computer system according to claim 18, wherein said out-of-sync detector is arranged to select one of the remaining two processing sets, to supply an input to the out-of-sync processing set and the remaining processing set to cause said out-of-sync and remaining processing sets to idle, to reintegrate one of said out-of-sync and remaining processing sets while maintaining a software log of memory write events, and then to reintegrate said other of said out-of-sync and remaining processing sets using said software log.

20. A method for reintegration of a processing set of a fault tolerant computer system comprising a plurality of synchronous processing sets, each comprising a processor and internal memory and operating in lockstep, and a fault detector for detecting a fault event and for generating a fault signal, said method comprising:

maintaining a temporary record of memory update events over a limited period, said temporary record being stored in a first-in-first-out buffer;

connecting a recording mechanism for recording an output of said first-in-first-out buffer;

responding to said fault signal to activate a further record of memory update events following said fault signal;

maintaining a record of recent memory update events up to a number sufficient to cover the time to activate said further record;

storing up to a predetermined number of update addresses in said first-in-first-out buffer, supplying the output of said first-in-first out buffer to an address decoder to generate a page signal representative of a memory update address output from said first-in-first-out buffer, and recording said page signal as part of said further record when said fault signal is active; and

performing memory reintegration in a processor in which a fault has occurred for at least those parts of memory identified in said temporary and further memory records.

21. A method according to claim 20, wherein said fault event is an out-of-sync event.

22. A method according to claim 20, wherein the further record is stored in a storage of a memory management unit, a page entry in said storage being provided for each page of memory, a code being written to a page entry each time that page is written to when said recording mechanism has been activated.

23. A method according to claim 20, wherein said temporary record is stored in a logic analyzer.

24. A method according to claim 20, comprising the steps of maintaining said temporary record of recent memory update events up to a number sufficient to cover the time to activate said first recording mechanism following a fault, said temporary record being inhibited in response to said fault signal.

25. A method according to claim 24, comprising generating a list of update events by software.

26. A method according to claim 20, comprising forming said temporary record using a table look-aside buffer and a memory access table maintained in main memory.

27. A method according to claim 20, comprising the step of reintegrating memory pages identified in said temporary and further records.

28. A method according to claim 20, comprising three synchronous processing sets operating in lockstep, wherein an out-of-sync detector determines an out-of-sync processing set by majority voting.

29. A method according to claim 20, comprising the steps of, in response to the identification of an out-of-sync processing set, selecting one of the remaining two processing sets, supplying an interrupt to the out-of-sync processing set and the remaining processing set to cause said out-of-sync and remaining processing sets to idle, reintegrating one of said out-of-sync and remaining processing sets while maintaining a software log of memory write events, and then reintegrating said other of said out-of-sync and remaining processing sets using said software log.

30. A memory management system for a fault tolerant computer system, said memory management system comprising:

a first recording mechanism which can be activated to record memory update events;

a fault input for a fault signal to activate said first recording mechanism in the event of a fault event;

a second recording mechanism having a capacity to record at least a limited number of memory update events, and to maintain a record of recent memory update events up to a number sufficient to cover the time to activate said first recording mechanism following said fault event; and

a memory reintegration mechanism to reintegrate at least parts of memory identified in said first and second recording mechanisms, wherein said memory reintegration mechanism is operative to reintegrate memory pages identified in said first and second recording mechanisms.

31. A fault tolerant computer system having a plurality of synchronous processing sets, each including a processor and internal memory and operating in lockstep, and an out-of-sync detector for detecting an out-of-sync event and for generating an out-of-sync signal, wherein each processing set includes a memory management system, said memory management system comprising:

a first recording mechanism which can be activated to record memory update events;

a fault input for a fault signal to activate said first recording mechanism in the event of a fault event;

a second recording mechanism having a capacity to record at least a limited number of memory update events, and to maintain a record of recent memory update events up to a number sufficient to cover the time to activate said first recording mechanism following said fault event; and

a memory reintegration mechanism to reintegrate at least parts of memory identified in said first and second recording mechanisms, wherein said memory reintegration mechanism is operative to reintegrate memory pages identified in said first and second recording mechanisms.

32. A fault tolerant computer system comprising a plurality of synchronous processing sets, each comprising a processor and internal memory and operating in lockstep, and an out-of-sync detector for detecting an out-of-sync event and for generating an out-of-sync signal, wherein each processing set also comprises:

a first recording mechanism which can be activated to record memory write events;

a fault input for receiving said out-of-sync signal to activate said first recording mechanism in the event of said out-of-sync event;

a second recording mechanism having a capacity to record at least a limited number of memory write events, and to maintain a record of recent memory update events up to a number sufficient to cover the time to activate said first recording mechanism following said out-of-sync event, and

a memory reintegration mechanism to reintegrate in an out-of-sync processing set at least parts of memory identified in said first and second recording mechanisms, wherein said memory reintegration mechanism is operative to reintegrate memory pages identified in said first and second recording mechanisms.

33. A method for reintegration of a processing set of a fault tolerant computer system including a plurality of synchronous processing sets, each having a processor and internal memory and operating in lockstep, and a fault detector for detecting a fault event and for generating a fault signal, said method comprising:

maintaining a temporary record of memory update events over a limited period;

responding to said fault signal to activate a further record of memory update events following said fault state;

maintaining a record of recent memory update events up to a number sufficient to cover the time to activate said further record;

performing memory reintegration in a processor in which a fault has occurred for at least those parts of memory identified in said temporary and further memory records.