Checkpoint restart method and apparatus utilizing multiple log memories

• US 5,828,821 A
• Filed: 06/18/1996
• Issued: 10/27/1998
• Est. Priority Date: 06/19/1995
• Status: Expired due to Fees

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Ohmori, Shigefumi, Fujii, Takahiro, Sakai, Hiroshi, Hoshina, Satoshi, Hirayama, Hideaki, Masubuchi, Yoshio
• Primary Examiner(s): Chan, Eddie P.
• Assistant Examiner(s): NGUYEN, THAN VINH
• Application Number:
  US08/665,542
• Time in Patent Office: 861 Days
• Field of Search: 395/181, 395/182.11, 395/182.13, 395/182.14, 395/182.15, 395/182.17, 395/182.18, 711/3, 711/142, 711/143, 711/147, 711/148, 711/157, 707/202, 707/204
• US Class Current: 714/15
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 12/0806 Multiuser, multiprocessor or multiprocessing cache systems
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064

Abstract

Log memories for recording updated history of a main memory are provided. CPUs record the updated history of the main memory to either of the log memories and writes context thereof and content of a cache memory to the main memory at a checkpoint acquisition. The updated history of the main memory is switched from one of CPUs that has finished a checkpoint processing to other one of the log memories in which the CPUs do not use to record the updated history of the main memory. Normal processing is restarted without waiting for finishing the checkpoint acquisition of the other ones of CPUs.

Claims

1. A checkpoint controlling apparatus for use in a multiprocessor system capable of recovering from a fault during execution of application programs, wherein the multiprocessor system has at least two processor modules, each having a processing unit and a cache memory, a shared memory for use by the processor modules to store data related to the execution of the application programs, and log memory means for recording log data as states of data in both the processor unit and the cache memory for the processor modules before updating the shared memory, and means for storing states of data in both the processor unit and the cache memory for the processor modules in the shared memory at intervals, and wherein each of the processor modules executes checkpoint processing independently, the checkpoint controlling apparatus comprising:

means for selecting one of a first portion and a second portion of the log memory means;

means for storing log data in the selected portion of the log memory means; and

means for switching to the other portion of the log memory means to store log data for the first processor module after the first processor module has completed execution of checkpoint processing.

2. The checkpoint controlling apparatus of claim 1, further comprising second switching means for switching to the other portion of the log memory means to store log data after each processor module completes execution of checkpoint processing.

3. The checkpoint controlling apparatus of claim 2, further comprising:

second determination means for determining whether all of the other processor modules to execute checkpoint processing has completed the execution of the checkpoint processing; and

means for clearing the selected portion of the log memory means when it is determined that all of the other processor modules to execute checkpoint processing completed the execution.

4. The checkpoint controlling apparatus of claim 1, further comprising means for changing a write policy of the cache memory of at least one of the processor modules from a "write" back cache to a "write through" cache, each of the processor modules having the write through cache memory for executing a real time process and stores only a state of data of the processor unit in the shared memory at a checkpoint processing.

5. The checkpoint controlling apparatus of claim 4, further comprising:

means for monitoring an operational state of the multiprocessor system; and

means for changing a number of the processor modules having a cache memory operating as a "write through" cache memory, in accordance with a throughput and a response time of the system monitored by the monitoring means.

6. The checkpoint controlling apparatus of claim 1, wherein the first portion and the second portion of the log memory means is one physical memory, the checkpoint controlling apparatus comprising:

a first counter for holding a first pointer value indicating an address for recording log data in the first portion of the log memory means and, in which the first pointer value is successively incremented from an initial address toward a last address of the physical memory; and

a second counter for holding a second pointer value indicating an address for recording log data in the second portion of the log memory means, in which the second pointer value is successively decremented from a last address toward an initial address of the physical memory.

7. The checkpoint controlling apparatus of claim 6, further comprising:

means for calculating a difference between the first pointer value and the second pointer value to detect whether a remaining size the log memory means is below a predetermined amount;

means for changing a log memory number in a log table to switch between the first and second portions of the log memory means for use after a checkpoint processing;

means for determining whether all of the processor modules have completed checkpoint processing after each of the processor modules performs checkpoint processing; and

means for setting the first pointer value of the first counter to the initial address if the processor module uses the first portion of the log memory means or the second pointer value of the second counter to the last address if the processor uses the second portion of the log memory means.

8. The checkpoint controlling apparatus of claim 1, wherein the selecting means includes:

a log table for storing information showing a corresponding relationship between the processor modules and the first portion and the second portion in the log memory means used by the processor modules, in accordance with a request for switching the first portion and the second portion in the log memory means from each of the processor modules;

means for determining one of the first portion and the second portion of the log memory means used by a processor module by referring to the log table; and

means for selecting a pointer value of one of first and the second counters corresponding to the log memory means.

9. A method of recording log data in log memories having a shared memory shared by processor modules in a multiprocessor system, the log memories including at least first and second log memories for recording log data of the shared memory, the method comprising the steps of:

storing a context of each of the processor modules and a content of each cache memory in the shared memory at each checkpoint processing by each of the processor modules;

switching from the log memory that the processor module uses to the other log memory that the processor module has not used when the checkpoint processing by the processor module is finished;

determining whether a processor module that finished a checkpoint processing is the last of the processor modules to complete checkpoint processing; and

clearing the content of the log memory used by the processor modules if it is determined that the last processor module has completed checkpoint processing.

10. The method of claim 9, wherein the multiprocessor system includes a log table having a log memory number for determining which log memory is used by each of the processor modules, and wherein the determining step comprises the step of:

investigating whether the processor modules, which have not changed the current log memory numbers, are present or not by reference to the log table.

11. The method of claim 9, further comprising the step of:

changing a write policy of the cache memory of at least one of the processor modules from a "write back" cache to a write through cache, each of the processor modules having the "write through" cache memory for executing a real time process and stores only context thereof in the shared memory at a checkpoint processing.

12. The method of claim 11, further comprising the steps of:

monitoring an operational state of the multiprocessor system; and

changing a number of the processor modules having a cache memory operating as a "write through" cache memory, in accordance with a throughput and a response time of the system monitored by the monitoring step.

13. The method of claim 9, wherein the first and the second log memories are included in one physical memory, a first counter for holding a pointer value indicating an address for recording log data in the first log memory and, in which a pointer value is successively incremented from an initial address toward a last address of the physical memory, and a second counter for holding a pointer value indicating an address for recording log data in the second log memory, in which a pointer value is successively decremented from a last address toward an initial address of the physical memory, the method further comprising the steps of:

calculating a difference between a pointer value in the first log counter and a pointer value in the second log counter to detect whether a remaining size the log memory is below a predetermined amount;

changing a current log memory number in a log table to switch the log memory for use after a checkpoint processing; and

determining whether a processor module that finished the checkpoint processing is the last processor module of the processor modules to complete the checkpoint processing; and

setting a pointer value of the first counter to the initial address if the processor module uses the first log memory or a pointer value of the second counter to the last address if the processor module uses the second log memory.

14. A checkpoint controlling method for use in a fault tolerant multiprocessor system capable of recovering from a fault during execution of application programs, wherein the fault tolerant multiprocessor system has a plurality of log memories, at least two processor modules each having a processor unit and a cache memory, a shared memory, and means for storing data in the shared memory at intervals, and wherein each of the processor modules executes checkpoint processing independently, the checkpoint controlling method comprising the steps of:

selecting one of a first log memory and a second log memory of the plurality of log memories of the fault tolerant multiprocessor system;

storing log data in the selected one of the first and the second log memories, the log data including a context of the processing unit and a content of the cache memory for the processor modules executing checkpoint processing;

determining whether all of the processor modules have completed execution of checkpoint processing; and

clearing the selected one of the first and the second log memories when it is determined that all of the processor modules have completed execution of checkpoint processing.

15. The method of claim 14 further comprising the steps of:

determining whether a first one of the processor modules has completed checkpoint processing; and

selecting one of the log memories that is different from the one of the log memories previously selected to store log data for the first processor module, when it has been determined that the first module has completed checkpoint processing.

16. The method of claim 15 wherein the fault tolerant multiprocessor system includes a log table for maintaining a list identifying which of the plurality of log memories is used to store log data for each of the plurality of processor modules, and wherein the selecting step comprises the substep of:

updating the log table to indicate that a different one of the log memories has been selected to store log data for the first processor module.

17. The method of claim 16 wherein a single physical memory includes a plurality of portions, each of said portions corresponding to one of the plurality of log memories, and the multiprocessor system includes a first counter for holding a first pointer value indicating an address for recording log data in the first log memory, the first pointer value being incremented from an initial address to a last address of the physical memory, and a second counter for holding a second pointer value indicating an address for recording log data in the second log memory, the second pointer value being decremented from the last address to the initial address of the physical memory, the method further comprising the steps of:

calculating a difference between the first pointer value and the second pointer value to detect whether a remaining size the log memory is below a predetermined amount;

changing a current log memory number of a log table to switch the log memory for use after a checkpoint processing;

determining whether a processor module that finished the checkpoint processing is last processor module of the processor modules to complete the checkpoint processing; and

setting the first pointer value to the initial address if the processor module uses the first log memory or to the second pointer value if the processor module uses the second log memory.

18. A checkpoint controlling method for use in a fault tolerant multiprocessor system capable of recovering from a fault during execution of application programs, wherein the fault tolerant multiprocessor system has a plurality of log memories, at least two processor modules each having a processor unit and a cache memory, a shared memory, and means for storing data in the shared memory at intervals, and wherein each of the processor modules repeatedly executes a checkpoint processing independently, the checkpoint controlling method comprising the steps of;

selecting one of a first log memory and a second log memory of the plurality of log memories of the fault tolerant multiprocessor system;

storing log data in the selected one of the first and the second log memories, the log data including a context of the processing unit and a content of the cache memory for the processor modules executing checkpoint processing;

determining whether all of the processor modules have completed execution of checkpoint processing; and

clearing the selected one of the first and the second log memories when it is determined that all of the processor modules have completed an iteration of the checkpoint processing.

19. A checkpoint controlling method for use in a fault tolerant multiprocessor system capable of recovering from a fault during execution of application programs, wherein the fault tolerant multiprocessor system has a plurality of log memories, at least two processor modules each having a processor unit and a cache memory, a shared memory, and means for storing data in the shared memory at intervals, and wherein each of the processor modules executes checkpoint processing independently, the checkpoint controlling method comprising the steps of:

(a) selecting a first log memory of the plurality of log memories of the fault tolerant multiprocessor system;

(b) determining whether a first one of the processor modules, has completed execution of checkpoint processing;

(c) storing log data for the first processor module in the selected first log memory, the log data including a context of the processing unit and a content of the cache memory for the first processor module at a time prior to execution of the checkpoint processing;

(d) switching to a second log memory different than the first log memory, selected during step (a);

(e) determining whether a second one of the processor modules has completed execution of checkpoint processing; and

(f) storing log data for the second processor module in the first log memory, the log data including a context of the processing unit and a content of the cache memory for the second processor module at a time prior to execution of the checkpoint processing.

20. A checkpoint controlling system for use in a fault tolerant multiprocessor system capable of recovering from a fault during execution of application programs, wherein the fault tolerant multiprocessor system has a plurality of log memories, at least two processor modules each having a processor unit and a cache memory, a shared memory, and means for storing data in the shared memory at intervals, and wherein each of the processor modules executes checkpoint processing independently, the checkpoint controlling system comprising:

means for selecting one of a first log memory and a second log memory of the plurality of log memories of the fault tolerant multiprocessor system;

means for storing log data in the selected one of the first and the second log memories, the log data including a context of the processing unit and a content of the cache memory for the processor modules executing checkpoint processing;

means for determining whether all of the processor modules have completed execution of checkpoint processing; and

means for clearing the selected one of the first and the second log memories when it is determined that all of the processor modules have completed execution of checkpoint processing.

21. A fault tolerant multiprocessor system capable of recovering from a fault during execution of application programs, comprising:

a plurality of log memories;

at least two processor modules, each having a processor unit and a cache memory, wherein each of the processor modules executes checkpoint processing independently;

a shared memory;

means for storing data in the shared memory at intervals;

means for selecting one of a first log memory and a second log memory of the plurality of log memories;

means for storing log data in the selected one of the first and the second log memories, the log data including a context of the processing unit and a content of the cache memory for the processor modules executing checkpoint processing;

means for determining whether all of the processor modules have completed execution of checkpoint processing; and

means for clearing the selected one of the first and the second log memories when it is determined that all of the processor modules have completed execution of checkpoint processing.