Fault tolerant multi-node computing system for parallel-running a program under different environments

• US 20040205377A1
• Filed: 03/05/2004
• Published: 10/14/2004
• Est. Priority Date: 03/28/2003
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Nakamura, Nobutatsu, Hiraike, Ryuichi
• Granted Patent: US 7,237,140 B2
• Time in Patent Office: Days
• US Class Current: 714/4
• CPC Class Codes: G06F 11/1494 N-modular type

Abstract

In a fault tolerant computing system, a number of processing nodes are connected via a communication channel to a management node. The management node has the function of setting the processing nodes in uniquely different configurations and operating the processing nodes in parallel to execute common application programs in parallel, so that the application programs are run under diversified environments. The management node has the function of respectively distributing request messages to the processing nodes. The configurations of the processing nodes are sufficiently different from each other to produce a difference in processing result between response messages when a failure occurs in one of the processing nodes. Alternatively, the management node has the function of selecting, from many processing nodes, those processing nodes which are respectively set in uniquely different configurations and operating the selected processing nodes in parallel to execute the common application programs.

Claims

1. A fault tolerant computing system comprising a plurality of processing nodes, said processing nodes respectively having uniquely different configurations for parallel-running a common application program so that the common application program is by the processing nodes run under respectively diversified environments which are sufficiently different from each other to produce a plurality of different processing results when a failure occurs in one of said processing nodes.

2. A fault tolerant computing system comprising:

a plurality of processing nodes interconnected by a communication channel, each of said processing nodes having an identical application program;

control means for respectively setting said processing nodes in uniquely different configurations and operating said processing nodes in parallel to execute the application programs in environments corresponding to said uniquely different configurations; and

distributing means for distributing request messages respectively to said processing nodes, so that said request messages are individually processed by said application programs to produce a plurality of response messages,

wherein the configurations of said processing nodes are sufficiently different from each other to produce a difference in processing result between said response messages when a failure occurs in one of said processing nodes.

3. The fault tolerant computing system of claim 2, wherein each of said configurations comprises hardware configuration, software configuration, external device configuration and program startup configuration.

4. The fault tolerant computing system of claim 3, wherein said hardware configuration data includes parameters indicating main memory size, virtual memory size, memory access timing, processor operating speed, bus transmission speed, bus width, number of processors, read cache memory size, write cache memory size, cache'"'"'s valid/invalid status, and types of processors and memories, and wherein each of said processing nodes differs from every other processing nodes in one of said parameters of hardware configuration data.

5. The fault tolerant computing system of claim 3, wherein said software configuration data includes parameters indicating operating system, basic software, various device drivers, and various library versions, and wherein each of said processing nodes differs from every other processing nodes in one of said parameters of software configuration data.

6. The fault tolerant computing system of claim 3, wherein said external device configuration data includes parameters indicating types of external devices, display unit, input/ output device, and communication interface, and wherein each of said processing nodes differs from every other processing nodes in one of said parameters of external device configuration data.

7. The fault tolerant computing system of claim 3, wherein said program startup configuration data includes parameters indicating (a) interruption and reboot of an application program by suspending the CPU while holding contents of main memory, (b) interruption and reboot of an application program by saving contents of the main and virtual memories on a hard disk, suspending operating system and restoring the saved memory contents when operating system is restarted, (c) interruption and reboot of an application program by suspending and restarting operating system, (d) interruption and reboot of an application program by forcibly interrupting and restarting the program and operating system, (e) restarting of an application program and operating system after reinstalling the program and operating system, and (f) restarting an application program and operating system after performing a clear install of the program and operating system, and wherein each of said processing nodes differs from every other processing nodes in one of said parameters of program startup configuration data.

8. The fault tolerant computing system of claim 2, wherein said distributing means includes a memory for storing distribution data, wherein the distributing means distributes said request messages to said processing nodes according to the stored distribution data, further comprising fault recovery means for monitoring said processing nodes and updating the stored distribution data if a failure is detected in said processing nodes.

9. The fault tolerant computing system of claim 8, wherein said fault recovery means is arranged to alter the configuration of one of said processing nodes when a failure is detected in said one processing node and restart said processing nodes.

10. The fault tolerant computing system of claim 8, wherein said fault recovery means is arranged to alter the program startup configuration of one of said processing nodes when a failure is detected in said one processing node and restart said processing nodes.

11. The fault tolerant computing system of claim 8, wherein said fault recovery means is arranged to alter checkpointing data of each of said processing nodes when a failure is detected so that the checkpointing data of each processing node differs from the checkpointing data of every other processing nodes.

12. The fault tolerant computing system of claim 2, wherein said control means includes a memory for storing a plurality of uniquely different configuration data corresponding to said processing nodes, and wherein said processing nodes are respectively set in said configurations according to the stored uniquely different configuration data.

13. The fault tolerant computing system of claim 2, further comprising decision means for receiving said response messages from said processing nodes and selecting one of the response messages.

14. The fault tolerant computing system of claim 13, wherein said decision means selects one of the response messages on a first-served-first-arrived basis.

15. The fault tolerant computing system of claim 13, wherein said decision means makes a majority decision on a plurality of said response messages which arrive in a specified time interval and selects one of the majority-decided response messages.

16. The fault tolerant computing system of claim 13, wherein said decision means makes a majority decision on a predetermined number of said response messages and selects one of the majority-decided response messages.

17. The fault tolerant computing system of claim 13, wherein said decision means calculates an average value from said plurality of response messages and formulates a response message with said average value.

18. The fault tolerant computing system of claim 13, wherein said decision means receives an external request message from an external source via said communication channel and sends the received message to one of said processing nodes.

19. The fault tolerant computing system of claim 13, wherein said decision means comprises:

a memory for storing distribution data;

means for updating the distribution data according to operating status of said processing nodes; and

means for supplying said external request message to said one of the processing nodes according to the distribution data.

20. The fault tolerant computing system of claim 2, wherein said control means and said distributing means are provided in a management node connected to said processing nodes via said communication channel.

21. The fault tolerant computing system of claim 12, further comprising a commonly-shared memory for storing a plurality of uniquely different configuration data corresponding to said processing nodes, and wherein said processing nodes are accessible to said memory for setting said configurations according to the stored uniquely different configuration data.

22. The fault tolerant computing system of claim 2, wherein each of said processing nodes comprises:

a memory for storing a plurality of uniquely different configuration data corresponding to said processing nodes;

fault recovery means for monitoring the processing node and updating corresponding one of the stored configuration data if a failure is detected in the processing node;

means for exchanging update information with every other processing nodes whenever there is a change in the stored configuration data.

23. The fault tolerant computing system of claim 2, wherein each of said processing nodes is one of a real machine.

24. The fault tolerant computing system of claim 2, wherein each of said processing nodes is one of a virtual machine.

25. A fault tolerant computing system comprising:

a plurality of processing nodes interconnected by a communication channel, each of said processing nodes having an identical application;

control means for selecting said processing nodes which are respectively set in uniquely different configurations and operating the selected processing nodes in parallel to execute the application programs in diversified environments corresponding to said uniquely different configurations; and

distributing means for distributing request messages respectively to the selected processing nodes, so that said request messages are individually processed by the application programs of the selected processing nodes to produce a plurality of response messages,

wherein the configurations of said selected processing nodes are sufficiently different from each other to produce a difference in processing result between said response messages when a failure occurs in one of said selected processing nodes.

26. The fault tolerant computing system of claim 25, wherein each of said node configuration data comprises hardware configuration data, software configuration data, external device configuration data and program startup configuration data.

27. The fault tolerant computing system of claim 26, wherein said hardware configuration data includes parameters indicating main memory size, virtual memory size, memory access timing, processor operating speed, bus transmission speed, bus width, number of processors, read cache memory size, write cache memory size, cache'"'"'s valid/invalid status, and types of processors and memories, and wherein each of said processing nodes differs from every other processing nodes in one of said parameters of hardware configuration data.

28. The fault tolerant computing system of claim 26, wherein said software configuration data includes parameters indicating operating system, basic software, various device drivers, and various library versions, and wherein each of said processing nodes differs from every other processing nodes in one of said parameters of software configuration data.

29. The fault tolerant computing system of claim 26, wherein said external device configuration data includes parameters indicating types of external devices, display unit, input/output device, and communication interface, and wherein each of said processing nodes differs from every other processing nodes in one of said parameters of external device configuration data.

30. The fault tolerant computing system of claim 26, wherein said program startup configuration data includes parameters indicating (a) interruption and reboot of an application program by suspending the CPU while holding contents of main memory, (b) interruption and reboot of an application program by saving contents of the main and virtual memories on a hard disk, suspending operating system and restoring the saved memory contents when operating system is restarted, (c) interruption and reboot of an application program by suspending and restarting operating system, (d) interruption and reboot of an application program by forcibly interrupting and restarting the program and operating system, (e) restarting of an application program and operating system after reinstalling the program and operating system, and (f) restarting an application program and operating system after performing a clear install of the program and operating system, and wherein each of said processing nodes differs from every other processing nodes in one of said parameters of program startup configuration data.

31. The fault tolerant computing system of claim 25, wherein said distributing means includes a memory for storing distribution data, wherein the distributing means distributes said request messages to the selected processing nodes according to the stored distribution data, further comprising fault recovery means for monitoring the selected processing nodes and updating the stored distribution data if a failure is detected in the selected processing nodes.

32. The fault tolerant computing system of claim 31, wherein said fault recovery means is arranged to alter the configuration of one of the selected processing nodes when a failure is detected in said one processing node and restart the selected processing nodes.

33. The fault tolerant computing system of claim 31, wherein said fault recovery means is arranged to alter the program startup configuration of one of the selected processing nodes when a failure is detected in said one processing node and restart the selected processing nodes.

34. The fault tolerant computing system of claim 31, wherein said fault recovery means is arranged to alter checkpointing data of each of the selected processing nodes when a failure is detected so that the checkpointing data of each of the selected processing nodes differs from the checkpointing data of every other selected processing nodes.

35. The fault tolerant computing system of claim 25, further comprising decision means for receiving said response messages from said processing nodes and selecting one of the response messages.

36. The fault tolerant computing system of claim 35, wherein said decision means selects one of the response messages on a first-served-first-arrived basis.

37. The fault tolerant computing system of claim 35, wherein said decision means makes a majority decision on a plurality of said response messages which arrive in a specified time interval and selects one of the majority-decided response messages.

38. The fault tolerant computing system of claim 35, wherein said decision means makes a majority decision on a predetermined number of said response messages and selects one of the majority-decided response messages.

39. The fault tolerant computing system of claim 35, wherein said decision means calculates an average value from said plurality of response messages and formulates a response message with said average value.

40. The fault tolerant computing system of claim 35, wherein said decision means receives an external request message from an external source via said communication channel and sends the received message to one of said processing nodes.

41. The fault tolerant computing system of claim 35, wherein said decision means comprises:

a memory for storing distribution data;

means for updating the distribution data according to operating status of said processing nodes; and

means for supplying said external request message to said one of the processing nodes according to the distribution data.

42. The fault tolerant computing system of claim 25, wherein each of said processing nodes is one of a real machine.

43. The fault tolerant computing system of claim 25, wherein each of said processing nodes is one of a virtual machine.

44. A method of operating a computing system, wherein the computing system comprises a plurality of processing nodes interconnected by a communication channel, each of said processing nodes having an identical application program, the method comprising the steps of:

respectively setting said processing nodes in uniquely different configurations which are sufficiently different from each other to produce is a difference in processing results respectively produced by said processing nodes when a failure occurs in one of said processing nodes;

operating said processing nodes in parallel to execute the application programs in environments corresponding to said uniquely different configurations; and

respectively distributing request messages to said processing nodes, so that said request messages are individually processed by said application programs to produce a plurality of response messages.

45. A method of operating a computing system, wherein the computing system comprises a plurality of processing nodes interconnected by a communication channel, each of said processing nodes having an identical application program, the method comprising the steps of:

selecting said processing nodes which are respectively set in uniquely different configurations which are sufficiently different from each other to produce a difference in processing results respectively produced by said processing nodes when a failure occurs in one of said processing nodes;

operating the selected processing nodes in parallel to execute the application programs in environments corresponding to said uniquely different configurations; and

distributing means for distributing request messages respectively to the selected processing nodes, so that said request messages are individually processed by the application programs of the selected processing nodes to produce a plurality of response messages.