Fault-tolerant multi-computer system

• US 4,356,546 A
• Filed: 02/05/1980
• Issued: 10/26/1982
• Est. Priority Date: 02/05/1980
• Status: Expired due to Term

• Current Assignee: The Bendix Corporation (Honeywell International Inc.)
• Original Assignee: The Bendix Corporation (Honeywell International Inc.)
• Inventors: Whiteside, Arliss E., Freedman, Morris D., Tasar, Omur, Rothschild, Alexander M.
• Primary Examiner(s): Nusbaum, Mark E.
• Assistant Examiner(s): Heckler, Thomas M.
• Application Number:
  US06/118,692
• Time in Patent Office: 994 Days
• Field of Search: 364/200 MS File, 371/9, 371/11, 371/67, 371/36
• US Class Current: 714/10
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: G06F 11/0784 Routing of error reports, e.g. with a specific transmission path or data flow
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/1683 at instruction level
• CPC Class Codes: G06F 11/1691 using a quantum
• CPC Class Codes: G06F 11/182 based on mutual exchange of the output between redundant processing components
• CPC Class Codes: G06F 11/188 where exact match is not required
• CPC Class Codes: G06F 15/173 using an interconnection network, e.g. matrix, shuffle, pyramid, star, snowflake

Abstract

A Fault-Tolerant Multi-Computer System for control applications is disclosed. The system has a plurality of Computers (10a-10n), each having an assigned set of tasks which it is capable of executing. No one Computer in the system acts as a master and no one Computer executes all of the tasks. Communication between the Computers is by individual communication links (16, 18, 20) over which each Computer sends information directly to all other Computers in the system. Each Computer comprises an Applications Computer (100) and an Operations Controller (200). The Operations Controller receives messages over the communication links and selects, from the assigned tasks, the tasks to be performed by the associated Applications Computer. Each Operations Controller includes a fault handler which checks the messages received from the other Computers. The fault handlers send and receive error messages, over the communication links, to assist in the identification of a faulty Computer. Subsequent messages from the Computers deemed to be faulty are ignored, and the tasks assigned to the faulty Computer are executed by alternate Computers in the system.

Claims

1. A fault-tolerant multi-computer system architecture, responsive to intercomputer messages and to inputs from external sources for executing a predetermined set of tasks to produce an output to at least one external device, comprising:

a plurality of computers for collectively executing the predetermined set of tasks in a coordinated manner to produce outputs to the at least one external device in response to the inputs from the external sources and the intercomputer messages, each of said computers having an assigned subset of the tasks which it is capable of selecting and executing in a predetermined order of priority, each task in said predetermined set of tasks being included in more than one of said assigned subset so that each task is capable of being selected and executed by more than one computer; and

a like plurality of communication links, one associated with each computer, each communication link transmitting only the intercomputer messages sent by the associated computer to all of the computers which require any message generated by the associated computer; and

wherein each of said plurality of computers comprises;

operations controller means for controlling the operation of its own computer in coordination with like operations controllers in the other computers, each operations controller including;

receiver means for receiving intercomputer messages, fault handler means for checking said intercomputer messages to detect the faulty operation of any computer in the system, and to exclude from further processing the messages received from faulty computers, scheduler means responsive to the receipt of all the data variables for the execution of at least one of its assigned tasks for selecting from its assigned subset the tasks to be executed, task communicator means responsive to the task selected by the scheduler means for assembling the data variables required for the execution of the selected task and transmitter means responsive to said fault handler means, said scheduler means and said task communicator means for sending intercomputer messages to all of the computers in the system, said messages containing an identification of the faulty computers, identification of the tasks it has selected, and the values of the data variables resulting from the execution of the selected tasks required for the execution of a subsequent task; and

applications computer means for executing the tasks selected by said scheduler means using the data assembled by said task communicator means.

2. The fault-tolerant multi-computer system of claim 1 further including combiner/voter means for combining the corresponding outputs from said plurality of computers to produce a single combined output, and to produce a voted output indicative of the output of the majority of computers when more than two computers execute the tasks generating said output to the at least one external device.

3. The fault-tolerant multi-computer system of claim 1 wherein each of said applications computer means comprises:

program memory means for storing a set of instructions for the execution of each task in said assigned subset of tasks;

central processing means, responsive to the task selected by said scheduler means, for executing the set of instructions stored in said program memory means corresponding to the selected task using values of said data variables assembled by said task communicator means, said central processing means generating values for said data variables through the execution of each task;

data storage means for storing intermediate variable and fixed data values as required by said sets of instructions for each task in said assigned subset of tasks; and

input/output means for transferring to said central processing means the inputs from the external sources required for the execution of the selected tasks and for outputting to the at least one external device the data values resulting from the executed tasks, when instructed by the central processor means.

4. The fault-tolerant multi-computer system of claim 1 wherein said receiver means comprises a plurality of receivers, each receiver connected to one communication link and each receiver receiving only the messages sent by the computer associated with the communication link.

5. The fault-tolerant multi-computer system of claim 4 wherein said receiver means further includes a receiver connected to the communication link associated with its own computer and receiving only the messages sent by its own computer.

6. The fault-tolerant multi-computer system of claim 1 wherein said fault handler means includes:

checker means for checking each message received by said receiver means and for generating an error signal identifying each received message containing an error and the computer which sent the message; and

fault tolerator means responsive to the error signals generated by said checker means and messages received from other computers identifying which computers are considered to be faulty, for passing on for further processing only error free messages received from non-faulty computers, said fault tolerator means further including means responsive to said error signal for enabling said transmitter means to send messages identifying the computers deemed to be faulty.

7. The fault-tolerant multi-computer system of claim 1 wherein said fault tolerator means further includes means for cancelling the fault status of a computer previously identified as faulty when said computer sends error free messages for a predetermined period of time.

8. The fault-tolerant multi-computer system of claim 1 wherein said scheduler means comprises:

status table means for storing current status information for each task capable of being executed by the computer, said status table means storing said tasks in their order of execution priority;

means for recording in said status table means the tasks selected, and the values of the data variables contained in messages received from all the computers in the system, said means for recording further including means for generating a dispatch signal when all the data required for the execution of a task has been recorded as received;

system status monitor means responsive to said messages identifying which computers are considered to be faulty for recording in said status table means as unselected each task previously recorded as being selected by the computer identified as faulty, said system status monitor means further including means for generating a dispatch signal in response to the unselecting of a task in said status table;

scheduling status table means for storing the tasks selected in the order in which they are selected; and

task selector means responsive to said dispatch signals for selecting from the status table means the highest priority task ready for execution, said task selector means further including means for recording said selected task in said scheduling status table means, means for sending a dispatch task signal identifying the selected task to said task communicator means and means for enabling said transmitter means to send a message to all of the computers identifying the task selected.

9. The fault-tolerant multi-computer system of claim 6 wherein each of said messages has a predetermined format containing information identifying the message type and the computer than sent the message, said checker means includes a message format checker for checking the format of each received message to generate said error signal when an error in the message format is detected.

10. The fault-tolerant multi-computer system of claim 9 wherein each of said operations controller means further sends started/completed messages identify the tasks completed and the new task started by its applications computer means, said checker means further includes execution time checker means responsive to said completed/started message for checking the execution time of each task executed by each applications computer means in the system to generate said error signal when the execution time exceeds predetermined limits.

11. The fault-tolerant multi-computer system of claim 10 wherein said intercomputer messages is a task data value message containing the value of a data variable, said checker means further includes reasonable limits checker means for checking the value of the data variable contained in each task data value message against predetermined maximum and minimum values to generate said error signal when the value of the data variable in the received task data value message is outside said predetermined maximum and minimum values.

12. The fault-tolerant multi-computer of claim 10 wherein said intercomputer messages include a redundant data value message containing a value for data variable redundantly computed by more than two computers, said checker means further includes redundant value voter means responsive to the redundant data value messages received from said more than two computers for finding a voted data value when the value of the data variable contained in a predetermined number of redundant data value messages agree to generate said error signal identifying each computer which sent a redundant data value message containing a value which did not agree with the voted data value.

13. The fault-tolerant multi-computer system of claim 10 wherein said intercomputer messages include a task selected message containing an identification of the task selected by the computer sending the message, said checker means further includes message sequence checker means responsive to task selected and completed/started messages for checking the proper sequence in which the tasks are selected and executed by each computer to generate said error signal when the selection and execution of tasks does not follow a correct sequence.

14. The fault-tolerant multi-computer system of claim 13 wherein each computer is assigned a priority for the execution of tasks and wherein said scheduler further includes means for unselecting a previously selected task selected by a higher priority computer and selecting a new task in response to a message received from a computer having a higher priority identifying that the higher priority computer has selected the same task and means for sending a task unselected/selected message identifying the task unselected and the new task selected, said message sequence checker is responsive to said task unselected/selected message and said completed/started message.

15. The fault-tolerant multi-computer system of claim 1 wherein said fault handler means further includes synchronizer means responsive to intercomputer messages for synchronizing the operation of its own computer with all of the other computers in the system.

16. The fault-tolerant multi-computer system of claim 14 wherein said fault handler further includes synchronizer means responsive to intercomputer messages for synchronizing the operation of its own computer with all of the other computers in the system.

17. The fault-tolerant multi-computer system of claims 15 or 16 wherein said synchronizer means comprises:

timer means for repetitively generating sampling periods and a sampling number indicative of the current sampling period;

means for enabling said transmitter means to send sampling number messages at the end of each sampling period containing said sampling number indicative of said current sampling period;

voter means for finding a voted sampling number indicative of a sampling number contained in a predetermined number of sampling number messages received from like synchronizer means in the other computers;

means responsive to the finding of a voted sampling number for correcting said generated sampling number to agree with said voted sampling number; and

means responsive to time within each sampling period when said voted sampling number is found for correcting the duration of the sampling period currently being generated to cause the end of the current sampling period to coincide with the end of the sampling periods in said like synchronizers in the other computers.

18. The fault-tolerant multi-computer system of claim 17 wherein said synchronizer further includes means response to the sampling numbers contained in the sampling number messages received from the other computers for generating said error signal identifying each computer which sent a sampling number message containing a sampling number which disagrees with said voted sampling number.

19. The fault-tolerant multi-computer system of claim 8 wherein said applications computer generates a task done signal each time it completes the execution of a task, said scheduler means further includes task releaser means for sending a release task signal to the task communicator means in response to said task done signal, said release task signal identifying the selected task stored in said scheduling status table means.

20. The fault-tolerant multi-computer system of claim 17 wherein said fault tolerator means further includes means for cancelling the fault status of a computer previously identified as faulty when said computer sends error free messages for a predetermined period of time.

21. The fault-tolerant multi-computer system of claim 19 wherein each task communicator means includes means for enabling said transmitter means to send a task completed-started message identifying the task completed and the new task started by its applications computer each time the execution of a task is completed and a new task started, said scheduler means further includes completed task recorder means for recording in said status table means the completed status of the tasks identified in said task completed/started messages.

22. The fault-tolerant multi-computer system of claim 1 wherein said task communicator means comprises:

a data values table storing with respect to each data variable the values of that data variable;

store data value means for recording in said data values table the values of the data variables contained in the error-free messages received from non-faulty computers passed on by said fault handler means;

task input means for communicating from said store data values table to said applications computer means the values of the data variables required for the execution of the task selected by said scheduler means;

task results sender means for enabling said transmitter to send data value messages to all of the computers containing the values of the data variables resulting from the task executed by the applications computer means; and

task output means for communicating from said applications computer means to said task results sender means the values of the data variables resulting from the execution of each task.

23. The fault-tolerant multi-computer system of claim 22 wherein said task input means comprises:

a task input table for storing the values of the data variables required for the execution of tasks;

task dispatcher means responsive to the task selected by said scheduler means for recording in said task input table the data values stored in said data values table required for execution of the selected task; and

task releaser means responsive to said applications computer means completing the execution of a preceeding task for communicating from said task input table to said applications computer means the values of the data variables stored in said task input table.

24. The fault-tolerant multi-computer system of claim 16 wherein each task communicator means includes means for enabling said transmitter means to send a task completed/started message identifying the task completed and the new task started by its applications computer means, said scheduler means comprises:

status table means for storing current status information for each task capable of being executed by said applications computer, said status table means storing said tasks in their order of priority;

record data value means receiving said data value messages for recording in said status table that the data identified in the messages is currently available for execution, said record data value means further including means for generating a dispatch signal when all the data required for the execution of a task has been recorded as received;

completed task recorder means responsive to said task completed/started messages for recording in said status table means that the task identified as completed in the received completed/started message has been completed;

25. The fault-tolerant multi-computer system of claim 24 wherein said applications computer means generates a task done signal each time it completes the execution of a task, said scheduler means further includes task releaser means for sending a release task signal to the task communicator means in response to said task done signal, said release task message identifying the task stored in said scheduling status table means which has been selected for execution.

26. The fault-tolerant multi-computer system of claim 25 wherein said system has a normal mode of operation, and at least one degraded mode of operation, said system status monitor means further generates a mode signal indicative of the system'"'"'s current mode of operation determined from the number of computers considered to be faulty and wherein said status table means comprises a plurality of task status tables, each task status table corresponding to one of said modes of operation, said mode signal identifying for said task selector means the task status table from which the tasks are to be selected.

27. The fault-tolerant multi-computer system of claims 22 or 23 wherein said task output means comprises a task output table storing the value of the data variables resulting from the task currently being executed;

and

wherein said task results message sender means enables said transmitter to send said messages containing the values of the data variables stored in said task output table after the execution of the current task is completed.

28. The fault-tolerant multi-computer system of claim 25 wherein said task communicator means comprises:

a data values table storing the values of the data variables;

store data values means for recording in said data values table the value of the data variables contained in the messages received from all the computers;

task input means for communicating from data values table to said applications computer means the values of the data variables required for the execution of the task identified in the release task signal;

task results sender means responsive to said release task signal for enabling said transmitter means to send to all of the computers data value messages containing the values of the data variables resulting from the execution of the task completed by said applications computer means; and

task output means for communication from said applications computer to said task results sender means the values of the data variables resulting from the execution of each task.

29. The fault-tolerant multi-computer system of claim 28 wherein said tank input means comprises:

a task input table for storing the values of the data variables required for the execution of at least two tasks;

task dispatcher means responsive to said release task signal for recording in said task input table the values of the data variables stored in said data values table required for the execution of the identified task; and

task releaser means responsive to said release task signal for communicating to said applications computer means the values of the data variables stored in said task input table recorded in response to the preceeding release task signal.

30. The fault-tolerant multi-computer system of claim 29 wherein said task output means comprises a task output table storing the values of the data variables resulting from the task currently being executed by the applications computer means;

and

wherein said task results message sender means enables said transmitter to send said data value messages containing the values stored in said task output table in response to said release task signal.

31. The fault-tolerant multi-computer system of claims 22 or 28 wherein said task communicator means further includes watch-dog timer means responsive to said applications computer completing the execution of each task for monitoring the execution time of each task executed by said applications computer to generate an error signal when the execution time for a task is not within predetermined limits.

32. The fault-tolerant multi-computer system of claims 22 or 28 wherein said messages containing values of the data variables further contain sequence numbers indicative of the sequence in which the values were generated to distinguish data values resulting from the execution of the same task at different times and using different values of the data variables, said data values table stores values of each data variable for a predetermined number of different sequence numbers, said task dispatcher means includes means for computing the sequence numbers of the data variables required for the execution of the selected task to communicate only the values of the data variables to the applications computer means having the computed sequence numbers, and wherein said messages containing values of the data variables sent by said task results message sender means further include a sequence number corresponding to the execution number of the executed task.

33. A method for controlling the operation of each computer in a fault tolerant multiple computer system wherein the system includes a communication network whereby each computer can send error, data value, task selection and task completed/started messages to every other computer in the system and each computer has an operations controller and applications computer and wherein each computer further has a set of tasks it is capable of selecting and executing, comprising the steps of:

checking for error with said operations controller all messages received by each computer from all the computers in system to generate an error signal identifying each computer which sent the message containing an error;

sending error messages to all of the other computers in response to said error signals identifying as a faulty computer each computer which sent a message containing an error;

recording in said operations controller as faulty each computer which sent a message containing an error in response to said error signals and each computer identified in error messages received from a predetermined number of other computers to generate a fault status table;

discarding all messages containing errors and messages received from computers recorded as faulty in said fault status table;

recording in a status table contained within said operations controller the status information contained in the task data value, task selection and task completed/started messages which were not discarded, said status table listing said tasks and their associated status information in their order of execution priority;

detecting when all the information required for the execution of any task is recorded in said status table to generate an dispatch signal;

recording as unselected in said status table, each task which was previously recorded as selected by a computer which has subsequently been recorded as faulty in said fault status table;

generating by said operations controller a dispatch signal to signify the tasks unselected are ready for execution;

selecting by said operations controller from said status table the highest priority task ready for execution and not selected by another computer in response to said dispatch signal;

recording in said operations controller the selected task as selected in said task status table;

sending to all of the computers a task selected message containing the identity of the task selected;

generating in said operations controller a release task signal containing the identify of the selected task in response to the computer signifying it has completed the execution of a preceeding task;

recording in a data values table contained within said operations controller the value of the data variable contained in each non discarded data value message received from all the computers;

communicating from said data values table to the applications computer the value of the data variables required for the execution of the selected task in response to said release task signal;

executing in the applications computer the selected task using the communicated values of the data variables to generate values for new data variables;

sending to all of the computers by said operations controller, data value messages containing the values of the new data variables received from the applications computer; and

sending to all of the computers by said operations controller a task completed/started message identifying the computer, the task completed, and the new task started by the identified computer after the execution of the task is completed.

34. The method of claim 33 wherein said messages sent between the computers include sampling number messages containing a sampling number identifying the current sampling period of that computer, said method further includes the steps of:

sequentially generating with a sampling period timer sampling periods having a predetermined time duration;

storing a current sampling number identifying the current sampling period;

comparing the sampling numbers contained in the sampling number messages received from all the computers to find a voted sampling number having a value which is the same as the value of the sampling numbers contained in a predetermined number of sampling number messages;

recording said voted sampling number as said current sampling number;

comparing the time remaining in the current sampling period, when said voted sampling number is found, with predetermined maximum and minimum values to determine if the sampling period timer is synchronized with like sampling period timers in the computer which sent the sampling number messages from which the voted sampling number was found;

correcting the remaining time in the sampling period timer when the current remaining time is outside said maximum and minimum values to synchronize the sampling period timer with like sampling period timers in the other computers;

recording in said status table as ready predetermined tasks to be executed once during each sampling period in response to the finding of said voted sampling number;

incrementing by one the stored current sampling number and restarting the sampling period timer at the end of each sampling period;

sending sampling number messages containing the current sampling number incremented by one;

to all of the other computers at the end of each sampling period in which a voted sampling number is obtained;

generating a restart signal at the end of each sampling period in which no voted sampling number is obtained; and

initializing the current sampling number to a predetermined number in response to said restart message.

35. The method of claim 34 wherein said method further includes the steps of:

comparing the voted sampling number with the sampling numbers contained in the sampling number messages received from the other computers to generate an error signal identifying each computer which sent a message containing a sampling number which disagrees with the voted sampling number;

recording in the fault status table as faulty in response to said error signals each computer which sent a sampling number message containing a sampling number which disagreed with the voted sampling number; and

sending error messages to all the computers in response to said error signals identifying each computer which sent a sampling number message containing a sampling number which disagreed with said voted sampling number.

36. The method of claim 34 wherein the computer is assigned at least one predetermined input task which samples the input data being received from at least one external source or at least one output task which outputs data to an external device, and wherein said predetermined input and output tasks are to be executed once during each sampling period said method further includes the step of generating an initiate input/output task signal recording said predetermined input and output tasks as ready in said status table.

37. The method of claim 33 wherein the computer is assigned at least one start-up task which performs predetermined functions required to facilitate the starting of the computer system when the whole system is being started or when the particular computer is being restarted said method further includes the step of detecting the first finding of a voted sampling number following the start up of the computer or the generation of a restart signal to generate an initiate start-up task signal recording said predetermined start-up task as ready in said status table.

38. The method of claim 33 wherein each computer has a plurality of receivers and each receiver can only receive messages from one associated computer, and wherein each message has a predetermined format which includes a message type code identifying the message type and a computer identification code identifying the computer which sent the message, said step of checking includes the step of checking the format of each received message, to generate an error signal when the message type code contained in the message is not a valid message type code and to generate said error signals when the computer associated with the receiver which received the message is not the same as the computer identified by the computer identification code contained in the message.

39. The method of claim 33 wherein said message type includes task data value messages said step of checking includes the step of comparing the value of the data variable contained in each task data value message with at least one predetermined limit value to generate said error signal when the value of the data variable contained in the message does not have a predetermined relationship to said predetermined limit value.

40. The method of claim 39 wherein said step of comparing compares the value of the data variable contained in each task data value message with predetermined maximum and minimum values to generate said error signal when the value of the data variable contained in the message is outside said predetermined maximum and minimum values.

41. The method of claim 33 wherein said messages include redundant data value messages containing values for data variables redundantly computed by more than two computers, said step of checking includes the steps of:

comparing the values contained in said redundant data value messages with the values received from other computers for the same data variable to find a voted data value when the values contained in a predetermined number redundant data value messages agree; and

comparing the voted data value with the value received in each redundant data value message containing a value for the same data variable, to generate said error signal identifying each computer which sent a message containing a value for the same redundantly computed data variable which disagrees with the voted data value.

42. The method of claim 33 wherein said step of checking includes the steps of:

monitoring the time required by each computer to execute the tasks identified in sequentially received task completed/started messages from the same computer; and

comparing the monitored execution time with at least one predetermined time to generate said error signal when the execution time exceeds said predetermine time.

43. The method of claim 42 wherein said messages include sampling number messages containing a sampling number identifying the current sampling period of the computer which sent the message, said method further includes the step of:

sequentially generating sampling periods having a predetermined time duration;

storing a current sampling number identifying the current sampling period;

comparing the sampling numbers contained in the sampling number messages received from all the computers to find a voted sampling number having a value which is the same as the value of the sampling numbers contained in a predetermined number of sampling number messages received from other computers;

recording said voted sampling number as said current sampling number;

comparing the time remaining in the current sampling period, when said voted sampling number is found, with predetermined maximum and minimum values to determine if the sampling period is synchronized with like sampling periods in the computers which sent the sampling number messages from which the voted sampling number was found;

correcting the remaining time in the sampling period when the current remaining time is outside said maximum and minimum values to synchronize the sampling period with like sampling periods in the other computers;

recording in said status table as ready predetermined tasks which must be executed in approximate synchronization with the other computers in the system in response to finding said voted sampling number;

comparing the voted sampling number with the sampling numbers contained in the sampling number message received from the other computers to generate said error signals identifying each computer which sent a sampling number message containing a sampling number which disagrees with the voted sampling number;

recording in the fault status table as faulty in response to said error signals each computer which sent a sampling number message containing a sampling number which disagreed with the voted sampling number;

sending error messages to all of the computers in response to said error signals identifying each computer which sent a sampling number message which disagreed with the voted sampling number;

incrementing by one the stored current sampling number and starting the sampling period at the end of each sampling period;

sending sampling number messages containing the current sampling number incremented by one to all of the other computers at the end of each sampling period in which a voted sampling number is obtained;

generating a restart signal at the end of each sampling period in which no voted sampling number is obtained; and

initializing the current sampling number to a predetermined number, in response to said restart message.

44. The method of claims 33 or 43 wherein said step of recording faulty computers in the fault status table further includes the step of storing for each computer recorded as faulty an elapsed time indicative of the time since the computer was last detected to be faulty;

and

recording in the fault status table as no longer faulty each computer previously recorded as faulty whose elapsed time exceeds a predetermined time.

45. The method of claim 33 wherein said step of checking includes the step of checking the sequence in which the tasks are selected, started, and completed by each computer as identified in sequentially received task selected and task completed/started messages to generate said error signal identifying each computer whose scheduling sequence does not follow a correct sequence.

46. The method of claim 43 wherein the data variables required for the execution of any task may be generated at different times, and several values for the same data variable may be generated before the task is executed requiring an earlier generated value for that data variable, and wherein said step of sending task data value messages further includes sending a task data value message containing the value of the data variable and a sequence number indicative of the sequential order in which the value of the data variable was generated, and wherein said status table includes a plurality of entries for each task and each entry for the same task further storing a different execution number for that task, said step of recording in said status table includes the steps of:

identifying each task which requires the value of the data variables contained in each message;

computing an execution number for each identified task from the sequence number contained in the message;

searching the status table to find an entry for the identified task having the same execution number as the computed execution number;

recording in the entry for the identified task having the computed execution number that the value of the data variable contained in the received message has been received when the computed execution number is found;

searching when the computed execution number is not found the entries for the identified task to find the oldest execution number;

recording the computed execution number in the entry having the oldest execution number when the oldest execution number found is older than the computed execution number; and

recording that the value of the data variable contained in the message has been received in the entry in which the computed execution number is recorded; and

repeating the above steps for each identified task until the reception of the value of the data variable is recorded for every task which requires the value of the data variable contained in the data value message;

and wherein said data values table includes a plurality of entries for each data variable and each entry stores the value of the data variable and the sequence number contained in received data value messages, said step of recording in said data values table includes the steps of;

searching said data values table to find an entry for the data variable identified in the received message having the same sequence number as the sequence number contained in the received message;

recording the value of the data variable contained in the message in the data variable entry having the same sequence number when a data variable entry having the same sequence number is found;

searching when the sequence number contained in the message is not found the entries for the data variable identified in the message to find the oldest sequence number;

recording the sequence number contained in the message in the entry having the oldest sequence number when the sequence number contained in the message is newer than the oldest sequence number; and

recording the value of the data variable contained in the message in the entry in which the newer sequence number was recorded;

said step of communicating the values of the data variables to said applications computer comprising the steps of;

identifying from the task identified in said release task signal the data variables required for the execution of that task;

computing from the execution number of the identified task the sequence number of each data variable required for the execution of the identified task;

communicating the value of the data variable from the entry in the data values table having the computed sequence number to said applications computer;

repeating the above steps until the value of each identified data variable is communicated to the applications computer.

47. The method of claim 46 wherein said step of communicating the value of the data variable having the computed sequence number comprises the steps of:

copying the value of the data variable from the entry in the data values table having the computed sequence number into a task input table storing the values of data variables for at least two tasks in response to said release task signal; and

releasing to the applications computer the values of the data variables copied into the task input table by the preceeding release task signal in response to said release task signal;

wherein while the values of the data variables required for the execution of the task identified in the release task signal are being copied into the task input table the values of the data variables for the preceeding task simultaneously being released to said applications computer.

48. The method of claim 33 wherein each computer is assigned a priority for the execution of its assigned tasks said step of recording in said status table further includes the steps of:

comparing the task identified as selected in each task selected message with task selected but not started by the computer receiving the message;

comparing the priority of the computer which sent in the task selected message with the priority of the computer which received the message when both computers have selected the same task to identify which computer has the higher priority;

recording in said status table that the selected task is unselected by the receiving computer when the computer which sent the message has a higher priority than the computer receiving the message; and

selecting from said status table the highest priority task ready for execution and not selected by any computer, when the computer which sent the message has a higher priority than the computer receiving the message;

and wherein said step of sending a task selected message sends in response to the unselection of a task and the selection of a new task a task unselected/selected message further identifying both the task unselected and the new task selected by that computer.

49. The method of claim 47 wherein said step of copying the values of data variables in to the task input table further includes the step of copying into said task input table, from a private table storing the starting address in the applications computer'"'"'s program memory where each tasks begins, the starting address for the task identified in said release task signal to generate a task input message containing both the starting address of the identified task and the values of the data variables;

and

wherein said step of releasing releases said task input message to said applications computer.

50. The method of claim 33 or 49 wherein the system has a normal mode of operation, and at least one degraded mode of operation said method further includes the step of computing from the number of computers recorded as faulty in said fault status table a mode signal indicative of the system current mode of operation;

and wherein said status table comprises a plurality of task status tables, each task status table corresponding to one of said modes of operation, said step of selecting from said status table is further responsive to said mode signal and selects said tasks from the task status table corresponding to the mode of operation identified by said mode signal.

51. The method of claim 50 further including the step of monitoring the execution time of each task executed by its own applications computer to generate an error signal when the execution time for a task is not within a predetermined time limit.

52. The method of claim 51 wherein predetermined data variables are redundantly computed by more than two tasks executed by more than two computers, said step of sending data value messages sends two different types of data value messages, task data messages containing the values of the data variables singularly computed, and redundant data value messages containing the values of the data variables which are redundantly computed.