Fault monitor for restarting failed instances of the fault monitor

• US 6,718,486 B1
• Filed: 07/24/2000
• Issued: 04/06/2004
• Est. Priority Date: 01/26/2000
• Status: Expired due to Fees

• Current Assignee: David E. Lovejoy
• Original Assignee: David E. Lovejoy
• Inventors: Blaser, Rico, Roselli, Drew Shaffer, Lechner, Mikel Carl
• Primary Examiner(s): Beausoliel, Robert
• Assistant Examiner(s): Chu, Gabriel L.
• Application Number:
  US09/624,747
• Time in Patent Office: 1,352 Days
• Field of Search: 714/4, 714/13, 714/31, 714/39, 714/41, 714/47, 714/55
• US Class Current: 714/41
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating

Abstract

A computer system having a fault-tolerance framework in an extendable computer architecture. The computer system is formed of clusters of nodes where each node includes computer hardware and operating system software for executing jobs that implement the services provided by the computer system. Jobs are distributed across the nodes under control of a hierarchical resource management unit. The resource management unit includes hierarchical monitors that monitor and control the allocation of resources. In the resource management unit, a first monitor, at a first level, monitors and allocates elements below the first level. A second monitor, at a second level, monitors and allocates elements at the first level. The framework is extendable from the hierarchy of the first and second levels to higher levels where monitors at higher levels each monitor lower level elements in a hierarchical tree. If a failure occurs down the hierarchy, a higher level monitor restarts an element at a lower level. If a failure occurs up the hierarchy, a lower level monitor restarts an element at a higher level. Each of the monitors includes termination code that causes an element to terminate if duplicate elements have been restarted for the same job. The termination code in one embodiment includes suicide code whereby an element will self-destruct when the element detects that it is an unnecessary duplicate element.

Claims

1. A fault tolerant computer system for executing one or more jobs on one or more nodes, comprising,

a hierarchy of monitors for monitoring operations in the computer system including,

one or more first monitors for monitoring first operations and, for any particular one of said first operations that fails, for restarting another instance of said particular one of said first operations,

one or more second monitors for monitoring said first monitors and, if any particular one of said first monitors fails, for restarting another instance of said particular one of said first monitors.

2. The system of claim 1 wherein,

said one or more of said second monitors are monitored by at least one of said first monitors and, if any particular one of said second monitors fails, said at least, one of said first monitors restarts another instance of said particular one of said second monitors.

3. The system of claim 2 wherein one or more of said second monitors operates to commit suicide if more than one of said another instance of said particular one of said second monitors is restarted.

4. The system of claim 1 wherein,

said nodes operate to execute processes in a service unit, a communication unit and a resource management unit.

5. The system of claim 1 wherein each of said nodes includes a computer having an operating system, wherein pluralities of nodes form clusters and wherein each cluster has a corresponding instantiation of said hierarchy of monitors for monitoring operations in the computer system.

6. The system of claim 5 wherein each instantiation of said hierarchy of monitors includes,

a first instantiation of said one or more first monitors for monitoring first instantiation operations and, for any particular one of said first instantiation operations that fails, for restarting another instance of said particular one of said first instantiation operations,

a second instantiation of said one or more second monitors for monitoring said first monitors of said first instantiation and, if any particular one of said first monitors of said first instantiation fails, for restarting another instance of said particular one of said first monitors of said first instantiation.

7. The system of claim 5 including first and second instantiations and wherein,

said one or more of said second monitors of said second instantiation are monitored by at least one of said first monitors of said first instantiation and, if any particular one of said one or more of said second monitors of said second instantiation fails, for restarting another instance of said particular one of said one or more of said second monitors of said second instantiation.

8. The system of claim 1 wherein,

said second monitors maintain a record of particular ones of the first monitors that are active and corresponding active particular ones of said first operations being monitored by said particular ones of the first monitors.

9. The system of claim 8 wherein,

said second monitors use said record to ensure that active particular ones of said first operations monitored by a failing one of said particular ones of the first monitors that are active is monitored by a new instance of said failing one of said particular ones of the first monitors that are active.

10. The system of claim 1 wherein said hierarchy of monitors includes,

one or more additional monitors for monitoring said first monitors or said second monitors, and, if any particular one of said first monitors or said second monitors fails, restarting another instance of said particular one of said first monitors or said second monitors.

11. The system of claim 10 wherein said hierarchy of monitors includes,

one or more other monitors for monitoring said first monitors, said second monitors or said additional monitors, and, if any particular one of said first monitors, said second monitors or said additional monitors fails, restarting another instance of said particular one of said first monitors, said second monitors or said additional monitors.

12. The system of claim 1 wherein,

said first operations are jobs running on said nodes for providing services and, for any particular one of said jobs that fails, one of said first monitors restarts another instance of said particular one of said jobs.

13. The system of claim 12 wherein said jobs implement e-commerce transaction services.

14. The system of claim 12 wherein said jobs implement transaction services for financial instruments.

15. The system of claim 12 wherein said first monitors are host agents for monitoring operations of a plurality of jobs on a plurality of nodes where each job is monitored by only one of said host agents.

16. The system of claim 12 wherein said first monitors are one or more agents operating on a first level, each of said agents for monitoring operations of jobs on nodes where each job is monitored by only one of said agents.

17. The system of claim 12 wherein,

said first monitors are one or more agents operating on a first level, each of said agents for monitoring operations of jobs on nodes where each job is monitored by only one of said agents, and

said one or more second monitors includes one or more local coordinators operating on a second level where each local coordinator monitors one or more of said agents.

18. The system of claim 12 wherein,

said first monitors are one or more agents operating on a first level, each of said agents for monitoring operations of jobs on nodes where each job is monitored by only one of said agents, and wherein a particular one of said agents runs on a particular one of said nodes where a job monitored by said particular one of said agents runs.

19. The system of claim 12 wherein,

said first monitors are one or more agents operating on a first level, each of said agents for monitoring operations of jobs on nodes where each job is monitored by only one of said agents, and wherein a particular one of said agents runs on a particular one of said nodes where a job monitored by said particular one of said agents runs on other of said nodes than said particular one of said nodes.

20. The system of claim 12 wherein,

said first monitors are one or more agents operating on a first level, each of said agents for monitoring operations of jobs on nodes where each job is monitored by only one of said agents, and wherein a particular one of said agents runs on a particular one of said nodes where a job monitored by said particular one of said agents runs,

said second monitors are one or more local coordinators operating on a second level, each of said local coordinators for monitoring operations of agents, and wherein a particular one of said local coordinators runs on a particular one of said nodes where an agent monitored by said particular one of said local coordinators runs.

21. The system of claim 12 wherein,

said first monitors are one or more agents operating on a first level, each of said agents for monitoring operations of jobs on nodes where each job is monitored by only one of said agents, and wherein a particular one of said agents runs on a particular one of said nodes where a job monitored by said particular one of said agents runs,

said second monitors are one or more local coordinators operating on a second level, each of said local coordinators for monitoring operations of agents, and wherein a particular one of said local coordinators runs on a particular one of said nodes other than where an agent monitored by said particular one of said local coordinators runs.

22. The system of claim 12 wherein,

said first monitors are one or more agents operating on a first level, each of said agents for monitoring operations of jobs on nodes where each job is monitored by only one of said agents,

said second monitors are one or more local coordinators operating on a second level, each of said local coordinators for monitoring operations of agents,

and wherein said hierarchy of monitors includes,

one or more third monitors for monitoring said one or more second monitors and, for any particular one of said second monitors that fails, restarting another instance of said particular one of said second monitors, and wherein a particular one of said third monitors that monitors said particular one of said second monitors runs on a different node than a node where said particular one of said second monitors runs.

23. The system of claim 22 wherein said hierarchy of monitors includes,

one or more fourth monitors for monitoring said one or more third monitors and, for any particular one of said third monitors that fails, restarting another instance of said particular one of said third monitors, and wherein a particular one of said fourth monitors that monitors said particular one of said third monitors runs on a different node than a node where said particular one of said third monitors runs.

24. The system of claim 12 wherein,

said first monitors are one or more agents operating on a first level, each of said agents for monitoring operations of jobs on nodes where each job is monitored by only one of said agents,

said second monitors are one or more local coordinators operating on a second level, each of said local coordinators for monitoring operations of agents,

and wherein said hierarchy of monitors includes,

one or more third monitors for monitoring said one or more second monitors and, for any particular one of said second monitors that fails, restarting another instance of said particular one of said second monitors, and wherein a particular one of said third monitors that monitors said particular one of said second monitors runs on a node where said particular one of said second monitors runs.

25. The system of claim 24 wherein said hierarchy of monitors includes,

one or more fourth monitors for monitoring said one or more third monitors and, for any particular one of said third monitors that fails, restarting another instance of said particular one of said third monitors, and wherein a particular one of said fourth monitors that monitors said particular one of said third monitors runs on a node where said particular one of said third monitors runs.

26. The system of claim 1 wherein said hierarchy of monitors includes,

one or more third monitors for monitoring said one or more second monitors and, for any particular one of said second monitors that fails, restarting another instance of said particular one of said second monitors.

27. The system of claim 26 wherein one or more of said second monitors operates to commit suicide if more than one of said instance of said particular one of said second monitors is restarted.

28. The system of claim 26 wherein said one or more third monitors run on different ones of said nodes than ones of said nodes on which said second monitors run.

29. The system of claim 26 wherein said hierarchy of monitors includes,

one or more fourth monitors for monitoring said one or more third monitors and, for any particular one of said third monitors that fails, restarting another instance of said particular one of said third monitors.

30. The system of claim 29 wherein said one or more fourth monitors run on different ones of said nodes than ones of said nodes on which said third monitors run.

31. The system of claim 29 wherein said one or more fourth monitors run on ones of said nodes which are the same as ones of said nodes on which said third monitors run.

32. The system of claim 29 wherein one or more of said third monitors operates to commit suicide if more than one of said instance of said particular one of said third monitors is restarted.

33. The system of claim 1 having a resource management unit including a load-balancing for distributing jobs among said nodes.

34. The system of claim 1 having a resource management unit including a persistent storage unit.

35. The system of claim 1 having a resource management unit including an interface unit.

36. The system of claim 1 wherein,

each of said nodes includes a plurality of computers each having an operating system.

37. The system of claim 1 having a plurality of clusters of said nodes, each cluster having a corresponding instantiation of said hierarchy of monitors for monitoring operations in the computer system.

38. The system of claim 37 wherein,

each of said clusters of nodes operates to execute processes organized into a service unit, a communication unit and a resource management unit.

39. The system of claim 37 wherein,

said clusters of nodes are organized into groups, each group having one or more of said clusters.

40. The system of claim 39 wherein,

a first one of said groups is located at a geographic location remote from a second one of said groups and said first one of said groups is connected to said second one of said groups by one or more networks.

41. The system of claim 39 wherein,

a first one of said groups is organized to execute on one subset of data and a second one of said groups is organized to execute on another subset of data.

42. The system of claim 37 wherein,

a first one of said groups is organized to execute on one subset of data and a second one of said groups is organized to provide backup for said one subset of data.

43. The system of claim 1 wherein,

said first operations are jobs running on said nodes for providing services,

said first monitor senses one or more conditions that can cause any particular one of said jobs to fail whether or not said particular one of said jobs has actually failed,

one of said first monitors terminates said particular one of said jobs and restarts another instance of said particular one of said jobs.

44. The system of claim 43 wherein,

said one of said first monitors that terminates said particular one of said jobs restarts said another instance of said particular one of said jobs in an environment where said one or more conditions are not present.

45. The system of claim 43 wherein,

said one of said conditions is a node failure and said another instance of said particular one of said jobs is started on a different non-failing node.

46. The system of claim 43 wherein,

said one of said conditions is a job failure and said another instance of said particular one of said jobs is started as a new instance of said job.

47. The system of claim 46 wherein,

said another instance of said particular one of said jobs is started as a new instance of said job on a node the same as a node on which said particular one of said jobs was running.

48. The system of claim 46 wherein,

said another instance of said particular one of said jobs is started as a new instance of said job on a new node different from a node on which said particular one of said jobs was running.

49. The system of claim 1 wherein each of said nodes includes a computer and wherein new ones of said nodes are added to the system without disturbing the operations of other of said nodes in the computer system and wherein jobs are assigned dynamically to said new ones of said nodes.

50. The system of claim 1 wherein each of said nodes includes a computer and wherein ones of said nodes are removed from the system without disturbing the operations of other of said nodes in the computer system and wherein particular jobs are reassigned dynamically to other of said nodes in the computer system.

51. The system of claim 1 wherein each of said nodes includes a computer of one type and wherein new ones of said nodes are added to the system including upgraded computers of a different type without disturbing the operations of other of said nodes in the computer system and wherein jobs are assigned dynamically from said other of said nodes to said new ones of said nodes to provide dynamic upgrade of said system without stopping said particular jobs.

52. The system of claim 1 wherein pluralities of nodes form clusters and wherein particular ones of said clusters are assigned for processing particular jobs at particulars times and wherein other ones of said clusters are assigned for processing said particular jobs at other times.

53. The system of claim 52 wherein said particular times and said other times are follow-the-sun times.

54. The system of claim 1 wherein a delay time is controlled before the restart of a job.

55. The system of claim 1 having an interface that allows humans to monitor the health of the system and to log statistics about uptime of each component in the system.

56. The system of claim 1 wherein a delay time is applied before said restarting another instance of said particular one of said first operations.

57. The system of claim 1 wherein in said hierarchy of monitors,

said one or more of said second monitors are monitored by at least one of said first monitors and, if any particular one of said second monitors fails, said at least one of said first monitors, after a first delay time, restarts another instance of said particular one of said second monitors on a node other than a node on which said particular one of said second monitors failed.

58. The system of claim 57 wherein,

if more than one instance of said another instance of said particular one of said second monitors is restarted, all but one instance of said another instance of said particular one of said second monitors commits suicide.

59. The system of claim 57 wherein said hierarchy of monitors includes,

one or more additional monitors for monitoring said first monitors and said second monitors, and, if any particular one of said first monitors or said second monitors fails, restarting, after a second delay time, another instance of said particular one of said first monitors or said second monitors.

60. The system of claim 59 wherein,

if more than one of instance of said another instance of said particular one of said first monitors or said second monitors is restarted, all but one instance of said another instance of said particular one of said first monitors or said second monitors operates to commit suicide.

61. The system of claim 59 wherein said hierarchy of monitors includes,

one or more other monitors for monitoring said first monitors, said second monitors and said additional monitors, and, if any particular one of said first monitors, said second monitors or said additional monitors fails, restarting, after a third delay time, another instance of said particular one of said first monitors, said second monitors or said additional monitors.

62. The system of claim 61 wherein,

if more than one instance of said another instance of said particular one of said first monitors, said second monitors or said additional monitors is restarted, all but one instance of said another instance of said particular one of said first monitors, said second monitors or said additional monitors operates to commit suicide.

63. The system of claim 1 wherein,

said first operations are jobs running on said nodes for providing services where a particular first one of said jobs associated with a first customer is running on a particular first node and a particular second one of said jobs associated with a second customer is running on said particular first node.

64. The system of claim 1 wherein,

said first operations are jobs running on said nodes for providing services where a particular first one of said jobs associated with a first customer is running on a particular first node and a particular second one of said jobs associated with a second customer is running on a particular second node whereby said first customer job is isolated from said second customer job.

65. The system of claim 1 wherein,

said first operations are jobs running on said nodes for providing services where,

particular first ones of said jobs are associated with a first customer with one of said particular first ones of said jobs running on a particular first node and with another one of said particular first ones of said jobs running on a particular other node;

particular second ones of said jobs are associated with a second customer with one of said particular second ones of said jobs running on a particular second node and with another one of said particular second ones of said jobs running on said particular other node.

66. The system of claim 1 including transaction initiators for starting said first operations as one or more jobs to initiate a transaction in a service.

67. The system of claim 1 including transaction processors for starting said first operations as one or more jobs to process a transaction in a service.

68. The system of claim 1 including,

transaction initiators for starting first ones or more of said first operations as one or more first jobs on a first node to initiate a transaction in a service;

transaction processors for starting other ones or more of said first operations as one or more other jobs on another node to process said transaction in said service.

69. The system of claim 1 including,

transaction initiators for starting first ones or more of said first operations as one or more first jobs on a first node to initiate a transaction in a service;

transaction processors for starting other ones or more of said first operations as one or more other jobs on said first node to process said transaction in said service.

70. In a fault tolerant computer system operating to execute one or more jobs on one or more nodes where the computer system includes a hierarchy of monitors for monitoring operations in the computer system, the method comprising,

monitoring first operations with one or more first monitors and, for any particular one of said first operations that fails, restarting another instance of said particular one of said first operations,

monitoring said first monitors with one or more second monitors and, if any particular one of said first monitors fails, restarting another instance of said particular one of said first monitors.

71. The method of claim 70 wherein,

monitoring said one or more of said second monitors with at least one of said first monitors and, if any particular one of said second monitors fails, restarting with said at least one of said first monitors another instance of said particular one of said second monitors.

72. The method of claim 71 wherein one or more of said second monitors operates to commit suicide if more than one of said another instance of said particular one of said second monitors is restarted.