Method and apparatus for providing scaleable levels of application availability

• US 6,058,490 A
• Filed: 04/21/1998
• Issued: 05/02/2000
• Est. Priority Date: 04/21/1998
• Status: Expired due to Term

• Current Assignee: WSOU Investments, LLC (WSOU Holdings, LLC)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Pokropinski, John H., Lee, Susan A., Haddad, Debra K., Allen, Reginald L., Smith, Raymond D., Schoonover, James P., Rathunde, Dale F., Prokopowicz, Bonnie L.
• Primary Examiner(s): WRIGHT, NORMAN M
• Application Number:
  US09/063,560
• Time in Patent Office: 742 Days
• Field of Search: 714/9, 714/7, 714/8, 714/5, 714/13
• US Class Current: 714/5.11
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2035 without idle spare hardware

Abstract

A method and an apparatus for providing scalable layers of highly available applications using loosely coupled commercially available computers. The software running on the loosely coupled computers is divided into three layers: the system layer, the platform layer, and the application layer, each having its own process group activation and fault recovery strategy. A process group contains software processes that depend upon a set of resources common to the process group. In addition to depending upon a common set of resources, processes within a process group share a fault recovery strategy. Fault recovery is performed at the process group level, such that if one process within a process group fails, fault recovery is takes place for all processes within the process group. In the preferred embodiment, an application layer process group may be paired with another application layer process group on a separate computer. As part of certain escalated process group fault recovery strategies, upon taking an application layer process group out of service, its paired application layer process group, if any exists, takes over performing the functions of the process group that was taken out of service.

Claims

1. A method for providing high availability applications comprising, in combination, the steps of:

a. running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. providing, on at least a first of said at least two computers, a system layer having at least one process group;

d. taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

e. re-booting said first computer upon a system layer process group fault occurring on said first computer;

f. providing at least one paired process group on a second of said at least two computers, said paired process group being paired with one of said one or more process groups on said first computer; and

g. activating said paired process group on said second computer upon a system layer process group fault occurring on said first computer.

2. The method of claim 1 further comprising, in combination, the step of:

using an independent computer to power cycle said first computer upon failure of said re-booting of said first computer to cure said system layer process group fault.

3. The method of claim 2 further comprising, in combination, the step of:

using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said system layer process group fault.

4. A method for providing high availability applications comprising, in combination, the steps of:

a. running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. providing, on at least a first of said at least two computers, a system layer having at least one process group;

d. taking all of said process groups out of service on said first computer upon a fault in a resource depended upon by at least one of said system layer process groups on said first computer;

e. re-booting said first computer upon said fault in said resource depended upon by at least one of said system layer process groups on said first computer;

f. providing at least one paired process group on a second of said at least two computers, said paired process group being paired with one of said process groups on said first computer; and

g. activating said paired process group on said second computer upon said fault in said resource depended upon by at least one of said system layer process groups on said first computer.

5. The method of claim 4 further comprising, in combination, the step of:

using an independent computer to power cycle said first computer upon failure of said re-booting of said first computer to cure said fault in said resource depended upon by at least one of said system layer process groups on said first computer.

6. The method of claim 5 further comprising, in combination, the step of:

using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said fault in said resource depended upon by at least one of said system layer process groups on said first computer.

7. A method for providing high availability applications comprising, in combination, the steps of:

a. running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. providing, on at least a first of said at least two computers, a system layer having at least one process group;

d. taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

e. re-booting said first computer upon a system layer process group fault occurring on said first computer;

f. providing, on at least said first computer, a platform layer having at least one process group;

g. taking all of said process groups, except each of said at least one process group in said system layer, out of service on said first computer upon a platform layer process group fault occurring on said first computer;

h. providing at least one paired process group on a second of said at least two computers, said paired process group being paired with one of said process groups on said first computer; and

i. activating said paired process group on said second computer upon said platform layer process group fault occurring on said first computer.

8. The method of claim 7 further comprising, in combination, the step of:

re-initializing all of said process groups, except each of said at least one process group in said system layer, on said first computer upon said platform layer process group fault occurring on said first computer.

9. The method of claim 8 further comprising, in combination, the step of:

re-booting said first computer upon failure of said re-initialization to cure said platform layer process group fault.

10. The method of claim 9 further comprising, in combination, the step of:

using an independent computer to power cycle said first computer upon failure of said re-booting of said first computer to cure said platform layer process group fault.

11. The method of claim 10 further comprising, in combination, the step of:

using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said platform layer process group fault.

12. A method for providing high availability applications comprising, in combination, the steps of:

a. running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. providing, on at least a first of said at least two computers, a system layer having at least one process group;

d. taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

e. re-booting said first computer upon a system layer process group fault occurring on said first computer;

f. providing, on at least a first of said at least two computers, a platform layer having at least one process group;

g. taking all of said process groups, except each of said at least one process group in said system layer, out of service on said first computer upon a fault in a resource depended upon by at least one of said platform layer process groups on said first computer;

h. providing at least one paired process group on a second of said at least two computers, said at least one paired process group being paired with one of said process groups on said first computer; and

i. activating said at least one paired process group on said second computer upon said fault in said resource depended upon by at least one of said platform layer process groups on said first computer.

13. The method of claim 12 further comprising, in combination, the steps of:

a. re-initializing said resource having said fault; and

b. re-initializing all of said process groups, except each of said at least one process group in said system layer, on said first computer upon said fault in said resource depended upon by at least one of said platform layer process groups on said first computer.

14. The method of claim 13 further comprising, in combination, the step of:

re-booting said first computer upon failure of said re-initializations to cure said fault in said resource depended upon by at least one of said platform layer process groups on said first computer.

15. The method of claim 14 further comprising, in combination, the step of:

using an independent computer to power cycle said first computer upon failure of said re-booting of said first computer to cure said fault in said resource depended upon by at least one of said platform layer process groups on said first computer.

16. The method of claim 15 further comprising, in combination, the step of:

using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said fault in said resource depended upon by at least one of said platform layer process groups on said first computer.

17. A method for providing high availability applications comprising, in combination, the steps of:

a. running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. providing, on at least a first of said at least two computers, a system layer having at least one process group;

d. taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

e. re-booting said first computer upon a system layer process group fault occurring on said first computer;

f. providing, on at least a first of said at least two computers, a platform layer having at least one process group;

g. restarting at least one of said platform layer process groups upon a platform layer process group fault occurring on said first computer;

h. taking all of said process groups, except each of said at least one process group in said system layer, out of service on said first computer upon failure of said re-start to cure said platform layer process group fault;

i. providing at least one paired process group on a second of said at least two computers, said at least one paired process group being paired with one of said process groups on said first computer; and

j. activating said at least one paired process group on said second computer upon failure of said re-start to cure said platform layer process group fault.

18. The method of claim 17 further comprising, in combination, the step of:

re-initializing all of said process groups, except each of said at least one process group in said system layer, on said first computer upon failure of said re-start to cure said platform layer process group fault.

19. The method of claim 18 further comprising, in combination, the step of:

re-booting said first computer upon failure of said re-initialization to cure said platform layer process group fault.

20. The method of claim 19 further comprising, in combination, the step of:

using an independent computer to power cycle said first computer upon failure of said re-booting of said first computer to cure said platform layer process group fault.

21. The method of claim 20 further comprising, in combination, the step of:

using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said platform layer process group fault.

22. A method for providing high availability applications comprising, in combination, the steps of:

a. running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. providing, on at least said first computer, a system layer having at least one process group;

d. taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

e. re-booting said first computer upon a system layer process group fault occurring on said first computer;

f. providing, on at least a first of said at least two computers, an application layer having at least one process group;

g. taking said at least one application layer process group out of service on said first computer upon a fault in said at least one application layer process group on said first computer;

h. providing at least one paired process group on a second of said at least two computers, said paired process group being paired with one of said at least one application layer process group taken out of service on said first computer;

i. activating said paired process group on said second computer upon said fault in said at least one application layer process group on said first computer; and

j. re-initializing all of said process groups, except each of said at least one process group in said system layer, on said first computer upon not being able to take said application layer process group having said fault out of service on said first computer.

23. The method of claim 22 further comprising, in combination, the step of:

re-booting said first computer upon failure of said re-initialization to cure said application layer process group fault.

24. The method of claim 23 further comprising, in combination, the step of:

using an independent computer to power cycle said first computer upon failure of said re-boot of said first computer to cure said application layer process group fault.

25. The method of claim 24 further comprising, in combination, the step of:

using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said application layer process group fault.

26. A method for providing high availability applications comprising, in combination, the steps of:

a. running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. providing, on at least a first of said at least two computers, an application layer having at least two process groups;

d. defining a dependency by at least a first of said at least two application layer process groups upon at least a second of said at least two application layer process groups;

e. taking said first and said second application layer process groups out of service on said first computer upon a fault in said second application layer process group on said first computer;

f. providing at least one paired process group on a second of said at least two computers, said paired process group being paired with said second application layer process group on said first computer; and

g. activating said paired process group on said second computer upon said fault in said second application layer process group on said first computer.

27. The method of claim 26 further comprising, in combination, the steps of:

a. providing, on at least said first computer, a system layer having at least one process group;

b. taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

c. re-booting said first computer upon a system layer process group fault occurring on said first computer; and

d. re-initializing all of said process groups, except each of said at least one process group in said system layer, on said first computer upon not being able to take said first application layer process group out of service on said first computer.

28. The method of claim 27 further comprising, in combination, the step of:

re-booting said first computer upon failure of said re-initialization to cure said fault in said second application layer process group.

29. The method of claim 28 further comprising, in combination, the step of:

using an independent computer to power cycle said first computer upon failure of said re-boot of said first computer to cure said fault in said second application layer process group.

30. A method for providing high availability applications comprising, in combination, the steps of:

a. running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. providing, on at least a first of said at least two computers, an application layer having at least one process group;

d. taking said at least one application layer process group out of service on said first computer upon a fault in a resource depended upon by said at least one application layer process group;

e. providing at least one paired process group on a second of said at least two computers, said paired process group being paired with one of said at least one application layer process group on said first computer; and

f. activating said paired process group on said second computer upon said fault in said resource depended upon by said at least one application layer process group.

31. The method of claim 30 further comprising, in combination, the steps of:

a. providing, on at least said first computer, a system layer having at least one process group;

b. taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

c. re-booting said first computer upon a system layer process group fault occurring on said first computer;

d. re-initializing said resource having said fault; and

e. re-initializing all of said process groups, except each of said at least one process group in said system layer, on said first computer upon not being able to take said at least one application layer process group out of service on said first computer.

32. The method of claim 31 further comprising, in combination, the step of:

re-booting said first computer upon failure of said re-initializations to cure said fault in said resource depended upon by said at least one application layer process group.

33. The method of claim 32 further comprising, in combination, the step of:

using an independent computer to power cycle said first computer upon failure of said re-boot of said first computer to cure said fault in said resource depended upon by said at least one application layer process group.

34. The method of claim 33 further comprising, in combination, the step of:

using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said fault in said resource depended upon by said at least one application layer process group.

35. A method for providing high availability applications comprising, in combination, the steps of:

a. running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. providing, on at least said first computer, a system layer having at least one process group;

d. taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

e. re-booting said first computer upon a system layer process group fault occurring on said first computer;

f. providing, on at least a first of said at least two computers, an application layer having at least one process group;

g. re-starting said at least one application layer process group on said first computer upon a fault in said at least one application layer process group;

h. taking said at least one application layer process group out of service on said first computer upon failure of said re-start to cure said fault in said at least one application layer process group;

i. providing at least one paired process group on a second of said at least two computers, said paired process group being paired with one of said at least one application layer process group taken out of service on said first computer;

j. activating said paired process group on said second computer upon failure of said re-start to cure said fault in said at least one application layer process group taken out of service on said first computer; and

k. re-initializing all of said process groups, except each of said at least one process group in said system layer, on said first computer upon not being able to take said application layer process group having said fault out of service on said first computer.

36. The method of claim 35 further comprising, in combination, the step of:

re-booting said first computer upon failure of said re-initialization to cure said application layer process group fault.

37. The method of claim 36 further comprising, in combination, the step of:

using an independent computer to power cycle said first computer upon failure of said re-boot of said first computer to cure said application layer process group fault.

38. The method of claim 37 further comprising, in combination, the step of:

using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said application layer process group fault.

39. The method of claim 38 further comprising, in combination, the step of:

using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said fault in said second application layer process group.

40. A method for providing high availability applications comprising, in combination, the steps of:

a. running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. providing, on at least a first of said at least two computers, an application layer having at least two process groups;

d. defining a dependency by at least a first of said at least two application layer process groups upon at least a second of said at least two application layer process groups;

e. re-starting said second application layer process group on said first computer upon a fault in said second application layer process group;

f. taking said first and said second application layer process groups out of service on said first computer upon failure of said re-start to cure said fault in said second application layer process group;

g. providing at least one paired process group on a second of said at least two computers, said paired process group being paired with said first application layer process group on said first computer; and

h. activating said at least one paired process group on said second computer upon failure of said re-start to cure said fault in said second application layer process group.

41. The method of claim 40 further comprising, in combination, the steps of:

a. providing, on at least said first computer, a system layer having at least one process group;

b. taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

c. re-booting said first computer upon a system layer process group fault occurring on said first computer; and

d. re-initializing all of said process groups, except each of said at least one process group in said system layer, on said first computer upon not being able to take said first application layer process group out of service on said first computer.

42. The method of claim 41 further comprising, in combination, the step of:

re-booting said first computer upon failure of said re-initialization to cure said fault in said second application layer process group.

43. The method of claim 42 further comprising, in combination, the step of:

using an independent computer to power cycle said first computer upon failure of said re-boot of said first computer to cure said fault in said second application layer process group.

44. The method of claim 43 further comprising, in combination, the step of:

using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said fault in said second application layer process group.

45. An apparatus for providing high availability applications comprising, in combination:

a. means for running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. means for running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. means for providing, on at least a first of said at least two computers, a system layer having at least one process group;

d. means for taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

e. means for re-booting said first computer upon a system layer process group fault occurring on said first computer;

f. means for providing at least one paired process group on a second of said at least two computers, said paired process group being paired with one of said one or more process groups on said first computer; and

g. means for activating said paired process group on said second computer upon a system layer process group fault occurring on said first computer.

46. The apparatus of claim 45 further comprising, in combination:

means for using an independent computer to power cycle said first computer upon failure of said re-booting of said first computer to cure said system layer process group fault.

47. The apparatus of claim 46 further comprising, in combination:

means for using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said system layer process group fault.

48. An apparatus for providing high availability applications comprising, in combination:

a. means for running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. means for running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. means for providing, on at least a first of said at least two computers, a system layer having at least one process group;

d. means for taking all of said process groups out of service on said first computer upon a fault in a resource depended upon by at least one of said system layer process groups on said first computer;

e. means for re-booting said first computer upon said fault in said resource depended upon by at least one of said system layer process groups on said first computer;

f. means for providing at least one paired process group on a second of said at least two computers, said paired process group being paired with one of said process groups on said first computer; and

g. means for activating said paired process group on said second computer upon said fault in said resource depended upon by at least one of said system layer process groups on said first computer.

49. The apparatus of claim 48 further comprising, in combination:

means for using an independent computer to power cycle said first computer upon failure of said re-booting of said first computer to cure said fault in said resource depended upon by at least one of said system layer process groups on said first computer.

50. The apparatus of claim 49 further comprising, in combination:

means for using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said fault in said resource depended upon by at least one of said system layer process groups on said first computer.

51. An apparatus for providing high availability applications comprising, in combination:

a. means for running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. means for running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. means for providing, on at least a first of said at least two computers, a system layer having at least one process group;

d. means for taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

e. means for re-booting said first computer upon a system layer process group fault occurring on said first computer;

f. means for providing, on at least said first computer, a platform layer having at least one process group;

g. means for taking all of said process groups, except each of said at least one process group in said system layer, out of service on said first computer upon a platform layer process group fault occurring on said first computer;

h. means for providing at least one paired process group on a second of said at least two computers, said paired process group being paired with one of said process groups on said first computer; and

i. means for activating said paired process group on said second computer upon said platform layer process group fault occurring on said first computer.

52. The apparatus of claim 51 further comprising, in combination:

means for re-initializing all of said process groups, except each of said at least one process group in said system layer, on said first computer upon said platform layer process group fault occurring on said first computer.

53. The apparatus of claim 52 further comprising, in combination:

means for re-booting said first computer upon failure of said re-initialization to cure said platform layer process group fault.

54. The apparatus of claim 53 further comprising, in combination:

means for using an independent computer to power cycle said first computer upon failure of said re-booting of said first computer to cure said platform layer process group fault.

55. The apparatus of claim 54 further comprising, in combination:

means for using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said platform layer process group fault.

56. An apparatus for providing high availability applications comprising, in combination:

a. means for running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. means for running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. means for providing, on at least a first of said at least two computers, a system layer having at least one process group;

d. means for taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

e. means for re-booting said first computer upon a system layer process group fault occurring on said first computer;

f. means for providing, on at least a first of said at least two computers, a platform layer having at least one process group;

g. means for taking all of said process groups, except each of said at least one process group in said system layer, out of service on said first computer upon a fault in a resource depended upon by at least one of said platform layer process groups on said first computer;

h. means for providing at least one paired process group on a second of said at least two computers, said at least one paired process group being paired with one of said process groups on said first computer; and

i. means for activating said at least one paired process group on said second computer upon said fault in said resource depended upon by at least one of said platform layer process groups on said first computer.

57. The apparatus of claim 56 further comprising, in combination:

a. means for re-initializing said resource having said fault; and

b. means for re-initializing all of said process groups, except each of said at least one process group in said system layer, on said first computer upon said fault in said resource depended upon by at least one of said platform layer process groups on said first computer.

58. The apparatus of claim 57 further comprising, in combination:

means for re-booting said first computer upon failure of said re-initializations to cure said fault in said resource depended upon by at least one of said platform layer process groups on said first computer.

59. The apparatus of claim 58 further comprising, in combination:

means for using an independent computer to power cycle said first computer upon failure of said re-booting of said first computer to cure said fault in said resource depended upon by at least one of said platform layer process groups on said first computer.

60. The apparatus of claim 59 further comprising, in combination:

means for using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said fault in said resource depended upon by at least one of said platform layer process groups on said first computer.

61. An apparatus for providing high availability applications comprising, in combination:

a. means for running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. means for running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. means for providing, on at least a first of said at least two computers, a system layer having at least one process group;

d. means for taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

e. means for re-booting said first computer upon a system layer process group fault occurring on said first computer;

f. means for providing, on at least a first of said at least two computers, a platform layer having at least one process group;

g. means for restarting at least one of said platform layer process groups upon a platform layer process group fault occurring on said first computer;

h. means for taking all of said process groups, except each of said at least one process group in said system layer, out of service on said first computer upon failure of said re-start to cure said platform layer process group fault;

i. means for providing at least one paired process group on a second of said at least two computers, said at least one paired process group being paired with one of said process groups on said first computer; and

j. means for activating said at least one paired process group on said second computer upon failure of said re-start to cure said platform layer process group fault.

62. The apparatus of claim 61 further comprising, in combination:

means for re-initializing all of said process groups, except each of said at least one process group in said system layer, on said first computer upon failure of said re-start to cure said platform layer process group fault.

63. The apparatus of claim 62 further comprising, in combination:

means for re-booting said first computer upon failure of said re-initialization to cure said platform layer process group fault.

64. The apparatus of claim 63 further comprising, in combination:

means for using an independent computer to power cycle said first computer upon failure of said re-booting of said first computer to cure said platform layer process group fault.

65. The apparatus of claim 64 further comprising, in combination:

means for using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said platform layer process group fault.

66. An apparatus for providing high availability applications comprising, in combination:

a. means for running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. means for running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. means for providing, on at least said first computer, a system layer having at least one process group;

d. means for taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

e. means for re-booting said first computer upon a system layer process group fault occurring on said first computer;

f. means for providing, on at least a first of said at least two computers, an application layer having at least one process group;

g. means for taking said at least one application layer process group out of service on said first computer upon a fault in said at least one application layer process group on said first computer;

h. means for providing at least one paired process group on a second of said at least two computers, said paired process group being paired with one of said at least one application layer process group taken out of service on said first computer;

i. means for activating said paired process group on said second computer upon said fault in said at least one application layer process group on said first computer; and

j. means for re-initializing all of said process groups, except each of said at least one process group in said system layer, on said first computer upon not being able to take said application layer process group having said fault out of service on said first computer.

67. The apparatus of claim 66 further comprising, in combination:

means for re-booting said first computer upon failure of said re-initialization to cure said application layer process group fault.

68. The apparatus of claim 67 further comprising, in combination:

means for using an independent computer to power cycle said first computer upon failure of said re-boot of said first computer to cure said application layer process group fault.

69. The apparatus of claim 68 further comprising, in combination:

means for using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said application layer process group fault.

70. An apparatus for providing high availability applications comprising, in combination:

a. means for running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. means for running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. means for providing, on at least a first of said at least two computers, an application layer having at least two process groups;

d. means for defining a dependency by at least a first of said at least two application layer process groups upon at least a second of said at least two application layer process groups;

e. means for taking said first and said second application layer process groups out of service on said first computer upon a fault in said second application layer process group on said first computer;

f. means for providing at least one paired process group on a second of said at least two computers, said paired process group being paired with said second application layer process group on said first computer; and

g. means for activating said paired process group on said second computer upon said fault in said second application layer process group on said first computer.

71. The apparatus of claim 70 further comprising, in combination:

a. means for providing, on at least said first computer, a system layer having at least one process group;

b. means for taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

c. means for re-booting said first computer upon a system layer process group fault occurring on said first computer; and

d. means for re-initializing all of said process groups, except each of said at least one process group in said system layer, on said first computer upon not being able to take said first application layer process group out of service on said first computer.

72. The apparatus of claim 71 further comprising, in combination:

means for re-booting said first computer upon failure of said re-initialization to cure said fault in said second application layer process group.

73. The apparatus of claim 72 further comprising, in combination:

means for using an independent computer to power cycle said first computer upon failure of said re-boot of said first computer to cure said fault in said second application layer process group.

74. The apparatus of claim 73 further comprising, in combination:

means for using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said fault in said second application layer process group.

75. An apparatus for providing high availability applications comprising, in combination:

a. means for running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. means for running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. means for providing, on at least a first of said at least two computers, an application layer having at least one process group;

d. means for taking said at least one application layer process group out of service on said first computer upon a fault in a resource depended upon by said at least one application layer process group;

e. means for providing at least one paired process group on a second of said at least two computers, said paired process group being paired with one of said at least one application layer process group on said first computer; and

f. means for activating said paired process group on said second computer upon said fault in said resource depended upon by said at least one application layer process group.

76. The apparatus of claim 75 further comprising, in combination:

a. means for providing, on at least said first computer, a system layer having at least one process group;

b. means for taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

c. means for re-booting said first computer upon a system layer process group fault occurring on said first computer;

d. means for re-initializing said resource having said fault; and

e. means for re-initializing all of said process groups, except each of said at least one process group in said system layer, on said first computer upon not being able to take said at least one application layer process group out of service on said first computer.

77. The apparatus of claim 76 further comprising, in combination:

means for re-booting said first computer upon failure of said re-initializations to cure said fault in said resource depended upon by said at least one application layer process group.

78. The apparatus of claim 77 further comprising, in combination:

means for using an independent computer to power cycle said first computer upon failure of said re-boot of said first computer to cure said fault in said resource depended upon by said at least one application layer process group.

79. The apparatus of claim 78 further comprising, in combination:

means for using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said fault in said resource depended upon by said at least one application layer process group.

80. An apparatus for providing high availability applications comprising, in combination:

a. means for running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. means for running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. means for providing, on at least said first computer, a system layer having at least one process group;

d. means for taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

e. means for re-booting said first computer upon a system layer process group fault occurring on said first computer;

f. means for providing, on at least a first of said at least two computers, an application layer having at least one process group;

g. means for re-starting said at least one application layer process group on said first computer upon a fault in said at least one application layer process group;

h. means for taking said at least one application layer process group out of service on said first computer upon failure of said re-start to cure said fault in said at least one application layer process group;

i. means for providing at least one paired process group on a second of said at least two computers, said paired process group being paired with one of said at least one application layer process group taken out of service on said first computer;

j. means for activating said paired process group on said second computer upon failure of said re-start to cure said fault in said at least one application layer process group taken out of service on said first computer; and

k. means for re-initializing all of said process groups, except each of said at least one process group in said system layer, on said first computer upon not being able to take said application layer process group having said fault out of service on said first computer.

81. The apparatus of claim 80 further comprising, in combination:

means for re-booting said first computer upon failure of said re-initialization to cure said application layer process group fault.

82. The apparatus of claim 81 further comprising, in combination:

means for using an independent computer to power cycle said first computer upon failure of said re-boot of said first computer to cure said application layer process group fault.

83. The apparatus of claim 82 further comprising, in combination:

means for using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said application layer process group fault.

84. A method for providing high availability applications comprising, in combination:

a. means for running, on at least two computers, one or more process groups, at least one of said process groups containing one or more processes that have a fault recovery strategy common to said at least one of said process groups;

b. means for running, on at least one of said at least two computers, a process group manager that initiates a fault recovery strategy for at least one of said one or more process groups;

c. means for providing, on at least a first of said at least two computers, an application layer having at least two process groups;

d. means for defining a dependency by at least a first of said at least two application layer process groups upon at least a second of said at least two application layer process groups;

e. means for re-starting said second application layer process group on said first computer upon a fault in said second application layer process group;

f. means for taking said first and said second application layer process groups out of service on said first computer upon failure of said re-start to cure said fault in said second application layer process group;

g. means for providing at least one paired process group on a second of said at least two computers, said paired process group being paired with said first application layer process group on said first computer; and

h. means for activating said at least one paired process group on said second computer upon failure of said re-start to cure said fault in said second application layer process group.

85. The apparatus of claim 84 further comprising, in combination:

a. means for providing, on at least said first computer, a system layer having at least one process group;

b. means for taking all of said process groups out of service on said first computer upon a system layer process group fault occurring on said first computer;

c. means for re-booting said first computer upon a system layer process group fault occurring on said first computer; and

d. means for re-initializing all of said process groups, except each of said at least one process group in said system layer, on said first computer upon not being able to take said first application layer process group out of service on said first computer.

86. The apparatus of claim 85 further comprising, in combination:

means for re-booting said first computer upon failure of said re-initialization to cure said fault in said second application layer process group.

87. The apparatus of claim 86 further comprising, in combination:

means for using an independent computer to power cycle said first computer upon failure of said re-boot of said first computer to cure said fault in said second application layer process group.

88. The apparatus of claim 87 further comprising, in combination:

means for using an independent computer to power down said first computer upon failure of said power cycling of said first computer to cure said fault in said second application layer process group.