Fault tolerance using logical checkpointing in computing systems

• US 6,954,877 B2
• Filed: 11/29/2001
• Issued: 10/11/2005
• Est. Priority Date: 11/29/2001
• Status: Expired due to Fees

• Current Assignee: Agami Systems, Inc.
• Original Assignee: Agami Systems, Inc.
• Inventors: Tofano, Jeff, Earl, William J.
• Primary Examiner(s): Iqbal, Nadeem
• Assistant Examiner(s): Bonura, Timothy M.
• Application Number:
  US09/997,877
• Publication Number:
  US 20040199812A1
• Time in Patent Office: 1,412 Days
• Field of Search: 714/10, 714/11, 714/12, 714/13
• US Class Current: 714/13
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 2003/0697 device management, e.g. handlers, drivers, I/O schedulers
• CPC Class Codes: G06F 3/0601 Interfaces specially adapted for storage systems

Abstract

A system and method for checkpointing a primary computer process to a backup computer process such that if there is a failure of a primary process, the backup process can takeover without interruption. In addition, upgrades to different version of software or equipment can take place without interruption. The invention provides a lightweight checkpointing method that allows checkpointing of only external requests or messages that change the state of the service instance, thereby reducing the overhead and performance penalties. In addition, the present invention checkpoints data for primary and backups that do not share resources but are logically equivalent. All communication between the primary and backup takes places using network protocols.

Claims

1. A service operating on a computer system to process requests from client processes, comprising:

a primary service instance operating on the computer system, including means for receiving and processing client requests, from the client processes; and

at least one backup service instance operating on the computer system, including means for receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the primary service instance includes means for determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change; and

means for communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

wherein each backup service instance includes means for communicating, to the primary service instance, acknowledgement of the determined logical requests.

2. The service of claim 1, wherein the primary service instance includes means for committing the determined logical requests in response to the acknowledgements of the determined logical requests.

3. The service of claim 2, wherein each backup service instance includes means for committing the results of processing the determined logical requests.

4. The service of claim 3, further comprising:

order ensuring means for ensuring that, for a sequence of determined logical requests, each backup service instance commits the results of processing the determined logical requests in the same order as the primary service instance commits the results.

5. The service of claim 4, wherein:

the order ensuring means includes means for maintaining a table of determined logical requests; and

each backup service instance includes means for processing the table of determined logical requests to determine an order of committing results of processing the determined logical requests.

6. The service of claim 5, wherein:

the means for maintaining a table of determined logical requests includes means for associating sequence values with the determined logical requests; and

the means for processing the table includes means for processing the sequence values.

7. The service of claim 2, wherein:

a sequence of associated client requests is a transaction;

each backup service instance includes means for communicating a single acknowledgement for the client requests of the transaction; and

each backup service instance includes means for committing the results of processing the client requests of the transaction as a single commitment.

8. A service operating on a computer system to process requests from client processes, comprising:

a primary service instance operating on the computer system, including means for receiving and processing client requests, from the client processes; and

at least one backup service instance operating on the computer system, including means for receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the primary service instance includes means for determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change;

means for communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance;

means for receiving an indication of a path failure between the primary service instance and a client process that occurs after a result of processing a particular client request is committed and before the client process receives the result from the service; and

means for receiving the particular client request again and for providing the response to the client request without again processing the client request.

9. A service operating on a computer system to process requests from client processes, comprising:

a primary service instance operating on the computer system, including means for receiving and processing client requests, from the client processes; and

at least one backup service instance operating on the computer system, including means for receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the primary service instance includes means for determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change;

means for communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

wherein each backup service instance includes means for receiving an indication of a failure of the primary service instance, and means for notifying the client process that each backup service instance has become a new primary service instance; and

the new primary service instance includes means for receiving client requests and, for client requests that have already been committed by the new primary service instance when the new primary service instance was a backup service instance, providing the committed response to the client request without again processing the request.

10. The service of claim 9, wherein:

the new primary service instance includes means for receiving client requests and, for client requests that have already been received but not committed by the new primary service instance when the new primary service instance was a backup service instance, allowing the new primary service instance to continue processing the requests.

11. The service of claim 10, wherein:

a sequence of associated client requests is a transaction;

each backup service instance includes means for communicating an acknowledgement for each client request of the transaction; and

each backup service instance includes means for committing the results of processing the client requests of the transaction as a single commitment.

12. The service of claim 11, wherein:

the transaction committing means of each backup service instance includes means for receiving a commitment request from the primary service instance, wherein the transaction committing means commits the results of the transaction in response to the received commitment request.

13. A service operating on a computer system to process requests from client processes, comprising:

a primary service instance operating on the computer system, including means for receiving and processing client requests, from the client processes; and

at least one backup service instance operating on the computer system, including means for receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the primary service instance includes means for determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change;

means for communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance;

means for creating an additional backup service instance, including means for replicating the logical view of the primary service instance to the additional backup service instance, wherein the means for replicating the logical view of the primary service instance to the additional backup service instance provides the determined logical requests to the additional backup service instance; and

processes a request history table to ensure that logical view of the additional backup service instance is the same as the logical view of the primary service instance.

14. The service of claim 13, wherein:

the replicating means includes means for configuring the determining means to process client requests received while the request history table is being processed.

15. The service of claim 14, wherein the means for configuring the determining means to process client requests received while the request history table is being processed operates in response to a termination of the request history table being processed.

16. The service of claim 15, wherein the transitioning means includes means for configuring the primary service instance to temporarily stop accepting client requests.

17. The service of claim 16, wherein the means for configuring the primary service instance to temporarily stop accepting client requests operates in response to a comparison of the logical view of the primary service instance to the logical view of the additional backup service instance.

18. A service operating on a computer system to process requests from client processes, comprising:

a primary service instance operating on the computer system, and being adapted to receive and process client requests from the client processes;

at least one backup service instance operating on the computer system, and being adapted to receive and process the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the primary service instance determines which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicates with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

a mechanism for creating and operating an additional backup service instance, including a replicating mechanism for replicating the logical view of the primary service instance to the additional backup service instance, and

for providing the determined logical requests to the additional backup service instance.

19. The service of claim 18, wherein:

the replicating mechanism is further adapted to process a request history table to ensure that logical view of the additional backup service instance is the same as the logical view of the primary service instance.

20. A service operating on a computer system to process requests from client processes, comprising:

a primary service instance operating on the computer system, and being adapted to receive and process client requests from the client processes;

at least one backup service instance operating on the computer system, and being adapted to receive and process the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the primary service instance determines which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicates with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

a mechanism for creating and operating an additional backup service instance, including a replicating mechanism for replicating the logical view of the primary service instance to the additional backup service instance, and

for implementing the additional backup service instance to have a physical behavior different from the physical behavior of the primary service instance.

21. The service of claim 20, further comprising:

a disabling mechanism for disabling the at least one backup service instance;

a mechanism for enabling and operating the at least one backup service instance with an implementation that has the same physical behavior as the additional backup service instance; and

a replicating mechanism replicating the logical view of the primary service instance to the at least one backup service instance.

22. The service of claim 21, a disabling mechanism for disabling the primary service instance;

and a mechanism for causing at least one of the at least one backup service instance and the additional backup service instance to become a primary service instance.

23. The service of claim 22, a mechanism for creating and operating a second additional backup service instance, including a replicating mechanism for replicating the logical view of the primary service instance to the second additional backup service instance.

24. A service operating on a computer system to process requests from client processes, comprising:

a primary service instance operating on the computer system, and being adapted to receive and process client requests from the client processes;

at least one backup service instance operating on the computer system, and being adapted to receive and process the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the primary service instance determines which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicates with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

a mechanism for creating and operating an additional backup service instance, including a replicating mechanism for replicating the logical view of the primary service instance to the additional backup service instance, and

for implementing the additional backup service instance to have a physical behavior the same as the physical behavior of the primary service instance.

25. The service of claim 24, further comprising:

a disabling mechanism for disabling the at least one backup service instance;

a mechanism for enabling and operating the at least one backup service instance with an implementation that has the same physical behavior as the additional backup service instance; and

a replicating mechanism for replicating the logical view of the primary service instance to the at least one backup service instance.

26. The service of claim 25, further comprising:

a disabling mechanism for disabling the primary service instance; and

a mechanism for causing at least one of the at least one backup service instance and the additional backup service instance to become a primary service instance.

27. The service of claim 26, further comprising:

a mechanism for creating and operating a second additional backup service instance, including a replicating mechanism for replicating the logical view of the primary service instance to the second additional backup service instance.

28. A method of operating a computer system to process requests from client processes, comprising:

operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes; and

operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

wherein the step of operating each backup service instance includes communicating, to the primary service instance, acknowledgement of the determined logical requests.

29. A method of operating a computer system to process requests from client processes, comprising:

operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes; and

operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance, and

committing the determined logical requests in response to the acknowledgements of the determined logical requests.

30. The method of claim 29, wherein the step of operating each backup service instance includes committing the results of processing the determined logical requests.

31. The method of claim 30, further comprising:

ensuring that, for a sequence of determined logical requests, each backup service instance commits the results of processing the determined logical requests in the same order as the primary service instance commits the results.

32. The method of claim 31, wherein:

the order ensuring step includes maintaining a table of determined logical requests; and

the step of operating each backup service instance includes processing the table of determined logical requests to determine an order of committing results of processing the determined logical requests.

33. The method of claim 32, wherein:

the step of maintaining a table of determined logical requests includes associating sequence values with the determined logical requests; and

the step of processing the table includes processing the sequence values.

34. The method of claim 29, wherein:

a sequence of associated client requests is a transaction;

the step of operating each backup service instance includes communicating a single acknowledgement for the client requests of the transaction; and

committing the results of processing the client requests of the transaction as a single commitment.

35. A method of operating a computer system to process requests from client processes, comprising:

operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes; and

operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance,

wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

wherein;

the step of operating the primary service instance includes receiving an indication of a path failure between the primary service instance and a client process that occurs after a result of processing a particular client request is committed and before the client process receives the result from the service, and receiving the particular client request again and providing the response to the client request without again processing the client request.

36. A method of operating a computer system to process requests from client processes, comprising:

operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes; and

operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

wherein;

operating each backup service instance includes receiving an indication of a failure of the primary service instance, and notifying the client process that each backup service instance has become a new primary service instance; and

operating the new primary service instance includes receiving client requests and, for client requests that have already been committed by the new primary service instance when the new primary service instance was a backup service instance, providing the committed response to the client request without again processing the request.

37. The method of claim 36, wherein:

the step of operating the new primary service instance includes receiving client requests and, for client requests that have already been received but not committed by the new primary service instance when the new primary service instance was a backup service instance, allowing the new primary service instance to continue processing the requests.

38. The method of claim 37, wherein a sequence of associated client requests is a transaction;

the step of operating each backup service instance includes the step of operating each backup service instance includes communicating an acknowledgement for each client request of the transaction, and committing the results of processing the client requests of the transaction as a single commitment.

39. The method of claim 38, wherein:

committing the results of processing the client requests of a transaction for each backup service instance includes receiving a commitment request from the primary service instance, and committing the results of the transaction in response to the received commitment request.

40. A method of operating a computer system to process requests from client processes, comprising:

operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes;

operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change; and

communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

creating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance;

wherein the step of replicating the logical view of the primary service instance to the additional backup service instance includes providing the determined logical requests to the additional backup service instance, and

processing a request history table to ensure that logical view of the additional backup service instance is the same as the logical view of the primary service instance.

41. The method of claim 40, wherein:

the step of replicating the logical view of the primary service instance includes configuring the determining step to process client requests received while the request history table is being processed.

42. The method of claim 41, wherein the step of configuring the determining step to process client requests received while the request history table is being processed is performed in response to a termination of the request history table being processed.

43. The method of claim 42, wherein the transitioning step includes configuring the replacement primary service instance to temporarily stop accepting client requests.

44. The method of claim 43, wherein the step of configuring the replacement primary service instance to temporarily stop accepting client requests is performed in response to a comparison of the logical view of the primary service instance to the logical view of the additional backup service instance.

45. A method of operating a service on a computer system to process requests from client processes, comprising:

operating a primary service instance on the computer system, including receiving and processing client requests from the client processes;

operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

creating and operating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance;

wherein;

the step of replicating the logical view of the primary service instance to the additional backup service instance includes providing the determined logical requests to the additional backup service instance.

46. A method of operating a service on a computer system to process requests from client processes, comprising:

operating a primary service instance on the computer system, including receiving and processing client requests from the client processes;

operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

creating and operating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance;

wherein;

the replicating step includes processing a request history table to ensure that logical view of the additional backup service instance is the same as the logical view of the primary service instance.

47. A method of operating a service on a computer system to process requests from client processes, comprising:

operating a primary service instance on the computer system, including receiving and processing client requests from the client processes;

operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

creating and operating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance;

wherein;

the step of replicating the logical view of the primary service instance to the additional backup service instance includes implementing the additional backup service instance to have a physical behavior different from the physical behavior of the primary service instance.

48. The method of claim 47, further comprising:

disabling the at least one backup service instance;

enabling and operating the at least one backup service instance with an implementation that has the same physical behavior as the additional backup service instance; and

replicating the logical view of the primary service instance to the at least one backup service instance.

49. The method of claim 48, comprising:

disabling the primary service instance; and

causing at least one of the at least one backup service instance and the additional backup service instance to become a primary service instance.

50. The method of claim 49, further comprising:

creating and operating a second additional backup service instance, including replicating the logical view of the primary service instance to the second additional backup service instance.

51. A method of operating a service on a computer system to process requests from client processes, comprising:

operating a primary service instance on the computer system, including receiving and processing client requests from the client processes;

operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

creating and operating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance,

wherein;

the step of replicating the logical view of the primary service instance to the additional backup service instance includes implementing the additional backup service instance to have a physical behavior the same as the physical behavior of the primary service instance.

52. The method of claim 51, further comprising:

disabling the at least one backup service instance;

enabling and operating the at least one backup service instance with an implementation that has the same physical behavior as the additional backup service instance; and

replicating the logical view of the primary service instance to the at least one backup service instance.

53. The method of claim 52, further comprising:

disabling the primary service instance; and

causing at least one of the at least one backup service instance and the additional backup service instance to become a primary service instance.

54. The method of claim 53, further comprising:

creating and operating a second additional backup service instance, including replicating the logical view of the primary service instance to the second additional backup service instance.

55. A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform method steps for operating a computer system to process requests from client processes, said method steps comprising:

operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes; and

operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

wherein the step of operating each backup service instance includes communicating, to the primary service instance, acknowledgement of the determined logical requests.

56. The program storage device of claim 55, wherein the step of operating the primary service instance includes committing the determined logical requests in response to the acknowledgements of the determined logical requests.

57. The program storage device of claim 56, wherein the step of operating each backup service instance includes committing the results of processing the determined logical requests.

58. The program storage device of claim 57, further comprising:

ensuring that, for a sequence of determined logical requests, each backup service instance commits the results of processing the determined logical requests in the same order as the primary service instance commits the results.

59. The program storage device of claim 58, wherein:

step order ensuring step includes maintaining a table of determined logical requests; and

the step of operating each backup service instance includes processing the table of determined logical requests to determine an order of committing results of processing the determined logical requests.

60. The program storage device of claim 59, wherein:

the step of maintaining a table of determined logical requests includes associating sequence values with the determined logical requests; and

the step of processing the table includes processing the sequence values.

61. The program storage device of claim 56, wherein:

a sequence of associated client requests is a transaction;

the step of operating each backup service instance includes communicating a single acknowledgement for the client requests of the transaction; and

the step of operating each backup service instance includes committing the results of processing the client requests of the transaction as a single commitment.

62. A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform method steps for operating a computer system to process requests from client processes, said method steps comprising:

operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes; and

operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance,

receiving an indication of a path failure between the primary service instance and a client process that occurs after a result of processing a particular client request is committed and before the client process receives the result from the service, and receiving the particular client request again and providing the response to the client request without again processing the client request.

63. A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform method steps for operating a computer system to process requests from client processes, said method steps comprising:

operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes; and

operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance;

wherein the step of operating each backup service instance includes receiving an indication of a failure of the primary service instance, and notifying the client process that each backup service instance has become a new primary service instance; and

wherein the step of operating the new primary service instance includes receiving client requests and, for client requests that have already been committed by the new primary service instance when the new primary service instance was a backup service instance, providing the committed response to the client request without again processing the request.

64. The program storage device of claim 63, wherein:

the step of operating the new primary service instance includes receiving client requests and, for client requests that have already been received but not committed by the new primary service instance when the new primary service instance was a backup service instance, allowing the new primary service instance to continue processing the requests.

65. The program storage device of claim 63, wherein:

a sequence of associated client requests is a transaction;

the step of operating each backup service instance includes communicating an acknowledgement for each client request of the transaction; and

the step of operating each backup service instance includes committing the results of processing the client requests of the transaction as a single commitment.

66. The program storage device of claim 65, wherein:

the step of committing a transaction for each backup service instance includes receiving a commitment request from the primary service instance, wherein the transaction committing step includes committing the results of the transaction in response to the received commitment request.

67. A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform method steps for operating a computer system to process requests from client processes, said method steps comprising:

operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes; and

operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

creating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance;

wherein the step of replicating the logical view of the primary service instance to the additional backup service instance includes providing the determined logical requests to the additional backup service instance, and

processing a request history table to ensure that logical view of the additional backup service instance is the same as the logical view of the primary service instance.

68. The program storage device of claim 67, wherein:

the step of replicating the logical view of the primary service instance includes configuring the determining step to process client requests received while the request history table is being processed.

69. The program storage device of claim 68, wherein the step of configuring the determining step to process client requests received while the request history table is being processed is performed in response to a termination of the request history table being processed.

70. The program storage device of claim 69, wherein the transitioning step includes configuring the replacement primary service instance to temporarily stop accepting client requests.

71. The program storage device of claim 70, wherein the step of configuring the replacement primary service instance to temporarily stop accepting client requests is performed in response to a comparison of the logical view of the primary service instance to the logical view of the additional backup service instance.

72. A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform method steps for operating a computer system to process requests from client processes, said method steps comprising:

operating a primary service instance on the computer system, including receiving and processing client requests from the client processes;

operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

creating and operating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance;

wherein;

the step of replicating the logical view of the primary service instance to the additional backup service instance includes providing the determined logical requests to the additional backup service instance.

73. The program storage device of claim 72, wherein:

the replicating step includes processing a request history table to ensure that logical view of the additional backup service instance is the same as the logical view of the primary service instance.

74. A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform method steps for operating a computer system to process requests from client processes, said method steps comprising:

operating a primary service instance on the computer system, including receiving and processing client requests from the client processes;

operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

creating and operating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance;

wherein;

the step of replicating the logical view of the primary service instance to the additional backup service instance includes implementing the additional backup service instance to have a physical behavior different from the physical behavior of the primary service instance.

75. The program storage device of claim 74, further comprising:

disabling the at least one backup service instance;

enabling and operating the at least one backup service instance with an implementation that has the same physical behavior as the additional backup service instance; and

replicating the logical view of the primary service instance to the at least one backup service instance.

76. The program storage device of claim 75, further comprising:

disabling the primary service instance; and

causing at least one of the at least one backup service instance and the additional backup service instance to become a primary service instance.

77. The program storage device of claim 76, further comprising:

creating and operating a second additional backup service instance, including replicating the logical view of the primary service instance to the second additional backup service instance.

78. A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform method steps for operating a computer system to process requests from client processes, said method steps comprising:

operating a primary service instance on the computer system, including receiving and processing client requests from the client processes;

operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;

wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and

creating and operating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance;

wherein;

the step of replicating the logical view of the primary service instance to the additional backup service instance includes implementing the additional backup service instance to have a physical behavior the same as the physical behavior of the primary service instance.

79. The program storage device of claim 78, further comprising:

disabling the at least one backup service instance;

enabling and operating the at least one backup service instance with an implementation that has the same physical behavior as the additional backup service instance; and

replicating the logical view of the primary service instance to the at least one backup service instance.

80. The program storage device of claim 79, further comprising:

disabling the primary service instance; and

causing at least one of the at least one backup service instance and the additional backup service instance to become a primary service instance.

81. The program storage device of claim 80, further comprising:

creating and operating a second additional backup service instance, including replicating the logical view of the primary service instance to the second additional backup service instance.