Horizontally scalable and reliable distributed transaction management in a clustered application server environment

• US 7,730,489 B1
• Filed: 12/10/2003
• Issued: 06/01/2010
• Est. Priority Date: 12/10/2003
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Oracle America, Inc. (Oracle Corporation)
• Inventors: Viswanatham, Satish C., Bhogi, Sankara Rao, Islam, Akm N., Duvur, Sreeram
• Primary Examiner(s): An; Meng-Ai
• Assistant Examiner(s): Arcos; Caroline
• Application Number:
  US10/732,387
• Time in Patent Office: 2,365 Days
• Field of Search: 718/102, 718/104, 718/105, 718/100, 714/2, 714/15, 714/21, 714/36, 714/48, 714/4, 714/16, 714/20, 709/224, 709/226, 709/232, 709/223
• US Class Current: 718/104
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 9/466 Transaction processing

Abstract

Various embodiments of a cluster of application servers in a networked computer system in which each application server includes a separate transaction manager instance are disclosed. In one embodiment, a separate transaction manager instance may be used to coordinate transactions initiated by an application component executing in process with the application server for a node of a cluster. If each transaction manager instance manages only transactions initiated within its own application server, then the loss of a single transaction manager instance may have no detrimental effect on transactions initiated in other nodes of the cluster. Further, if each transaction manager instance maintains its transaction logs in highly available memory accessible to all nodes of the cluster, in flight transactions of a failed node may be recovered by an instance in a different node.

Claims

1. A system, comprising:

a server cluster, comprising;

a plurality of server nodes, wherein each server node comprises;

an application server instance;

one or more applications configured to execute within the application server instance and to initiate one or more global transactions each involving a set of data modifying operations to be committed to a plurality of storage resources atomically; and

one or more separate transaction manager instances each configured to execute within the application server instance;

wherein each separate transaction manager instance for each application server instance is configured to coordinate transactions initiated within that application server instance, and to log in a transaction log an intermediate status of each uncommitted transaction of the transactions initiated within the application server instance;

wherein the server cluster is configured to;

detect a failure of one of the plurality of application server instances of the cluster; and

in response to said detecting, select one or more surrogate transaction manager instances from one or more of the application server instances of the server cluster that are still operational;

wherein the one or more surrogate transaction manager instances are configured to read one or more transaction logs of the failed application server and to complete one or more uncommitted transactions indicated by the one or more transaction logs of the failed application server;

wherein the one or more application server instances having the one or more surrogate transaction manager instances are configured to attempt to recover each in-flight transaction entry in a transaction log of the failed server, wherein for each log entry one or more of the application server instances having one of the one or more surrogate transaction manager instances are configured to open connections to all data sources involved in the transaction corresponding to the log entry, and if all connections are successful, recover the transaction.

2. The system as recited in claim 1, wherein the one or more applications comprise a plurality of applications that initiate transactions, and wherein the one or more separate transaction manager instances comprise a separate transaction manager instance for each application that initiates transactions.

3. The system as recited in claim 1, wherein the cluster is configured to detect the failure of one of the plurality of application server instances by a heartbeat communications protocol.

4. The system as recited in claim 1, wherein the cluster is configured to detect the failure of one of the plurality of application server instances by a periodic polling communications protocol.

5. The system as recited in claim 1, wherein the cluster is configured to select the one or more surrogate transaction manager instances from the one or more application server instances that are still operational based on an identity of the failed application server.

6. The system as recited in claim 1, wherein the one or more surrogate transaction manager instances are configured to recover transactions of the failed application server instance by transferring ownership of transaction logs from the failed application server instance to the one or more surrogate transaction manager instances.

7. The system as recited in claim 6, wherein the one or more surrogate transaction manager instances are configured to transfer ownership of transaction logs from the failed application server instance to the one or more surrogate transaction manager instances as part of a transaction.

8. The system as recited in claim 1, wherein one or more of the application server instances having one of the one or more surrogate transaction manager instances are configured to open connections to all data sources involved in the transaction corresponding to the log entry within a specified period of time, and if all connections are successful, recover the transaction and delete the corresponding log entry.

9. The system as recited in claim 1, wherein the application server instance that failed is configured to restart and attempt to recover the transactions from the log, and in response to determining that no other application server instance has yet attempted to recover the transactions, the failed application server instance is configured to attempt recovery of the transactions upon restart without re-acquiring ownership of the log.

10. The system as recited in claim 1, wherein each application server instance having one of the one or more surrogate transaction manager instances and attempting to recover the transactions from the log is configured to obtain ownership of the transaction log prior to attempting to recover any transactions.

11. The system as recited in claim 1, wherein each application server instance having one of the one or more surrogate transaction manager instances and attempting to recover the transactions is configured to perform an initial recovery attempt and deferred recovery attempts at a predetermined interval to recover remaining transactions not recovered by the initial recovery attempt for a predetermined number of attempts after normal application server instance operations have been resumed after the initial recovery attempt.

12. The system as recited in claim 1, wherein the transaction log is stored in highly available storage accessible to each of the plurality of application server instances of the cluster.

13. The system as recited in claim 12, wherein the highly available storage is a highly available file system, wherein a location of the transaction logs in the highly available file system is configurable by an administrator.

14. A method, comprising:

operating one or more separate transaction manager instances in each node of an application server cluster;

one or more applications initiating a plurality of global transactions in each server node, wherein each of the plurality of global transactions involves a set of data modifying operations to be committed to a plurality of storage resources atomically;

coordinating transactions initiated in a particular server node by the one or more transaction manager instances operating in that node;

each transaction manager instance maintaining transaction logs for transactions it manages in a highly available storage, wherein each transaction log contains an intermediate status of each uncommitted transaction of the transactions it manages and is usable by another transaction manager instance to recover the uncommitted transactions if the transaction manager fails;

detecting a failure of one of the plurality of application server instances of the cluster;

in response to said detecting, selecting one or more surrogate transaction manager instances from one or more of the application server instances of the server cluster that are still operational;

reading, by the one or more selected surrogate transaction manager instances, one or more transaction logs of the failed application server and to complete one or more uncommitted transactions indicated by the one or more transaction logs of the failed application server; and

attempting, by one or more application server instances having the one or more selected surrogate transaction manager instances, to recover each in-flight transaction entry in a transaction log of the failed server, wherein said attempting comprises, for each log entry, opening connections to all data sources involved in the transaction corresponding to the log entry, and if all connections are successful, recover the transaction.

15. The method as recited in claim 14, wherein the one or more applications comprise a plurality of applications in one of the server nodes initiating transactions, and wherein the one or more separate transaction manager instances comprise a separate transaction manager instance operating for each application generating transactions.

16. The method as recited in claim 14, wherein the server nodes of the cluster are a plurality of application server nodes, wherein the highly available storage is accessible to each of the plurality of application server nodes of the cluster, and wherein the highly available storage stores the transaction logs in a location designated by an administrator for the plurality of application server nodes of the cluster.

17. The method as recited in claim 14, further comprising detecting a failure of an application server including one of the separate transaction manager instances on one of the server nodes within the application server cluster.

18. The method as recited in claim 17, further comprising restarting the failed application server of the application server cluster and the transaction manager instance of the restarted application server attempting to recover in flight transactions for an initial predetermined period of time.

19. The method as recited in claim 18, further comprising, in response to one or more transactions not being recovered within the initial predetermined period of time, the transaction manager instance of the restarted application server delaying recovery of the one or more transactions not recovered within the initial period until after the restarted application server has resumed normal server operations while transaction recovery is still in progress.

20. The method as recited in claim 19, further comprising, in response to un-recovered transactions remaining after a maximum number of deferred recovery attempts has been made, the application server generating an error requesting operator intervention.

21. The method as recited in claim 17, further comprising designating, for the failed application server, a surrogate application server including one of the transaction manager instances, and the transaction manager instance of the surrogate application server attempting to recover in flight transactions of the failed application server for an initial predetermined period of time.

22. The method as recited in claim 21, further comprising the transaction manager instance of the surrogate application server implementing deferred transaction recovery in response to one or more transactions not being recovered within the initial period, wherein deferred transaction recovery attempts to recover transactions subsequent to the initial period and after normal application server operations have resumed.

23. The method as recited in claim 22, further comprising, in response to un-recovered transactions remaining after a maximum number of deferred recovery attempts has been made, the surrogate application server generating an error requesting operator intervention.

24. The method as recited in claim 17, wherein the failure of the application server is detected by a heartbeat communications protocol.

25. The method as recited in claim 17, wherein the failure of the application server is detected by a periodic polling communications protocol.

26. The method as recited in claim 21, wherein the surrogate application server is selected from remaining operational application servers of the cluster based on an identity of the failed application server.

27. The method as recited in claim 21, wherein ownership of the transaction logs is transferred from the failed application server to the surrogate as part of a transaction.

28. A computer accessible storage medium storing program instructions that when executed on one or more computers cause the one or more computers to:

operate a separate transaction manager instance in each node in an application server cluster;

initiate one or more transactions in each server node, wherein each of the one or more transactions involves a set of data modifying operations to be committed to a plurality of storage resources atomically;

coordinate transactions initiated in a particular server node by the transaction manager instance operating in that node; and

maintain transaction logs for each transaction manager instance in highly available storage, wherein each transaction log contains an intermediate status of each initiated but uncommitted transaction of the one or more transactions managed by the transaction manager and wherein the log is usable by another transaction manager instance to recover the uncommitted transactions if the transaction manager fails;

detect a failure of one of the plurality of application server instances of the cluster;

in response to said detect, select one or more surrogate transaction manager instances from one or more of the application server instances of the server cluster that are still operational;

read, by the one or more selected surrogate transaction manager instances, one or more transaction logs of the failed application server and to complete one or more uncommitted transactions indicated by the one or more transaction logs of the failed application server; and

attempt, by one or more application server instances having the one or more selected surrogate transaction manager instances, to recover each in-flight transaction entry in a transaction log of the failed server, wherein said attempting comprises, for each log entry, opening connections to all data sources involved in the transaction corresponding to the log entry, and if all connections are successful, recover the transaction.

29. The computer accessible storage medium as recited in claim 28, wherein the separate transaction manager instance is one of a plurality of separate transaction manager instances on one of the server nodes, wherein the program instructions are further computer-executable to operate one of the separate transaction manager instances for each of a plurality of applications that initiates a transaction on that one of the server nodes.

30. The computer accessible storage medium as recited in claim 28, wherein the server nodes of the cluster are a plurality of application server nodes, wherein the highly available storage is accessible to each of the plurality of application server nodes of the server cluster, and wherein the highly available storage stores the transaction logs for the plurality of application server nodes of the cluster.

31. The computer accessible storage medium as recited in claim 28, wherein the program instructions are further computer-executable to detect a failure of an application server including one of the separate transaction manager instances on one of the server nodes within the application server cluster.

32. The computer accessible storage medium as recited in claim 31, wherein the program instructions are further computer-executable to, in response to restarting of the failed application server of the application server cluster, attempt to recover in flight transactions for an initial predetermined period of time.

33. The computer accessible storage medium as recited in claim 32, wherein the program instructions are further computer-executable to, in response to one or more transactions not being recovered within the initial predetermined period of time, delay recovery of one or more transactions not recovered within the initial period and resume normal application server operations while transaction recovery is still in progress.

34. The computer accessible medium storage as recited in claim 33, wherein the program instructions are further computer-executable to, in response to un-recovered transactions remaining after a maximum number of deferred recovery attempts has been made, generate an error requesting operator intervention.

35. The computer accessible medium storage as recited in claim 31, wherein the program instructions are further computer-executable to designate one or more surrogates for the failed application server and to attempt to recover in flight transactions of the failed application server for an initial predetermined period of time.

36. The computer accessible storage medium as recited in claim 35, wherein the program instructions are further computer-executable to implement deferred transaction recovery if one or more transactions are not recovered within the initial period, wherein deferred transaction recovery attempts to recover transactions subsequent to the initial period and after normal application server operations have resumed.

37. The computer accessible storage medium as recited in claim 36, wherein the program instructions are further computer-executable to, if un-recovered transactions remain after a maximum number of deferred recovery attempts has been made, generate an error requesting operator intervention.

38. The computer accessible storage medium as recited in claim 31, wherein the failure of the application server is detected by a heartbeat communications protocol.

39. The computer accessible storage medium as recited in claim 31, wherein the failure of the application server is detected by a polling communications protocol.

40. The computer accessible storage medium as recited in claim 35, wherein the surrogate application server is selected from remaining operational application servers of the cluster based on an identity of the failed application server.

41. The computer accessible storage medium as recited in claim 35, wherein ownership of the transaction logs is transferred from the failed application server to the one or more surrogates as part of a transaction.