DISTRIBUTED TRANSACTION PROCESSING

• US 20110041006A1
• Filed: 03/05/2010
• Published: 02/17/2011
• Est. Priority Date: 08/12/2009
• Status: Active Grant

• Current Assignee: New Technology/Enterprise, Ltd.
• Original Assignee: New Technology/Enterprise, Ltd.
• Inventors: Fowler, Matthew
• Granted Patent: US 8,838,534 B2
• Time in Patent Office: Days
• US Class Current: 714/10
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 9/466 Transaction processing
• CPC Class Codes: G06F 9/5072 Grid computing

Abstract

A system and method for processing a distributed transaction for an application are disclosed. Conventionally transactions on critical data (e.g. financial information) are processed using a database architecture whereby a persistent database (typically a redundant disk array) comprises the master record. In cases where large amounts of data need to be accessed but absolute data integrity is less critical, for example search engines, processing is conducted on live in-memory data without all the data being backed up, which can be much faster but data can be lost when processors fail. There have been attempts to use data grid architectures with some backup to persistent stores for more important data but these have either introduced disk access bottlenecks or required manual intervention in the event of failure. Aspects of the invention provide committal of distributed transactions independently of persistent storage which can make the speed advantages of data grid computing available for high volume distributed transactions, without losing the reliability of conventional database systems. The methods, systems and architecture are independent of the nature of the application or data.

Claims

1. A method of processing a distributed transaction associated with an application, the application comprising application data stored in an in-memory data grid comprising a plurality of processing elements each having respective associated memory for storing a respective portion of the application data, the method comprising:

initiating a distributed transaction by a client connected to the in-memory data grid;

executing the distributed transaction by processing elements of the in-memory data grid, the executing comprising performing processing at multiple processing elements of the grid, the processing resulting in changes to be made to application data stored in memory at respective processing elements;

transmitting processing result information defining the changes to be made to the application data from respective processing elements to a transaction manager during execution of the distributed transaction, the transaction manager storing transaction data enabling the state of the distributed transaction to be reconstructed at a processing element after an error occurring at the processing element during execution of the distributed transaction;

in response to an indication from the client, committing the changes to the in-memory application data at the processing elements, whereby modified application data elements are made available for subsequent access by the application; and

updating a persistent copy of application data stored on a persistent storage medium based on the processing result information received by the transaction manager.

2. A method according to claim 1, comprising reconstructing the state of the distributed transaction at a processing element, after an error at the processing element during execution of the distributed transaction, using information received from the transaction manager.

3. A method according to claim 2, wherein the reconstructing step comprises sending a status query from the processing element to the transaction manager, and receiving a status response indicating the status of the distributed transaction.

4. A method according to claim 3, wherein the information received from the transaction manager used in reconstructing the state of the distributed transaction comprises processing result information previously transmitted to the transaction manager by the processing element.

5. A method according to claim 4, wherein the status response includes the previously transmitted processing result information.

6. A method according to claim 4, comprising committing changes to the in-memory application data at the processing element based on the processing result information received from the transaction manager.

7. A method according to claim 2, wherein the reconstructing step further uses information backed up to a backup processing element associated with the processing element.

8. A method according to claim 1, wherein the error comprises non-receipt at the processing element of a commit or abort instruction from the transaction manager.

9. A method according to claim 1, wherein the error comprises a failure of the processing element, the reconstructing step comprising switching the processing for the processing element to a backup processing element, and reconstructing the state of the transaction at the backup processing element, preferably using one or more of:

transaction status information received from the transaction manager;

processing result information received from the transaction manager; and

application state previously backed up to the backup processing element.

10. A method according to claim 1, wherein the processing result information is transmitted from respective processing elements before the changes are committed at the processing elements.

11. A method according to claim 1, wherein updating of the persistent copy of application data and committing of changes at the processing elements occur independently, preferably asynchronously.

12. A method according to claim 1, wherein the persistent copy of application data is updated after committing the changes at the processing elements.

13. A method according to claim 1, comprising, in response to the commit indication received at the transaction manager from the client, transmitting a commit acknowledgement to the client indicating that the distributed transaction is deemed committed, preferably whereby the client treats the transaction as committed after receiving the commit acknowledgement.

14. A method according to claim 13, wherein the transmission of the acknowledgement occurs asynchronously with one or both of the committing of changes at the processing elements and the updating of the persistent copy of application data at the persistent storage medium.

15. A method of processing a distributed transaction associated with an application, the application comprising application data stored in an in-memory data grid comprising a plurality of processing elements each having respective associated memory for storing a respective portion of the application data, the method comprising:

initiating a distributed transaction by a client connected to the in-memory data grid;

executing the distributed transaction by processing elements of the in-memory data grid, the executing comprising performing processing at multiple processing elements of the grid, the processing resulting in changes to be made to application data stored in memory at respective processing elements;

transmitting processing result information defining the changes to be made to the application data from respective processing elements to a transaction manager;

in response to an indication from the client, committing the changes to the in-memory application data at the processing elements, whereby modified application data elements are made available for subsequent access by the application;

asynchronously with the committing step, updating a persistent copy of application data stored on a persistent storage medium based on the processing result information received by the transaction manager.

16. A method according to claim 15, comprising transmitting an acknowledgement to the client that the distributed transaction is deemed committed asynchronously with, preferably prior to, one or both of the committing step and the updating step.

17. A method according to claim 15, wherein the persistent copy is updated after the changes are committed at the processing elements.

18. A method according to claim 15, wherein the changes made to application data at processing elements comprise first changes to one or more data elements stored in memory at the processing elements, the method comprising executing a subsequent transaction involving second changes to one or more of the previously modified data elements prior to updating the persistent storage medium based on the first changes.

19. A method according to claim 15, wherein the processing result information is transmitted before the changes are committed at the processing elements.

20. A method according to claim 15, comprising collecting processing results relating to multiple calls to a processing element during the distributed transaction, and transmitting the collected processing results to the transaction manager as a group.

21. A method according to claim 15, comprising transmitting a registration message from a processing element to the transaction manager, and registering the processing element as a participant in the distributed transaction at the transaction manager in response to the message, the message preferably including the processing results.

22. A method according to claim 21, comprising, in response to a commit instruction received at the transaction manager, transmitting a commit message to each processing element registered at the transaction manager as a participant in the distributed transaction.

23. A method according to claim 15, comprising storing the processing results received from processing elements in a buffer associated with the transaction manager.

24. A method according to claim 23, wherein the buffer is held in memory associated with the transaction manager.

25. A method according to claim 23, comprising updating the persistent storage medium based on information stored in the buffer.

26. A method according to claim 15, comprising, after receiving at the transaction manager a notification from a processing element indicating that the processing element has recovered from a failure, transmitting processing results previously received from the processing element and stored in the buffer to the processing element.

27. A method according to claim 15, wherein one or more, preferably each of the processing elements are each associated with a primary processing device and one or more backup processing devices.

28. A method according to claim 27, comprising, in response to failure of a primary processing device, switching processing for a processing element to an associated backup processing device, and preferably transmitting a notification to the transaction manager indicating recovery from a failure.

29. A method according to claim 15, wherein performing processing at a processing element comprises executing a local transaction at the processing element, and committing the local transaction in response to a commit message received from the transaction manager.

30. A method according to claim 15, comprising querying the transaction manager by a processing element to determine the distributed transaction status if no commit or abort message is received within a specified time limit, and transmitting a commit or abort message to the processing element in response to the query.

31. A method according to claim 15, wherein each processing element comprises:

a managed data area for storing application data; and

code for performing processing using data stored in the managed data area.

32. A method according to claim 31, wherein code stored at processing elements implements one or more services in a service-oriented architecture.

33. A method according to claim 31, wherein code stored at processing elements implements a messaging or event-driven application interface.

34. A method according to claim 31, comprising backing up the managed data area to one or more backup processing elements, preferably wherein the backup is updated when a local transaction is committed.

35. A method according to claim 15, wherein the processing elements are software processing elements, preferably virtual machines, arranged to run on processing devices.

36. A method according to claim 35, comprising assigning two or more software processing elements to a processing device.

37. A method according to claim 35, comprising, in response to a change in processing load at one or more processing devices, modifying the assignment of software processing elements to processing devices.

38. A method according to claim 15, comprising partitioning the application data across different processing elements based on content of the data.

39. A method according to claim 38, comprising grouping entities in the application data based on relationships between the entities and/or based on application functionality associated with the entities, and assigning different entity groups to different processing elements.

40. A method according to claim 15, comprising selecting a data field associated with a data entity in the application data as a key, partitioning the application data into partitions based on values of the key;

and assigning data partitions to processing elements of the data grid.

41. A method according to claim 40, comprising routing a data access request or service call specifying a key value to one of a plurality of processing elements in dependence on the key value.

42. A method according to claim 40 wherein the key is associated with a group of related entities.

43. A method according to claim 15, wherein updating the persistent storage medium comprises processing the sets of processing results by a plurality of concurrent update processes, the method preferably further comprising identifying dependencies between sets of processing results stored in the buffer of the transaction manager, and processing the sets of processing results by the plurality of concurrent update processes in an order based on identified dependencies.

72. A computer program or computer program product or computer readable medium comprising software code adapted, when executed on a data processing apparatus, to perform a method as claimed in claim 15.

44. A method of processing a transaction, comprising:

initiating a transaction by a client;

performing processing at a processing element connected to the client, the processing element storing a plurality of data elements in a data area in memory, the processing resulting in changes to one or more of the data elements stored in the in-memory data area;

transmitting processing result information specifying the changes to the data elements from the processing element to a transaction manager;

receiving an instruction from the client to commit the transaction;

committing the changes to the in-memory data area at the processing element, whereby modified data elements are made available for access by other transactions; and

updating a persistent storage medium storing persistent copies of the data elements based on the processing result information received by the transaction manager.

45. A method according to claim 44, comprising transmitting an acknowledgement to the client indicating that the transaction is deemed committed prior to updating the persistent storage medium and preferably prior to committing the changes to the in-memory data area at the processing element.

46. A method of performing processing at a processing element, the processing element having access to data elements stored in a data area in memory associated with the processing element;

the method comprising;

receiving a message to initiate processing;

performing processing in response to the message, the processing resulting in changes to one or more data elements in the in-memory data area;

transmitting processing result information specifying the changes to a transaction manager;

receiving a commit message from the transaction manager; and

committing the changes to the in-memory data area in response to the commit message.

47. A method according to claim 46, wherein performing processing comprises starting a local transaction;

and wherein committing the changes comprises committing the local transaction.

48. A method according to claim 46, wherein the processing element is associated with a primary processing device, the method comprising providing one or more backup processing devices;

and backing up the in-memory data area associated with the processing element to in-memory data area(s) of the one or more backup processing devices.

49. A method according to claim 48, comprising updating the backup of the data area at the one or more backup processing devices when changes are committed to the data area at the primary processing device.

50. A method according to claim 48, comprising switching processing for the processing element to one of the backup processing devices in response to failure of the primary processing device.

51. A method according to claim 50, comprising sending a status request to the transaction manager after the processing element has recovered from a failure.

52. A method according to claim 51, comprising receiving transaction status information from the transaction manager, the transaction status information preferably including processing result information previously transmitted to the transaction manager, and updating the state of the processing element based on the received information.

53. A method according to claim 46, comprising storing processing result information relating to processing performed in response to multiple received messages at the processing element, and transmitting the stored processing result information to the transaction manager as a group and/or in a single message.

54. A method according to claim 46, wherein processing in response to a given message is performed in a thread, the method comprising collecting processing result information for multiple threads associated with a given transaction, and transmitting the collected processing result information to the transaction manager when no more threads for the transaction are executing at the processing element.

55. A method according to claim 54, comprising storing a counter for counting processing threads associated with a given transaction;

incrementing the counter when a new processing thread is initiated for the transaction at the processing element; and

decrementing the counter when a processing thread for the transaction completes;

wherein processing result information collected for a plurality of threads associated with the transaction is preferably transmitted to the transaction manager when the counter is decremented to a value indicating that no threads remain executing for the transaction.

56. A method according to claim 46, wherein the processing element is a software processing element, preferably in the form of or including a virtual machine, the software processing element preferably further comprising one or more of:

a service interface for initiation of processing at the processing element;

a data interface for modifying data elements in the data area under a local transaction; and

a transaction control interface for transmitting processing result information to the transaction manager and receiving transaction control messages from the transaction manager.

57. A method of managing a transaction by a transaction manager, comprising:

receiving from one or more processing elements participating in a transaction, processing results defining changes to be made to application data stored at the processing elements;

receiving from a client an instruction to commit the transaction;

performing commit processing to commit the transaction, the commit processing comprising transmitting a commit message to each of the processing elements instructing the processing elements to commit the application data changes at the processing elements;

asynchronously with performing the commit processing, outputting update information for updating a copy of the application data stored in a storage medium separate from the processing elements, the update information based on the processing results received from the processing elements.

58. A method according to claim 57, comprising transmitting the update information to a persistent storage medium storing a persistent copy of the application data, to update the persistent copy of the application data in accordance with the received processing results.

59. A method according to claim 57, comprising transmitting update information to a plurality of external systems and/or persistent storage media.

60. A method according to claim 57, wherein application data is stored at the processing elements in memory.

61. A method according to claim 57, comprising transmitting a commit response to the client specifying that the transaction has been committed prior to outputting the update information.

62. A method according to claim 57, comprising outputting the update information after the application data changes have been committed at the processing elements.

63. A method according to claim 57, comprising storing the received processing results in a buffer associated with the transaction manager.

64. A method according to claim 63, wherein the outputting step comprises adding items representing sets of processing results for the transaction to a queue of processing results to be persisted to the persistent storage medium in the order the processing results were received at the transaction manager.

65. A method according to claim 64, comprising selecting, by an update process, a set of processing results in the queue and persisting the changes defined in the selected processing results to the persistent storage medium.

66. A method according to claim 65, comprising identifying dependencies between sets of processing results, and wherein the selecting step is performed based on identified dependencies, the dependencies preferably identifying sets of processing results relating to common data elements.

67. A method according to claim 65, comprising accessing the queue by a plurality of concurrent update processes, each process preferably selecting a next available set of processing results for persisting based on:

the order in which processing results were placed on the queue; and

any identified dependencies.

68. A method according to claim 57, comprising providing a plurality of transaction managers configured to manage transactions concurrently;

preferably wherein transactions are allocated to transaction managers based on a transaction identifier.

69. A method according to claim 57, comprising providing one or more backup transaction managers associated with the transaction manager or with each transaction manager, and switching to a backup transaction manager on failure of a transaction manager.

70. A method according to claim 69, comprising storing transaction information for a transaction managed by a transaction manager in an in-memory data area associated with the transaction manager;

and backing up the transaction information to an in-memory data area of a backup transaction manager.

71. A method according to claim 70, comprising storing processing results received from processing elements in a log file on permanent storage;

and

restoring processing results from the log file to a transaction manager, preferably after restart in the event of failure of the transaction manager and all associated backup transaction managers before processing results have been persisted.

73. A processing system comprising:

an in-memory data grid for storing application data for an application and processing distributed transactions operating on the application data;

a persistent data storage medium for persistently storing application data; and

a transaction manager for receiving information specifying changes made to the in-memory data grid during processing of distributed transactions by the application and applying the changes to the persistent data storage medium.

74. A processing system comprising:

an in-memory data grid for storing application data for an application, the in-memory data grid comprising a plurality of processing elements, each configured to store a portion of the application data in memory and to perform processing actions relating to the portion of the application data as part of distributed transactions;

a persistent data storage medium for persistently storing application data; and

a transaction manager for managing distributed transactions executed by the application, wherein the transaction manager is configured to;

receive information from each of a plurality of processing elements participating in a distributed transaction specifying application data changes resulting from processing performed by the respective processing element; and

commit the distributed transaction by a process including;

receiving a message instructing the transaction manager to commit the transaction; and

after receipt of the message, transmitting messages to each participating processing element to commit data changes to the respective processing element'"'"'s in-memory application data;

wherein the transaction manager is further configured, after the transaction has been committed, to update the persistent data storage medium based on the received information specifying application data changes.

75. A processing system according to claim 74, wherein each processing element is configured to store a given portion of the application data allocated to the processing element, and wherein at least one processing element stores the allocated portion of data entirely in memory.

76. A processing system according to claim 74, wherein each processing element is configured to store a given portion of the application data allocated to the processing element, and wherein at least one processing element stores a subset of the allocated portion of data in memory.

77. A processing system according to claim 76, wherein the at least one processing element is configured to fetch required data not currently stored in memory into memory when access to the data is requested by the application and/or when data changes are to be committed at the processing element.

78. A processing element comprising:

a data interface for accessing and modifying data elements in an in-memory data area associated with the processing element, the data interface configured to store separately from the data elements update information specifying changes made to data elements in the data area; and

a transaction control interface for transmitting the update information to a transaction manager.

79. A processing element according to claim 78, comprising a processing module for performing processing at the processing element, the processing module configured to modify data elements in the data area using the data interface.

80. A processing element according to claim 79, wherein the data interface provides a transactional interface for modifying data elements, the processing module configured to modify data elements in the in-memory data area using a transaction.

81. A processing element according to claim 80, configured to commit the transaction to the in-memory data area in response to a commit message received from the transaction manager.

82. A processing element according to claim 78, configured to reconstruct the state of a transaction at the processing element after an error using information received from the transaction manager.

83. A processing element according to claim 78, comprising code routines for execution by a virtual machine, the in-memory data area associated with the virtual machine.

84. A processing element according to claim 78, comprising software code for execution on a processing device and/or a processing device having a processor and memory for storing the in-memory data area.

85. A transaction buffer manager, comprising:

a buffer for storing transaction data for a transaction executed by one or more processing elements connected to the transaction buffer manager;

a transaction buffer interface for receiving processing results from the one or more processing elements defining changes to be made to application data stored in memory at the processing elements and storing the received processing results in the buffer; and

a persistence interface for updating a copy of the application data stored in a persistent storage medium based on the processing results stored in the buffer.

86. A transaction buffer manager according to claim 85, wherein the transaction buffer interface is configured to transmit a commit message to each of the processing elements instructing the processing elements to commit the application data changes at the processing elements.

87. A transaction buffer manager according to claim 85, configured to transmit a commit acknowledgement message to the client;

and wherein the persistence interface is configured to update the persistent copy of the application data asynchronously with, preferably after, the sending of the commit acknowledgement message to the client.

88. A transaction buffer manager according to claim 85, comprising software code for execution on a processing device and/or a processing device having a processor and associated memory for storing the buffer.