Processing transactions in graph-based applications

• US 8,667,329 B2
• Filed: 12/15/2009
• Issued: 03/04/2014
• Est. Priority Date: 09/25/2009
• Status: Active Grant

• Current Assignee: AB Initio Orginal Works LLC
• Original Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Inventors: Douros, Bryan Phil, Atterbury, Matthew Darcy, Stanfill, Craig W., Wholey, Joseph Skeffington III, Bromley, H. Mark
• Primary Examiner(s): TRUONG, LOAN
• Application Number:
  US12/638,588
• Publication Number:
  US 20110078500A1
• Time in Patent Office: 1,540 Days
• Field of Search: 714/15, 707/703, 706/45
• US Class Current: 714/15
• CPC Class Codes: G06F 11/1474 in transactions G06F16/20 takes precedence
• CPC Class Codes: G06F 9/4494 data driven
• CPC Class Codes: G06F 9/466 Transaction processing

Abstract

A graph-based computation has a plurality of nodes representing graph components connected by one or more links representing data flows between the components. Preparing the computation includes: identifying at least a first set of components to process a plurality of transactions in the computation; and associating with a first transaction a first group of one or more data records and data operations corresponding to the data records, and associating with a second transaction a second group of one or more data records and data operations corresponding to the data records. Executing the computation to process at least the first and second transactions includes delaying execution by the first set of components of a data operation corresponding to an initial data record of the second group until after execution by the first set of components of a data operation corresponding to a final data record of the first group.

Claims

1. A computer-implemented method, including:

preparing a graph-based computation having a plurality of nodes representing graph components connected by one or more links representing data flows between the graph components, including;

identifying at least a first set of multiple graph components to process a plurality of transactions in the computation; and

associating with a first transaction a first group of one or more data records and data operations corresponding to the data records, and associating with a second transaction a second group of one or more data records and data operations corresponding to the data records; and

executing the graph-based computation to process at least the first and second transactions, including delaying execution by the first set of multiple graph components of a data operation corresponding to an initial data record of the second group until after execution by the first set of multiple graph components of a data operation corresponding to a final data record of the first group.

2. The method of claim 1 wherein executing the graph-based computation further includes:

executing the first and second transactions as a first batch of transactions by;

processing the data records and data operations corresponding to the first and second transactions; and

upon detecting a failure, rolling back the first batch of transactions by rolling back all processed data records and data operations corresponding to the processed transactions.

3. The method of claim 1 further including:

specifying a first graph component having at least an output port as a beginning of a data flow for the first set of multiple graph components; and

specifying a second graph component having at least an input port as an end of the data flow for the first set of multiple graph components.

4. The method of claim 1 further including dynamically invoking the first set of multiple graph components to process the one or more transactions.

5. The method of claim 1 further including, upon successfully processing substantially all transactions in the first batch of transactions, committing the first batch of transactions.

6. The method of claim 5 in which committing the first batch of transactions includes committing all the data operations corresponding to each one of the transactions in the first batch of transactions at substantially the same time.

7. The method of claim 2 in which detecting a failure includes detecting a failure of any of the data operations corresponding to each one of the data records in a transaction of the first batch of transactions.

8. The method of claim 2 further including after detecting the failure, re-processing, as a second different batch of transactions, unprocessed transactions of the first batch of transactions and the processed transactions that were not committed during processing of the first batch of transactions.

9. The method of claim 2 further including, after detecting the failure, re-processing, as a series of discrete batches, each one of the processed and unprocessed transactions associated with the first batch of transactions.

10. The method of claim 2 further including:

upon detecting the failure, identifying one or more transactions that failed as failed transactions,

re-processing, as a second different batch, the first batch of transactions excluding the failed transactions, and

re-processing, as third batch, the failed transactions.

11. The method of claim 2 further including, in the course of executing the first transaction, performing one or more data operations in a first database by a set of graph components belonging to the graph-based computation.

12. The method of claim 11 further including, in the course of executing the first transaction, establishing a first session with the first database, the first session being shared by one or more of a first set of graph components processing the first transaction.

13. The method of claim 2 in which a size of the first transactions is specified by a user.

14. A non-transitory computer-readable non-transitory storage device storing a computer program, the computer program including instructions for causing a computer to:

prepare a graph-based computation having a plurality of nodes representing graph components connected by one or more links representing data flows between the graph components, including;

identifying at least a first set of multiple graph components to process a plurality of transactions in the computation; and

associating with a first transaction a first group of one or more data records and data operations corresponding to the data records, and associating with a second transaction a second group of one or more data records and data operations corresponding to the data records; and

execute the graph-based computation to process at least the first and second transactions, including delaying execution by the first set of multiple graph components of a data operation corresponding to an initial data record of the second group until after execution by the first set of multiple graph components of a data operation corresponding to a final data record of the first group.

15. The non-transitory computer-readable non-transitory storage device of claim 14 wherein executing the graph-based computation further includes:

executing the first and second transactions as a first batch of transactions by;

processing the data records and data operations corresponding to the first and second transactions; and

upon detecting a failure, rolling back the first batch of transactions by rolling back all processed data records and data operations corresponding to the processed transactions.

16. The non-transitory computer-readable non-transitory storage device of claim 14, wherein the computer program further includes instructions for causing a computer to:

specify a first graph component having at least an output port as a beginning of a data flow for the first set of multiple graph components; and

specify a second graph component having at least an input port as an end of the data flow for the first set of multiple graph components.

17. The non-transitory computer-readable non-transitory storage device of claim 14 wherein the computer program further includes instructions for causing a computer to dynamically invoke the first set of multiple graph components to process the one or more transactions.

18. The non-transitory computer-readable non-transitory storage device of claim 14 wherein the computer program further includes instructions for causing a computer to, upon successfully processing substantially all transactions in the first batch of transactions, commit the first batch of transactions.

19. The non-transitory computer-readable non-transitory storage device of claim 18 in which committing the first batch of transactions includes committing all the data operations corresponding to each one of the transactions in the first batch of transactions at substantially the same time.

20. The non-transitory computer-readable non-transitory storage device of claim 15 in which detecting a failure includes detecting a failure of any of the data operations corresponding to each one of the data records in a transaction of the first batch of transactions.

21. The non-transitory computer-readable non-transitory storage device of claim 15 wherein the computer program further includes instructions for causing a computer to, after detecting the failure, re-process, as a second different batch of transactions, unprocessed transactions of the first batch of transactions and the processed transactions that were not committed during processing of the first batch of transactions.

22. The non-transitory computer-readable non-transitory storage device of claim 15 wherein the computer program further includes instructions for causing a computer to, after detecting the failure, re-process, as a series of discrete batches, each one of the processed and unprocessed transactions associated with the first batch of transactions.

23. The non-transitory computer-readable non-transitory storage device of claim 15 wherein the computer program further includes instructions for causing a computer to:

upon detecting the failure, identify one or more transactions that failed as failed transactions,

re-process, as a second different batch, the first batch of transactions excluding the failed transactions, and

re-process, as third batch, the failed transactions.

24. The non-transitory computer-readable non-transitory storage device of claim 15 wherein the computer program further includes instructions for causing a computer to, in the course of executing the first transaction, perform one or more data operations in a first database by a set of graph components belonging to the graph-based computation.

25. The non-transitory computer-readable non-transitory storage device of claim 24 wherein the computer program further includes instructions for causing a computer to, in the course of executing the first transaction, establish a first session with the first database, the first session being shared by one or more of a first set of graph components processing the first transaction.

26. The non-transitory computer-readable non-transitory storage device of claim 15 in which a size of the first transactions is specified by a user.

27. A system, including:

a computing system, including;

means for preparing a graph-based computation having a plurality of nodes representing graph components connected by one or more links representing data flows between the graph components, including;

identifying at least a first set of multiple graph components to process a plurality of transactions in the computation; and

associating with a first transaction a first group of one or more data records and data operations corresponding to the data records, and associating with a second transaction a second group of one or more data records and data operations corresponding to the data records; and

means for executing the graph-based computation to process at least the first and second transactions, including delaying execution by the first set of multiple graph components of a data operation corresponding to an initial data record of the second group until after execution by the first set of multiple graph components of a data operation corresponding to a final data record of the first group.

28. The system of claim 27 wherein the means for executing the graph-based computation further includes:

means for executing the first and second transactions as a first batch of transactions by;

processing the data records and data operations corresponding to the first and second transactions; and

upon detecting a failure, rolling back the first batch of transactions by rolling back all processed data records and data operations corresponding to the processed transactions.

29. The system of claim 27 further including:

means for specifying a first graph component having at least an output port as a beginning of a data flow for the first set of multiple graph components; and

means for specifying a second graph component having at least an input port as an end of the data flow for the first set of multiple graph components.

30. The system of claim 27 further including means for dynamically invoking the first set of multiple graph components to process the one or more transactions.

31. The system of claim 27 further including means for, upon successfully processing substantially all transactions in the first batch of transactions, committing the first batch of transactions.

32. The system of claim 31 in which committing the first batch of transactions includes committing all the data operations corresponding to each one of the transactions in the first batch of transactions at substantially the same time.

33. The system of claim 28 in which detecting a failure includes detecting a failure of any of the data operations corresponding to each one of the data records in a transaction of the first batch of transactions.

34. The system of claim 28 further including means for, after detecting the failure, re-processing, as a second different batch of transactions, unprocessed transactions of the first batch of transactions and the processed transactions that were not committed during processing of the first batch of transactions.

35. The system of claim 28 further including means for, after detecting the failure, re-processing, as a series of discrete batches, each one of the processed and unprocessed transactions associated with the first batch of transactions.

36. The system of claim 28 further including:

means for, upon detecting the failure, identifying one or more transactions that failed as failed transactions,

means for re-processing, as a second different batch, the first batch of transactions excluding the failed transactions, and

means for re-processing, as third batch, the failed transactions.

37. The system of claim 28 further including means for, in the course of executing the first transaction, performing one or more data operations in a first database by a set of graph components belonging to the graph-based computation.

38. The system of claim 37 further including means for, in the course of executing the first transaction, establishing a first session with the first database, the first session being shared by one or more of a first set of graph components processing the first transaction.

39. The system of claim 28 in which a size of the first transactions is specified by a user.

40. A computing system, including:

at least one processor configured to;

prepare a graph-based computation having a plurality of nodes representing graph components connected by one or more links representing data flows between the graph components, including;

identifying at least a first set of multiple graph components to process a plurality of transactions in the computation; and

associating with a first transaction a first group of one or more data records and data operations corresponding to the data records, and associating with a second transaction a second group of one or more data records and data operations corresponding to the data records; and

execute the graph-based computation to process at least the first and second transactions, including delaying execution by the first set of multiple graph components of a data operation corresponding to an initial data record of the second group until after execution by the first set of multiple graph components of a data operation corresponding to a final data record of the first group.

41. The computing system of claim 40 wherein executing the graph-based computation further includes:

executing the first and second transactions as a first batch of transactions by;

processing the data records and data operations corresponding to the first and second transactions; and

upon detecting a failure, rolling back the first batch of transactions by rolling back all processed data records and data operations corresponding to the processed transactions.

42. The computing system of claim 40 wherein the processor is further configured to:

specify a first graph component having at least an output port as a beginning of a data flow for the first set of multiple graph components; and

specify a second graph component having at least an input port as an end of the data flow for the first set of multiple graph components.

43. The computing system of claim 40 wherein the processor is further configured to dynamically invoke the first set of mulitple graph components to process the one or more transactions.

44. The computing system of claim 40 wherein the processor is further configured to, upon successfully processing substantially all transactions in the first batch of transactions, commit the first batch of transactions.

45. The computing system of claim 44 in which committing the first batch of transactions includes committing all the data operations corresponding to each one of the transactions in the first batch of transactions at substantially the same time.

46. The computing system of claim 41 in which detecting a failure includes detecting a failure of any of the data operations corresponding to each one of the data records in a transaction of the first batch of transactions.

47. The computing system of claim 41 wherein the processor is further configured to, after detecting the failure, re-process, as a second different batch of transactions, unprocessed transactions of the first batch of transactions and the processed transactions that were not committed during processing of the first batch of transactions.

48. The computing system of claim 41 wherein the processor is further configured to, after detecting the failure, re-process, as a series of discrete batches, each one of the processed and unprocessed transactions associated with the first batch of transactions.

49. The computing system of claim 41 wherein the processor is further configured to:

upon detecting the failure, identify one or more transactions that failed as failed transactions,

re-process, as a second different batch, the first batch of transactions excluding the failed transactions, and

re-process, as third batch, the failed transactions.

50. The computing system of claim 41 wherein the processor is further configured to, in the course of executing the first transaction, perform one or more data operations in a first database by a set of graph components belonging to the graph-based computation.

51. The computing system of claim 50 wherein the processor is further configured to, in the course of executing the first transaction, establish a first session with the first database, the first session being shared by one or more of a first set of graph components processing the first transaction.

52. The computing system of claim 41 in which a size of the first transactions is specified by a user.