Set-oriented real-time data processing based on transaction boundaries

• US 20050125436A1
• Filed: 12/03/2003
• Published: 06/09/2005
• Est. Priority Date: 12/03/2003
• Status: Active Grant

• Current Assignee: Informatica LLC (Informatica, Inc. (California))
• Original Assignee: Informatica Corporation
• Inventors: To, Raymond G., Mudunuri, Gautam H.
• Granted Patent: US 7,254,590 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2428 Query predicate definition using graphical user interfaces, including menus and forms G06F16/2423 takes precedence
• CPC Class Codes: G06F 16/24547 Optimisations to support specific applications; Extensibility of optimisers
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

A method is provided for real-time data processing based on data sets or data sets defined by applicable rules or user-defined logic. Also provided is a real-time data processing system that implements such method. Transaction boundaries are set and propagated through cascades of multiple data transformations on multitudes of data records or rows. The method may be implemented in conjunction with a relational database management system (DBMS), object DBMS, or any other DBMS. The method and system for real-time data processing may be advantageously applied in Extract, Transform, and Load (ETL), Enterprise Application Integration (EAI), and Enterprise Information Integration (EII).

Claims

1. A method for processing a plurality of data records, comprising:

setting transaction boundaries among said plurality of data records thereby dividing the plurality of data records into one or more data sets;

processing each of the data set thereby producing a multiplicity of results from the one or more transaction sets; and

completing the processing of the plurality of data records by synchronizing the transaction boundaries and combining said multiplicity of results.

2. The method of claim 1, wherein the setting of transaction boundaries is performed based on the row count of the data records.

3. The method of claim 1, wherein the setting of transaction boundaries is performed based on the time stamp of the data records.

4. The method of claim 1, wherein the setting of transaction boundaries is performed based on the result of a previous data transformation.

5. The method of claim 1, wherein the setting of transaction boundaries is performed based on a user-defined logic, wherein the user-defined logic is one or more rules defined by a user.

6. The method of claim 5, wherein the user-defined logic is on a real-time basis.

7. The method of claim 5, wherein the rules comprise one or more tables in a database.

8. The method of claim 5, wherein the rules comprise one or more statements defining relationships and actions in a suitable programming language.

9. The method of claim 8, wherein the suitable programming language is one of Generation III Languages (3GL), Generation IV Languages (4GL), and Generation V (5GL) Languages.

10. The method of claim 8, wherein the suitable programming language is an expert system tool.

11. The method of claim 1, wherein said processing comprises at least one of insert, update, delete, aggregation, rank, sort, sequence, and join.

12. A method for performing a series of transformations on a plurality of data records, wherein said series of transformations initiate at a source and conclude at a target, said method comprises:

setting transaction boundaries among said plurality of data records at said source thereby dividing the plurality of data records into one or more data sets;

propagating the transaction boundaries through the series of transformations from the source to the target;

performing said series of transformations based on the one or more data sets thereby producing a multiplicity of results from said series of set-based transformations; and

completing the series of transformations by synchronizing the transaction boundaries and combining said multiplicity of results.

13. The method of claim 12, wherein the setting of transaction boundaries is performed based on the row count of the data records.

14. The method of claim 12, wherein the setting of transaction boundaries is performed based on the time stamp of the data records.

15. The method of claim 12, wherein the setting of transaction boundaries is performed based on the result of a previous data transformation.

16. The method of claim 12, wherein the setting of transaction boundaries is performed based on a user-defined logic, wherein the user-defined logic is one or more rules defined by a user.

17. The method of claim 16, wherein the user-defined logic is on a real-time basis.

18. The method of claim 16, wherein the rules comprise one or more tables in a database.

19. The method of claim 16, wherein the rules comprise one or more statements defining relationships and actions in a suitable programming language.

20. The method of claim 19, wherein the suitable programming language is one of Generation III Languages (3GL), Generation IV Languages (4GL), and Generation V Languages (5GL).

21. The method of claim 19, wherein the suitable programming language is an expert system tool.

22. The method of claim 12, wherein the propagating comprises setting and maintaining one or more transaction queues capable of defining the boundaries of the data sets.

23. The method of claim 22, wherein the transaction queues comprise one or more tables in a database.

24. The method of claim 22, wherein said transaction queues are maintained in a computer memory.

25. The method of claim 12, wherein said series of transformations comprise at least one of insert, update, delete, aggregation, rank, sort, sequence, and join.

26. A system for processing a plurality of data records, comprising:

means for setting transaction boundaries among said plurality of data records thereby dividing the plurality of data records into one or more data sets;

means for processing each of said data set thereby producing a multiplicity of results from the one or more data sets; and

means for synchronizing the transaction boundaries and combining said multiplicity of results thereby completing said processing.

27. The system of claim 26, wherein the means for setting transaction boundaries defines the transaction boundaries based on the row count of the data records.

28. The system of claim 26, wherein the means for setting transaction boundaries defines the transaction boundaries based on the time stamp of the data records.

29. The system of claim 26, wherein the means for setting transaction boundaries defines the transaction boundaries based on the result of a previous data transformation.

30. The system of claim 26, the means for setting transaction boundaries comprises defines the transaction boundaries based on a user-defined logic, wherein the user-defined logic is one or more rules defined by a user.

31. The system of claim 30, wherein the user-defined logic is on a real-time basis.

32. The system of claim 30, wherein the rules comprise one or more tables in a database.

33. The system of claim 30, wherein the rules comprise one or more statements defining relationships and actions in a suitable programming language.

34. The system of claim 33, wherein the suitable programming language is one of Generation III Languages (3GL), Generation IV Languages (4GL), and Generation V Languages (5GL).

35. The system of claim 33, wherein the suitable programming language is an expert system tool.

36. The system of claim 26, wherein said processing comprises at least one of insert, update, delete, aggregation, rank, sort, sequence, and join.

51. A computer program product implementing the system of claim 26.

37. A system for performing a series of transformations on a plurality of data records, wherein said series of transformations initiate at a source and conclude at a target, said system comprises:

means for setting transaction boundaries among said plurality of data records at the source thereby dividing the plurality of data records into one or more data sets;

means for propagating the transaction boundaries through the series of transformations from the source to the target;

means for performing said series of transformations based on the one or more data sets thereby producing a multiplicity of results from said series of set-based transformations; and

means for synchronizing the transaction boundaries and combining the multiplicity of results thereby completing the series of transformations.

38. The system of claim 37, wherein the means for setting transaction boundaries defines the transaction boundaries based on the row count of the data records.

39. The system of claim 37, wherein the means for setting transaction boundaries defines the transaction boundaries based on the time stamp of the data records.

40. The system of claim 37, wherein the means for setting transaction boundaries defines the transaction boundaries based on the result of a previous data transformation.

41. The system of claim 38, the means for setting transaction boundaries defines the transaction boundaries based on a user-defined logic, wherein the user-defined logic is one or more rules defined by a user.

42. The system of claim 41, wherein the user-defined logic is on a real-time basis.

43. The system of claim 41, wherein the rules comprise one or more tables in a database.

44. The system of claim 41, wherein the rules comprise one or more statements defining relationships and actions in a suitable programming language.

45. The system of claim 44, wherein the suitable programming language is one of Generation III Languages (3GL), Generation IV Languages (4GL), and Generation V Languages (5GL).

46. The system of claim 44, wherein the suitable programming language is an expert system tool.

47. The system of claim 37, wherein the means for propagating the transaction boundaries comprises setting and maintaining one or more transaction queues capable of defining the boundaries of the data sets.

48. The system of claim 47, wherein said transaction queues comprise one or more tables in a database.

49. The system of claim 47, wherein said transaction queues are maintained in a computer memory.

50. The system of claim 37, wherein said series of transformations comprise at least one of insert, update, delete, aggregation, rank, sort, sequence, and join.

52. A computer program product implementing the system of claim 37.

53. A computer readable medium having recorded thereon program instructions which when processed by a computer are capable of executing a method for processing a plurality of data records, said method comprising:

setting transaction boundaries among said plurality of data records thereby dividing the plurality of data records into one or more data sets;

processing each of said data set thereby producing a multiplicity of results from the one or more data sets; and

completing the processing of said plurality by synchronizing the transaction boundaries and combining said multiplicity of results.

54. A computer readable medium having recorded thereon program instructions which when processed by a computer are capable of executing a method for performing a series of transformations on a plurality of data records, said method comprising:

setting transaction boundaries among said plurality of data records thereby dividing the plurality of data records into one or more data sets;

propagating the transaction boundaries through said series of transformations;

performing said series of transformations based on the one or more data sets thereby producing a multiplicity of results from said series of set-based transformations; and

completing the series of transformations by synchronizing the transaction boundaries and combining said multiplicity of results.