System and method for conversion of JMS message data into database transactions for application to multiple heterogeneous databases

• US 9,047,392 B2
• Filed: 12/05/2013
• Issued: 06/02/2015
• Est. Priority Date: 07/23/2010
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Wilkes, Stephen, Nielsen, Michael Scott, Pora, Codin
• Primary Examiner(s): LE, HUNG D
• Application Number:
  US14/098,124
• Publication Number:
  US 20140172886A1
• Time in Patent Office: 544 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2452 Query translation
• CPC Class Codes: G06F 16/90 Details of database functions independent of the retrieved data types
• CPC Class Codes: G06F 16/901 Indexing; Data structures therefor; Storage structures for retrieval from the web G06F16/951

Abstract

A system and method for performing real-time conversion of data which is present in the form of messages on JMS-compliant or other messaging systems into database transactions; which can then subsequently be applied to multiple heterogeneous databases or other systems. In accordance with an embodiment, the invention provides a means by which data can be read from messages in a messaging system, and converted to a set of database operations that are then stored as a persistent trail file. The operations can then be routed via a network and applied to target systems as required, for example to maintain a replicated set of information at one or more different or heterogeneous systems. The data in the message can be formatted in a variety of ways, for example, as fixed width, delimited, or XML data, and the system can be configured to convert this data as appropriate.

Claims

1. A system for integrating a message-based system with a database-based system, comprising:

a computer including a microprocessor;

a data parsing component, provided as a shared library to integrate a dataflow of a source system, wherein the data parsing component includes a plurality of parsers, wherein each of the plurality of parsers is configured to parse messages received from a message-based system having a particular message format; and

a properties file which indicates properties and data definitions that define how the system is to parse messages from a source format of the source system, to a target format of a target system;

wherein the data parsing component operates to

read a message from a messaging topic at the source system,

present values read from a header of the message to the properties file,

select, using the properties file, a parser from the plurality of parsers to parse the message,

parse, using the selected parser and the properties file, the message to determine transaction boundaries and operation types,

store the parsed message in a persistent trail file, which is subsequently routed via a network to the target system,

generate a source definition file from properties in the properties file, which is provided to the target system, and

apply the parsed message as database operations at a database at the target system, wherein, if the target system needs to be recovered from an earlier point in time, the persistent trail is deciphered using the source definition file to replay the database operations.

2. The system of claim 1, wherein the message read from the message topic is in a transaction fashion and includes data changes at the source system.

3. The system of claim 1, wherein the message topic is either existing or newly created.

4. The system of claim 1, wherein the parsed message is stored in binary data in the trail file.

5. The system of claim 4, wherein the trail file includes transaction semantics corresponding to the data changes at the source system.

6. The system of claim 1, wherein the message read from the message topic is in a data format selected from a plurality of data formats comprising XML, fixed width, or delimited data.

7. The system of claim 1, wherein the message is put on the message topic using Java Message Service techniques.

8. A method for integrating data changes from a source system to a target system, comprising:

providing a data parsing component as a shared library to integrate a dataflow of a source system, wherein the data parsing component includes a plurality of parsers, wherein each of the plurality of parsers is configured to parse messages received from a message-based system having a particular message format; and

providing a properties file which indicates properties and data definitions that define how the system is to parse messages from a source format of the source system, to a target format of a target system;

wherein the data parsing component operates to

read a message from a messaging topic at the source system,

present values read from a header of the message to the properties file,

select, using the properties file, a parser from the plurality of parsers to parse the message,

parse, using the selected parser and the properties file, the message to determine transaction boundaries and operation types,

store the parsed message in a persistent trail file, which is subsequently routed via a network to the target system,

generate a source definition file from properties in the properties file, which is provided to the target system, and

apply the parsed message as database operations at a database at the target system, wherein, if the target system needs to be recovered from an earlier point in time, the persistent trail is deciphered using the source definition file to replay the database operations.

9. The method of claim 8, wherein the message read from the message topic is in a transaction fashion and includes data changes at the source system.

10. The method of claim 8, wherein the message topic is either existing or newly created.

11. The method of claim 8, wherein the parsed message is stored in binary data in the trail file.

12. The method of claim 11, wherein the trail file includes transaction semantics corresponding to the data changes at the source system.

13. The method of claim 8, wherein the message read from the message topic is in a data format selected from a plurality of data formats comprising XML, fixed width, or delimited data.

14. The method of claim 8, wherein the topic is put on the message topic using Java Message Service techniques.

15. A non-transitory computer readable storage medium, including instructions stored thereon which when read and executed by a computer cause the computer to perform the steps comprising:

providing a data parsing component as a shared library to integrate a dataflow of a source system, wherein the data parsing component includes a plurality of parsers, wherein each of the plurality of parsers is configured to parse messages received from a message-based system having a particular message format; and

providing a properties file which indicates properties and data definitions that define how the system is to parse messages from a source format of the source system, to a target format of a target system;

wherein the data parsing component operates to

read a message from a messaging topic at the source system,

present values read from a header of the message to the properties file,

select, using the properties file, a parser from the plurality of parsers to parse the message,

parse, using the selected parser and the properties file, the message to determine transaction boundaries and operation types,

store the parsed message in a persistent trail file, which is subsequently routed via a network to the target system,

generate a source definition file from properties in the properties file, which is provided to the target system, and

apply the parsed message as database operations at a database at the target system, wherein, if the target system needs to be recovered from an earlier point in time, the persistent trail is deciphered using the source definition file to replay the database operations.

16. The non-transitory computer readable storage medium of claim 15, wherein the message read from the message topic is in a transaction fashion and includes data changes in data at the source system.

17. The non-transitory computer readable storage medium of claim 15, wherein the message topic is either existing or newly created.

18. The non-transitory computer readable storage medium of claim 15, wherein the parsed message is stored in binary data in the trail file.

19. The non-transitory computer readable storage medium of claim 18, wherein the trail file includes transaction semantics corresponding to the data changes at the source system.

20. The non-transitory computer readable storage medium of claim 15, wherein the message read from the message topic is om a data format selected from a plurality of data formats comprising XML, fixed width, or delimited data.