Transfer of data from transactional data sources to partitioned databases in restartable environments
First Claim
1. A method for transferring data from a transactional data source to a database, the database managed by a database management system (DBMS), the method comprising:
- reading a message from the transactional data source, the message comprising one or more records to be inserted into the database comprising a plurality of partitions, wherein each of the plurality of partitions is a logical division of the database that is defined based on one or more partitioning keys in at least one column of a table of the database, and wherein the transactional data source comprises a queue of data packets that is independent and distinct from both the database and the DBMS;
initiating a first transaction, using a single thread, to both (i) insert the one or more records into a partition of the plurality of partitions and (ii) delete the message from the transactional data source, wherein the message comprising the one or more records remains in the transactional data source until the first transaction is committed;
upon determining that both (i) the insertion of the one or more records into the partition and (ii) the deletion of the message from the transactional data source have successfully completed, committing, using the single thread, both (i) the insertion into the partition and (ii) the deletion of the message from the transactional data source; and
upon determining that at least one of (i) the insertion of the one or more records into the partition and (ii) the deletion of the message from the transactional data source has failed, rolling back, using the single thread, both (i) the insertion into the partition and (ii) the deletion of the message from the transactional data source.
3 Assignments
0 Petitions
Accused Products
Abstract
Method, system, and computer program product for transferring data from transactional data sources to partitioned databases are provided. One or more messages from a transactional data source are read. Each message includes one or more records to be inserted into a database comprising a plurality of partitions. One of the partitions of the database in which the one or more records of each message are to be inserted is then identified. A transaction is initiated to insert the one or more records of each message into the one partition and to delete the one or more messages from the transactional data source. The transaction is committed only when insertion of the one or more records of each message into the one partition and deletion of the one or more messages from the transactional data source succeed.
-
Citations
14 Claims
-
1. A method for transferring data from a transactional data source to a database, the database managed by a database management system (DBMS), the method comprising:
-
reading a message from the transactional data source, the message comprising one or more records to be inserted into the database comprising a plurality of partitions, wherein each of the plurality of partitions is a logical division of the database that is defined based on one or more partitioning keys in at least one column of a table of the database, and wherein the transactional data source comprises a queue of data packets that is independent and distinct from both the database and the DBMS; initiating a first transaction, using a single thread, to both (i) insert the one or more records into a partition of the plurality of partitions and (ii) delete the message from the transactional data source, wherein the message comprising the one or more records remains in the transactional data source until the first transaction is committed; upon determining that both (i) the insertion of the one or more records into the partition and (ii) the deletion of the message from the transactional data source have successfully completed, committing, using the single thread, both (i) the insertion into the partition and (ii) the deletion of the message from the transactional data source; and upon determining that at least one of (i) the insertion of the one or more records into the partition and (ii) the deletion of the message from the transactional data source has failed, rolling back, using the single thread, both (i) the insertion into the partition and (ii) the deletion of the message from the transactional data source. - View Dependent Claims (3, 4, 5, 6, 7, 11, 12)
-
-
2. A computer program product comprising a computer readable medium, the computer readable medium including a computer program for transferring data from a transactional data source to a database, the database managed by a database management system (DBMS), wherein the computer program, when executed on a computer, causes the computer to:
-
read a message from the transactional data source, the message comprising one or more records to be inserted into the database comprising a plurality of partitions, wherein each of the plurality of partitions is a logical division of the database that is defined based on one or more partitioning keys in at least one column of a table of the database, and wherein the transactional data source comprises a queue of data packets that is independent and distinct from both the database and the DBMS; initiate a first transaction, using a single thread, to both (i) insert the one or more records into a partition of the plurality of partitions and (ii) delete the message from the transactional data source, wherein the message comprising the one or more records remains in the transactional data source until the first transaction is committed; upon determining that both (i) the insertion of the one or more records into the partition and (ii) the deletion of the message from the transactional data source have successfully completed, commit, using the single thread, both (i) the insertion into the partition and (ii) the deletion of the message from the transactional data source; and upon determining that at least one of (i) the insertion of the one or more records into the partition and (ii) the deletion of the message from the transactional data source has failed, roll back, using the single thread, both (i) the insertion into the partition and (ii) the deletion of the message from the transactional data source. - View Dependent Claims (8, 9, 10, 13, 14)
-
Specification