Transfer of data from positional data sources to partitioned databases in restartable environments

• US 8,156,084 B2
• Filed: 01/17/2008
• Issued: 04/10/2012
• Est. Priority Date: 01/17/2008
• Status: Expired due to Fees

• Current Assignee: GlobalFoundries, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Jang, Jing-Song, McArdle, James Michael, Spicer, Michael John Elvery
• Primary Examiner(s): Pham, Khanh
• Assistant Examiner(s): CHEEMA, AZAM M
• Application Number:
  US12/016,039
• Publication Number:
  US 20090187787A1
• Time in Patent Office: 1,545 Days
• Field of Search: None
• US Class Current: 707/684
• CPC Class Codes: G06F 11/1443 Transmit or communication errors
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity

Abstract

Method, computer program product, and system for transferring data from positional data sources to partitioned databases are provided. A record is read from a positional data source. The record is to be written to one of a plurality of partitions of a database. A position of the record in the positional data source is obtained. A transaction is initiated to write the record to the one partition and to store the position of the record in the database. The transaction is committed after the record is successfully written to the one partition and the position of the record is successfully stored in the database.

Claims

1. A computer-implemented method for transferring data from data sources to database partitions, the method comprising:

reading a record from a data source in a computer system, the record having a position that uniquely identifies the record in the data source relative to a starting position of the data source, wherein the record is to be written to one of a plurality of partitions of a database in the computer system, wherein the record is one of a plurality of records written to the one partition and the position of the record is farthest from the starting position of the data source among the plurality of records written to the one partition;

obtaining the position of the record in the data source;

initiating a transaction to write the record to the one partition and to store the position of the record in the database;

committing the transaction after the record is successfully written to the one partition and the position of the record is successfully stored in the database; and

upon a restart following a failure, avoiding writing the record of the committed transaction to the one partition by rejecting any record with a position that is less than or equal to the position of the committed transaction.

2. The method of claim 1, further comprising:

rolling back the transaction when the record is not successfully written to the one partition or the position of the record is not successfully stored in the database.

3. The method of claim 1, further comprising:

responsive to a restart after a failure, using the position of the record stored in the database to determine which position in the data source to restart reading a record from the data source.

4. The method of claim 3, wherein both the record and at least one record that is written to a different partition on the database are stored into the database in parallel, and wherein determining which position to restart reading from the data source after the failure further comprises:

determining whether the position of the record is closer to the starting position of the data source than a position of the at least one record, wherein the position to restart reading from the data source is the closer position.

5. The method of claim 1, wherein the plurality of records are written to the one partition in order and the record is a last of the plurality of records written to the one partition.

6. The method of claim 1, wherein the position of the record is stored in the one partition of the database.

7. The method of claim 1, wherein the data source is a file.

8. A system for transferring data from data sources to database partitions, the system comprising:

a processor; and

a data engine executing on the processor, the data engine

reading a record from a data source, the record having a position that uniquely identifies the record in the data source relative to a starting position of the data source, wherein the record is to be written to one of a plurality of partitions of a database, wherein the record is one of a plurality of records written to the one partition and the position of the record is farthest from the starting position of the data source among the plurality of records written to the one partition,

obtaining the position of the record in the data source,

initiating a transaction to write the record to the one partition and to store the position of the record in the database,

committing the transaction after the record is successfully written to the one partition and the position of the record is successfully stored in the database;

upon a restart following a failure, avoiding writing the record of the committed transaction to the one partition by rejecting any record with a position that is less than or equal to the position of the committed transaction.

9. The system of claim 8, wherein the data engine further

rolls back the transaction when the record is not successfully written to the one partition or the position of the record is not successfully stored in the database.

10. The system of claim 8, wherein responsive to a restart after a failure, the data engine further comprises:

using the position of the record stored in the database to determine which position in the data source to restart reading a record from the data source.

11. The system of claim 10, wherein both the record and at least one record that is written to a different partition on the database are written into the database in parallel, and wherein determining which position to restart reading from the data source after the failure further comprises:

determining whether the position of the record is closer to the starting position of the data source than a position of the at least one record, wherein the position to restart reading from the data source is the closer position.

12. The system of claim 8, wherein the plurality of records are written to the one partition in order and the record is a last of the plurality of records written to the one partition.

13. The system of claim 8, wherein the position of the record is stored in the one partition of the database.

14. The system of claim 8, wherein the data source is a file.

15. A computer readable storage medium encoded with a computer program for transferring data from data sources to database partitions,

wherein the computer program, when executed on a computer, causes the computer to:

read a record from a data source, the record having a position that uniquely identifies the record in the data source relative to a starting position of the data source,

wherein the record is to be written to one of a plurality of partitions of a database,

wherein the record is one of a plurality of records written to the one partition and the position of the record is farthest from a starting position of the data source among the plurality of records written to the one partition;

obtain the position of the record in the data source;

initiate a transaction to write the record to the one partition and to store the position of the record in the database;

commit the transaction after the record is successfully written to the one partition and the position of the record is successfully stored in the database; and

upon a restart following a failure, avoiding writing the record of the committed transaction to the one partition by rejecting any record with a position that is less than or equal to the position of the committed transaction.

16. The medium of claim 15, wherein the computer program further causes the computer to:

roll back the transaction when the record is not successfully written to the one partition or the position of the record is not successfully stored in the database.

17. The medium of claim 15, wherein responsive to a restart after a failure, the computer program further causes the computer to:

use the position of the record stored in the database to determine which position in the data source to restart reading a record from the data source.

18. The medium of claim 15, wherein the plurality of records are written to the one partition in positional order and the record is a last of the plurality of records written to the one partition.

19. The medium of claim 15, wherein the position of the record is stored in the one partition of the database.