TRANSFER OF DATA FROM POSITIONAL DATA SOURCES TO PARTITIONED DATABASES IN RESTARTABLE ENVIRONMENTS

• US 20090187787A1
• Filed: 01/17/2008
• Published: 07/23/2009
• Est. Priority Date: 01/17/2008
• Status: Active Grant

• Current Assignee: GlobalFoundries, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Mcardle, James Michael, JANG, Jing-Song, Spicer, Michael John Elvery
• Granted Patent: US 8,156,084 B2
• Time in Patent Office: Days
• US Class Current: 714/16
• CPC Class Codes: G06F 11/1443 Transmit or communication errors
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity

Abstract

Method, computer program product, and system for transferring data from positional data sources to partitioned databases are provided. A record is read from a positional data source. The record is to be written to one of a plurality of partitions of a database. A position of the record in the positional data source is obtained. A transaction is initiated to write the record to the one partition and to store the position of the record in the database. The transaction is committed after the record is successfully written to the one partition and the position of the record is successfully stored in the database.

Claims

1. A method for transferring data from positional data sources to partitioned databases, the method comprising:

reading a record from a positional data source, wherein the record is to be written to one of a plurality of partitions of a database;

obtaining a position of the record in the positional data source;

initiating a transaction to write the record to the one partition and to store the position of the record in the database; and

committing the transaction after the record is successfully written to the one partition and the position of the record is successfully stored in the database.

2. The method of claim 1, further comprising:

rolling back the transaction when the record is not successfully written to the one partition or the position of the record is not successfully stored in the database.

3. The method of claim 1, further comprising:

responsive to a restart after a failure, using the position of the record stored in the database to determine which position in the positional data source to restart reading a record from the positional data source.

4. The method of claim 1, wherein the record is one of a plurality of records written to the one partition and the position of the record is farthest from a starting position of the positional data source among the plurality of records written to the one partition.

5. The method of claim 4, wherein the plurality of records are written to the one partition in positional order and the record is a last of the plurality of records written to the one partition.

6. The method of claim 1, wherein the position of the record is stored in the one partition of the database.

7. The method of claim 1, wherein the positional data source is a file.

8. A system for transferring data from positional data sources to partitioned databases, the system comprising:

a processor; and

a data engine executing on the processor, the data engine

reading a record from a positional data source, wherein the record is to be written to

one of a plurality of partitions of a database,

obtaining a position of the record in the positional data source,

initiating a transaction to write the record to the one partition and to store the position

of the record in the database, and

committing the transaction after the record is successfully written to the one partition and the position of the record is successfully stored in the database.

9. The system of claim 8, wherein the data engine further

rolls back the transaction when the record is not successfully written to the one partition or the position of the record is not successfully stored in the database.

10. The system of claim 8, wherein responsive to a restart after a failure, the data engine further

uses the position of the record stored in the database to determine which position in the positional data source to restart reading a record from the positional data source.

11. The system of claim 8, wherein the record is one of a plurality of records written to the one partition and the position of the record is farthest from a starting position of the positional data source among the plurality of records written to the one partition.

12. The system of claim 11, wherein the plurality of records are written to the one partition in positional order and the record is a last of the plurality of records written to the one partition.

13. The system of claim 8, wherein the position of the record is stored in the one partition of the database.

14. The system of claim 8, wherein the positional data source is a file.

15. A computer program product comprising a computer readable medium encoded with a computer program for transferring data from positional data sources to partitioned databases, wherein the computer program, when executed on a computer, causes the computer to:

read a record from a positional data source, wherein the record is to be written to one of a plurality of partitions of a database;

obtain a position of the record in the positional data source;

initiate a transaction to write the record to the one partition and to store the position of the record in the database; and

commit the transaction after the record is successfully written to the one partition and the position of the record is successfully stored in the database.

16. The computer program product of claim 15, wherein the computer program further causes the computer to:

roll back the transaction when the record is not successfully written to the one partition or the position of the record is not successfully stored in the database.

17. The computer program product of claim 15, wherein responsive to a restart after a failure, the computer program further causes the computer to:

use the position of the record stored in the database to determine which position in the positional data source to restart reading a record from the positional data source.

18. The computer program product of claim 15, wherein the record is one of a plurality of records written to the one partition and the position of the record is farthest from a starting position of the positional data source among the plurality of records written to the one partition.

19. The computer program product of claim 18, wherein the plurality of records are written to the one partition in positional order and the record is a last of the plurality of records written to the one partition.

20. The computer program product of claim 15, wherein the position of the record is stored in the one partition of the database.