System and method for efficiently writing data from an in-memory database to a disk database

• US 7,113,953 B2
• Filed: 06/30/2003
• Issued: 09/26/2006
• Est. Priority Date: 06/30/2003
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Brown, Kevin, Spicer, Michael John Elvery
• Primary Examiner(s): Gaffin, Jeffrey
• Assistant Examiner(s): PYO, MONICA M
• Application Number:
  US10/611,552
• Publication Number:
  US 20040267776A1
• Time in Patent Office: 1,184 Days
• Field of Search: 707/100, 707/101, 707/102, 707/206, 707/1
• US Class Current: 707/615
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

A system and associated method write data from an in-memory database to a disk database in an efficient manner and with a relatively short lag time. The integration of data from in-memory to disk is achieved by limiting the operations of the in-memory database to insert only. The system shortens lag time by reducing the number of transactions required to transfer data from in-memory database to disk memory. The system compiles into an RDBMS, knowledge about the structure of the in-memory database, and then uses the end of the transaction callbacks from the RDBMS to keep the in-memory database and the disk memory in synch. The RDBMS includes a daemon that runs periodically to find records in the in-memory database that have not yet been written to the RDBMS, and then writes the found records to the RDBMS as part of a single transaction. If the transaction completes successfully, the in-memory database is updated to reflect which records have been “flushed” to the RDBMS. If the transaction fails, no action is taken. The present system synchronizes the in-memory database with the RDBMS.

Claims

1. A processor-implemented method of efficiently writing records from an in-memory database to a disk database comprising:

linking the records by a linked list;

creating a header data structure of the linked records;

linking a new record in the in-memory database to the header data structure;

transferring the records in the linked list and the new record from the in-memory database to the disk database using the header data structure;

wherein transferring the records includes transferring blocks of the records as a single transaction to the disk database, in order to minimize a number of transfers from the in-memory database to the disk database;

dynamically setting a last commit pointer and a last flush pointer to keep track of;

(i) a block of the records that have been transferred to the disk database but not yet committed;

(ii) a block of the records that have not been transferred to the disk database; and

(iii) a block of the records that have been transferred and committed to the disk database, in order to maintain synchronization between the in-memory database and the disk database; and

determining whether a committing operation of the block of the records that have been transferred to the disk database but not yet committed is successful, and if the committing operation is determined to be unsuccessful, deeming the committing operation of all the records in the block of the records that have been transferred to the disk database but not yet committed is successful, to have failed.

2. The method of claim 1, wherein each record contains information about a single transaction.

3. The method of claim 1, wherein the header data structure comprises an entity name that identifies the new record.

4. The method of claim 3, wherein the header data structure further comprises a last commit pointer.

5. The method of claim 4, wherein the header data structure further comprises a last flush pointer.

6. The method of claim 5, wherein the header data structure further comprises a head pointer.

7. The method of claim 6, wherein the header data structure further comprises a tail pointer.

8. The method of claim 7, further comprising locating a new header data structure for the new record.

9. The method of claim 8, further comprising determining if the head pointer in the header data structure of the new record points to another record.

10. The method of claim 9, wherein if the head pointer in the header data structure of the new record points to another record getting the other record.

11. The method of claim 7, further comprising determining if the last commit pointer of the header data structure points at a record.

12. The method of claim 11, further comprising setting the last commit pointer equal to the head pointer if the last commit pointer does not point to a record.

13. The method of claim 11, further comprising getting the record to which the last commit pointer points.

14. The method of claim 13, further comprising determining whether there is a last record after the record to which the last commit pointer points.

15. The method of claim 14, further comprising writing to the disk database all records in the link list that occur after the record to which the last commit pointer points.

16. The method of claim 15, further comprising setting a last flush pointer of the header data structure equal to a last count if the writing of the records to the disk memory ended successfully.

17. The method of claim 16, further comprising setting the last flush pointer equal to the last commit pointer if the writing of the records to the disk memory did not end successfully.

20. The computer program product of claim 17, wherein the header data structure comprises an entity name that identifies the new record.

21. The computer program product of claim 20, wherein the header data structure further comprises a last commit pointer.

22. The computer program product of claim 21, wherein the header data structure further comprises a last flush pointer.

23. The computer program product of claim 22, wherein the header data structure further comprises a head pointer.

24. The computer program product of claim 23, wherein the header data structure further comprises a tail pointer.

18. A computer program product having instruction codes stored on a computer-readable medium for efficiently writing records from an in-memory database to a disk database, comprising:

a set of instruction codes for linking the records by a linked list;

a set of instruction codes for creating a header data structure of the linked records;

a set of instruction codes for linking a new record in the in-memory database to the header data structure; and

a set of instruction codes for transferring the records in the linked list and the new record from the in-memory database to the disk database using the header data structure;

wherein the set of instruction codes for transferring the records includes a set of instruction codes for transferring blocks of the records as a single transaction to the disk database, in order to minimize a number of transfers from the in-memory database to the disk database;

a set of instruction codes for dynamically setting a last commit pointer and a last flush pointer to keep track of;

(i) a block of the records that have been transferred to the disk database but not yet committed;

(ii) a block of the records that have not been transferred to the disk database; and

(iii) a block of the records that have been transferred and committed to the disk database, in order to maintain synchronization between the in-memory database and the disk database; and

a set of instruction codes for determining whether a committing operation of the block of the records that have been transferred to the disk database but not yet committed is successful, and if the committing operation is determined to be unsuccessful, deeming the committing operation of all the records in the block of the records that have been transferred to the disk database but not yet committed is successful, to have failed.

19. The computer program product of claim 18, wherein each record contains information about a single transaction.

25. A processor-implemented system for efficiently writing records from an in-memory database to a disk database, comprising:

means for linking the records by a linked list;

means for creating a header data structure of the linked records;

means for linking a new records in the in-memory database to the header data structure;

means for transferring records in the linked list and the new record from the in-memory database to the disk database using the header data structure;

wherein the means for transferring the records includes a means for transferring blocks of records as a single transaction to the disk database, in order to minimize a number of transfers from the in-memory database to the disk database;

means for dynamically setting a last commit pointer and a last flush pointer to keep track of;

(i) a block the records that have been transferred to the disk database but not yet committed;

(ii) a block the records that have not been transferred to the disk database; and

(iii) a block of the records that have been transferred and committed to the disk database, in order to maintain synchronization between the in-memory database and the disk database; and

means for determining whether a committing operation of the block of the records that have been transferred to the disk database but not yet committed is successful, and if the committing operation is determined to be unsuccessful, deeming the committing operation of all the records in the block of the records that have been transferred to the disk database but not yet committed is successful, to have failed.

26. The system of claim 25, wherein each record contains information about a single transaction.

27. The system of claim 25, wherein the header data structure comprises an entity name that identifies the new record.

28. The system of claim 27, wherein the header data structure further comprises a last commit pointer.

29. The system of claim 28, wherein the header data structure further comprises a last flush pointer and a head pointer.