Hybrid table implementation by using buffer pool as permanent in-memory storage for memory-resident data

• US 10,114,908 B2
• Filed: 11/13/2012
• Issued: 10/30/2018
• Est. Priority Date: 11/13/2012
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Parkkinen, Jarmo, Raatikka, Vilho T., Ruuth, Jarmo K., Soini, Petri U., Wolski, Antoni
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): Santos, Pedro J
• Application Number:
  US13/675,634
• Publication Number:
  US 20140136510A1
• Time in Patent Office: 2,177 Days
• Field of Search: 707706, 707711, 707712, 707713, 707696
• CPC Class Codes: G06F 16/119 Details of migration of file systems migration mechanisms in storage systems G06F3/0647
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/901 Indexing; Data structures therefor; Storage structures for retrieval from the web G06F16/951

Abstract

Embodiments relate to a method, system, and computer program product for data management. An aspect includes a search index that is created for a database. The search index has a reference value of a first or a second type associating data to a storage location in a first memory or a second memory respectively. Responsive to a data retrieval request, data is accessed using the reference value from the first or the second memory. If there is a data migration between the first and second memory storage, the reference values are then recalculated and altered in the search index accordingly.

Claims

1. A data management method, comprising:

generating, by a processor of a computing system, a database comprising a database table that is stored in both a first memory comprising an in-memory storage and a second memory comprising a disk memory storage, wherein the database table comprises a plurality of rows, wherein a first portion of the rows is stored in the in-memory storage and a second portion of the rows is stored in the disk memory storage, wherein the subset of the rows of the database table stored in the disk memory device is not stored in the in-memory storage, wherein the database table is indexed by a search index having a plurality of search index reference values each of a first type or second type, the first type being a page pointer to the in-memory storage in a buffer pool, the second type being a page disk address of a page in the disk memory storage that is to be proceed by said buffer pool, and wherein the search index has densely indexed rows that correspond to the subset of the rows of the database stored in the in-memory storage and sparsely indexed rows that correspond to the subset of the rows of the database stored in the disk memory storage;

responsive to a data retrieval request, accessing, by the processor, data from said first or said second memory using a reference value of the plurality of search index reference values, wherein said reference value is of said second type, wherein accessing said data includes determining, by the processor, whether said data is accessible from said buffer pool and then using said reference value of said second type to access said data from said buffer pool if it is determined that said data is accessible from said buffer pool; and

altering said plurality of search index reference values from page pointers to the in-memory storage to page disk addresses in the disk memory in response to data being migrated from the in-memory storage to the disk memory storage based on a least recently used queue of the in-memory storage.

2. The method of claim 1, further comprising using said reference value of said second type to access said particular data from a buffer pool if it is determined that said particular data is accessible from said buffer pool.

3. The method of claim 1, wherein said search index is a tree structured index.

4. The method of claim 3, wherein said tree has leaf nodes.

5. The method of claim 3, wherein data access is provided using said page pointer or said page disk address according to said reference value provided.

6. The method of claim 4, wherein said leaf node indexing reference value for m-data in said permanent memory is a pointer to a page in said buffer and said leaf node indexing reference value for d-data is a disk address.

7. The method of claim 4, wherein said search index is a binary index structure.

8. The method of claim 3, wherein said index has a plurality of rows and columns and said rows have associated keys.

9. The method of claim 8, wherein said ordering of keys correspond to said ordering of rows.

10. The method of claim 6, wherein said migration from one component to another can be accomplished on a page replacement mechanism and said page is being generated by said buffer.

11. The method of claim 10, wherein said migration from one component to another can be accomplished on a least recently used basis.

12. The method of claim 6, wherein said migration from one component to another can be accomplished on an explicit basis.

13. A computer program product for data management the computer program product comprising a non-transitory computer readable storage medium having computer readable program code embodied therewith, said computer readable program code being executable by a computer to:

generate a database comprising a database table that is stored in both a first memory comprising an in-memory storage and a second memory comprising a disk memory storage, wherein the database table comprises a plurality of rows, wherein a first portion of the rows is stored in the in-memory storage and a second portion of the rows is stored in the disk memory storage, wherein the subset of the rows of the database table stored in the disk memory device is not stored in the in-memory storage, wherein the database table is indexed by a search index having a plurality of search index reference values each of a first type or second type, the first type being a page pointer to the in-memory storage in a buffer pool or a second type that is a page disk address of a page in the disk memory storage that is to be proceed by said buffer pool, and wherein the search index has densely indexed rows that correspond to the subset of the rows of the database table stored in the in-memory storage and sparsely indexed rows that correspond to the subset of the rows of the database table stored in the disk memory storage;

responsive to a data retrieval request, access data from said first or said second memory using a reference value of the plurality of search index reference values, wherein said reference value is of said second type, wherein accessing said data includes determining, by the processor, whether said data is accessible from said buffer pool and then using said reference value of said second type to access said data from said buffer pool if it is determined that said data is accessible from said buffer pool; and

alter said plurality of search index reference values from page pointers to the in-memory storage to page disk addresses in the disk memory in response to data being migrated from the in-memory storage to the disk memory storage based on a least recently used queue of the in-memory storage.

14. The computer program product of claim 13, wherein said index has a plurality of rows and columns and said index has associated keys corresponding to each row.

15. The computing program product of claim 14, wherein indexing granularity is a page and said index is enabled to have an additional on-page access.

16. The computer program product of claim 15, wherein said index is a sparse index and said d and m pages are identical in structure except for content.

17. A system comprising:

a database that comprises data stored in both an in-memory storage and a disk memory storage that is in processing communication with the in-memory storage, the database comprising a database table comprising a plurality of rows, wherein a first portion of the rows is stored in the in-memory device storage and a second portion of the rows is stored in the disk memory storage, wherein the subset of the rows of the database stored in the disk memory storage is not stored in the in-memory storage;

a buffer in processing communication with said in-memory and disk memory storage, said buffer including a buffer pool manager for handling buffer pool; and

a database search engine having at least a processor for managing a data search index structure, said search index having a plurality of reference values each of a first or a second type such associating data to said in-memory and disk memory storage, wherein the search index has densely indexed rows that correspond to the subset of the rows of the database stored in the in-memory storage and sparsely indexed rows that correspond to the subset of the rows of the database stored in the disk memory storage;

said reference value of first type being a page pointer to in-memory storage in said buffer pool and said reference value of second type being a page disk address to be processed by said buffer pool;

wherein the database search engine is configured to;

access data from said first or said second memory using a reference value of the plurality of reference values, wherein said reference value is of said second type, wherein accessible from said data includes determining, by the processor, whether said data is accessible from said buffer pool and then using said reference value of said second type to access said data from said buffer pool if it is determined that said data is accessible from said buffer pool; and

alter said plurality of search index reference values from page pointers to the in-memory storage to page disk addresses in the disk memory in response to data being migrated from the in-memory storage to the disk memory storage, wherein the data is migrated based on a least recently used queue of the in-memory storage.