Full and partial materialization of data from an in-memory array to an on-disk page structure

• US 10,311,048 B2
• Filed: 11/17/2015
• Issued: 06/04/2019
• Est. Priority Date: 11/25/2014
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP SE
• Inventors: Wein, David
• Primary Examiner(s): Saeed, Usmaan
• Assistant Examiner(s): Weinrich, Brian E.
• Application Number:
  US14/944,068
• Publication Number:
  US 20170139981A1
• Time in Patent Office: 1,295 Days
• Field of Search: 707615
• CPC Class Codes: G06F 16/1774 Locking methods, e.g. locking methods for file systems allowing shared and concurrent access to files
• CPC Class Codes: G06F 16/219 Managing data history or versioning querying versioned data G06F16/2474; querying temporal data G06F16/2477
• CPC Class Codes: G06F 16/2308 Concurrency control transaction processing G06F9/466
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/2329 using versioning
• CPC Class Codes: G06F 16/2336 Pessimistic concurrency control approaches, e.g. locking or multiple versions without time stamps
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/2393 Updating materialised views
• CPC Class Codes: G06F 16/24539 using cached or materialised query results
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

Changes to data in a column in an in memory array resulting from operations of a transaction thread can be written to a database object associated with the column. The database object can be marked for materialization upon completion of all of the plurality of operations of the transaction thread such that the column and the changes are materialized from the in-memory array to one or more pages stored on a persistent storage based on the marking.

Claims

1. A method for implementation by one or more data processors forming part of at least one computing system, the method comprising:

maintaining an in-memory array in main system memory of the at least one computing system, the in-memory array comprising a column of a database table forming part of an in-memory database system;

writing, to a database object associated with the column, one or more changes to the column resulting from one or more operations executed on the column by a transaction thread comprising a plurality of operations;

marking the database object for materialization in response to completion of all of the plurality of operations of the transaction thread;

allocating, by the transaction thread, one or more pages; and

materializing, based on the marking, the column from the in-memory array to the one or more pages stored on a persistent storage, the materializing comprising applying the one or more changes written in the database object, the one or more pages comprising one or more blocks for storing the column,

wherein the database object forms part of an object tree corresponding to fragments in a delta store, the object tree including fragment objects in a fragment level, columns in a column level, and dictionaries in a dictionary level;

wherein the persisting is iteratively performed at a fragment level of the in-memory array, at a column level of the in-memory array, and at a dictionary level of the in-memory array;

wherein the materializing comprises persisting one or more metadata values and one or more data values from the in-memory array to the one or more blocks of the one or more pages;

wherein the one or more metadata values comprise one or more of a row count, an encoding type, a bit size, a value count, and a number of blocks;

wherein the one or more data values comprise one or more of a consolidated column data array, a column data value, and a dictionary value array.

2. The method of claim 1, wherein the marking is performed by the transaction thread, and the allocating and the materializing are performed by a materialization thread.

3. The method of claim 2, wherein the materializing consolidates one or more intermediate rollovers resulting from the transaction thread into a single rollover.

4. The method of claim 1, wherein the marking and the allocating are performed by the transaction thread, and the materializing is performed by a materialization thread.

5. The method of claim 1, wherein the materializing is performed in accordance with a predetermined time interval.

6. The method of claim 1, wherein the transaction thread comprises a data manipulation language thread.

7. A computer program product comprising a non-transitory machine readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:

maintaining an in-memory array in main system memory of at least one computing system, the in-memory array comprising a column of a database table forming part of an in-memory database system;

writing, to a database object associated with the column, one or more changes to the column resulting from one or more operations executed on the column by a transaction thread comprising a plurality of operations;

marking the database object for materialization in response to completion of all of the plurality of operations of the transaction thread;

allocating, by the transaction thread, one or more pages; and

materializing, based on the marking, the column from the in-memory array to the one or more pages stored on a persistent storage, the materializing comprising applying the one or more changes written in the database object, the one or more pages comprising one or more blocks for storing the column,

wherein the database object forms part of an object tree corresponding to fragments in a delta store, the object tree including fragment objects in a fragment level, columns in a column level, and dictionaries in a dictionary level;

wherein the persisting is iteratively performed at a fragment level of the in-memory array, at a column level of the in-memory array, and at a dictionary level of the in-memory array;

wherein the materializing comprises persisting one or more metadata values and one or more data values from the in-memory array to the one or more blocks of the one or more pages;

wherein the one or more metadata values comprise one or more of a row count, an encoding type, a bit size, a value count, and a number of blocks;

wherein the one or more data values comprise one or more of a consolidated column data array, a column data value, and a dictionary value array.

8. The computer program product of claim 7, wherein the marking is performed by the transaction thread, and the allocating and the materializing are performed by a materialization thread.

9. The computer program product of claim 8, wherein the materializing consolidates one or more intermediate rollovers resulting from the transaction thread into a single rollover.

10. The computer program product of claim 7, wherein the marking and the allocating are performed by the transaction thread, and the materializing is performed by a materialization thread.

11. The computer program product of claim 7, wherein the materializing is performed in accordance with a predetermined time interval.

12. A system comprising:

computer hardware comprising at least one programmable processor configured to execute instructions causing the at least one programmable processor to perform operations comprising;

maintaining an in-memory array in main system memory of at least one computing system, the in-memory array comprising a column of a database table forming part of an in-memory database system;

writing, to a database object associated with the column, one or more changes to the column resulting from one or more operations executed on the column by a transaction thread comprising a plurality of operations;

marking the database object for materialization in response to completion of all of the plurality of operations of the transaction thread;

allocating, by the transaction thread, one or more pages; and

materializing, based on the marking, the column from the in-memory array to the one or more pages stored on a persistent storage, the materializing comprising applying the one or more changes written in the database object, the one or more pages comprising one or more blocks for storing the column,

wherein the database object forms part of an object tree corresponding to fragments in a delta store, the object tree including fragment objects in a fragment level, columns in a column level, and dictionaries in a dictionary level;

wherein the persisting is iteratively performed at a fragment level of the in-memory array, at a column level of the in-memory array, and at a dictionary level of the in-memory array;

wherein the materializing comprises persisting one or more metadata values and one or more data values from the in-memory array to the one or more blocks of the one or more pages;

wherein the one or more metadata values comprise one or more of a row count, an encoding type, a bit size, a value count, and a number of blocks;

wherein the one or more data values comprise one or more of a consolidated column data array, a column data value, and a dictionary value array.