Materializing data from an in-memory array to an on-disk page structure

• US 9,830,109 B2
• Filed: 12/05/2016
• Issued: 11/28/2017
• Est. Priority Date: 11/25/2014
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP SE
• Inventors: Wein, David, Andrei, Mihnea, Thomsen, Dirk, Schreter, Ivan
• Primary Examiner(s): CARDWELL, ERIC
• Application Number:
  US15/369,618
• Publication Number:
  US 20170083243A1
• Time in Patent Office: 358 Days
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/24552 Database cache management
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0673 Single storage device
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays

Abstract

The subject matter disclosed herein provides methods for materializing data from an in-memory array to one or more pages. An in-memory array holding a column of data can be maintained. One or more pages can be maintained. Each of the one or more pages can have one or more rows for storing the column of data. At least one of the one or more pages can be marked for materialization. The column of data can be materialized by copying the data from the in-memory array to the one or more rows of the one or more pages. The materializing can be based on the marking. Related apparatus, systems, techniques, and articles are also described.

Claims

1. A method for implementation by one or more data processors forming part of at least one computing system, the method comprising:

inserting, by a transaction thread, a data value into a column of data held in an in-memory array, the in-memory array being located in main system memory of one or more computing systems and having a memory address associated with it, the transaction thread writing the data value directly into the memory array;

reserving, by the transaction thread, one or more rows of a page-based layout into which the in-memory array is mirrored, the reserving comprising indicating that the reserved one or more rows are in use, the page-based layout comprising one or more pages maintained in a persistent storage;

materializing the column of data, the materializing comprising copying data in the column of data from the in-memory array to at least one page of the one or more pages, the at least one page being marked for materialization.

2. The method of claim 1, wherein the materializing is performed by the transaction thread, the transaction thread copying the column of data from the in-memory array to the one or more pages as the data value is added to the in-memory array.

3. The method of claim 1, wherein the materializing is performed by a flusher thread, which is separate from the transaction thread, when a savepoint is encountered.

4. The method of claim 3, wherein the predetermined condition specifies a threshold number of reserved rows or a threshold percentage of reserved rows.

5. The method of claim 1, wherein the materializing is performed by a flusher thread using a lazy materialization protocol that copies the column of data to the at least one page when a predetermined condition is satisfied.

6. The method of claim 1, wherein the materializing is performed by a flusher thread using a batch materialization protocol that copies the column of data to the at least one page when all of the one or more rows on the at least one page are reserved.

7. The method of claim 1 further comprising using an unload bitmap to indicate when the one or more rows on the at least one page have been reserved by one or more transaction threads.

8. The method of claim 7, wherein all of the one or more rows in the unload bitmap are open when the at least one page is allocated, and

wherein at least one of the rows in the unload bitmap is closed when the at least one row is reserved by the transaction thread.

9. The method of claim 8, further comprising:

tagging the at least one page as ready for materialization when all of the one or more rows in the unload bitmap are closed.

10. The method of claim 1, wherein each of the one or more pages is associated with a pending materialization bit, and

wherein a page of the least one page is marked for materialization at least by changing a value of the pending materialization bit.

11. The method of claim 10, wherein the changing the value of the pending materialization bit is performed by transaction thread that reserves at least one row of the one or more rows on the at least one page.

12. The method of claim 1, wherein no translation is required when copying the column of data from the in-memory array to the at least one page.

13. The method of claim 1, wherein the column of data spans at least one or more bytes of memory, and

wherein the materializing starts the copying of the column of data at a bit within the one or more bytes of memory.

14. The method of claim 1 further comprising persisting the data on the one or more pages to a disk and discarding the one or more pages.

15. A non-transitory computer-readable medium containing instructions to configure a processor to perform operations comprising:

inserting, by a transaction thread, a data value into a column of data held in an in-memory array, the in-memory array being located in main system memory of one or more computing systems and having a memory address associated with it, the transaction thread writing the data value directly into the memory array;

reserving, by the transaction thread, one or more rows of a page-based layout into which the in-memory array is mirrored, the reserving comprising indicating that the reserved one or more rows are in use, the page-based layout comprising one or more pages maintained in a persistent storage;

materializing the column of data, the materializing comprising copying data in the column of data from the in-memory array to at least one page of the one or more pages, the at least one page being marked for materialization.

16. The non-transitory computer-readable medium of claim 15, wherein the materializing comprises:

the transaction thread copying the column of data from the in-memory array to the one or more pages as the data value is added to the in-memory array;

or

a flusher thread, which is separate from the transaction thread, copying data in the column from the in-memory array to the at least one page, the copying occurring when a savepoint is encountered, when a predetermined condition is satisfied, or using a batch materialization protocol that copies the column of data to the at least one page when all of the one or more rows on the at least one page are reserved.

17. The non-transitory computer-readable medium of claim 15, wherein the operations further comprise using an unload bitmap to indicate when the one or more rows on the at least one page have been reserved by one or more transaction threads,

wherein all of the one or more rows in the unload bitmap are open when the at least one page is allocated, and

wherein at least one of the rows in the unload bitmap is closed when the at least one row is reserved by the transaction thread.

18. A system comprising:

at least one processor; and

at least one memory, wherein the at least one processor and the at least one memory are configured to perform operations comprising;

inserting, by a transaction thread, a data value into a column of data held in an in-memory array, the in-memory array being located in main system memory of one or more computing systems and having a memory address associated with it, the transaction thread writing the data value directly into the memory array;

reserving, by the transaction thread, one or more rows of a page-based layout into which the in-memory array is mirrored, the reserving comprising indicating that the reserved one or more rows are in use, the page-based layout comprising one or more pages maintained in a persistent storage;

materializing the column of data, the materializing comprising copying data in the column of data from the in-memory array to at least one page of the one or more pages, the at least one page being marked for materialization.

19. The system of claim 18, wherein the materializing comprises:

the transaction thread copying the column of data from the in-memory array to the one or more pages as the data value is added to the in-memory array;

or

a flusher thread, which is separate from the transaction thread, copying data in the column from the in-memory array to the at least one page, the copying occurring when a savepoint is encountered, when a predetermined condition is satisfied, or using a batch materialization protocol that copies the column of data to the at least one page when all of the one or more rows on the at least one page are reserved.

20. The system of claim 18, wherein the operations further comprise using an unload bitmap to indicate when the one or more rows on the at least one page have been reserved by one or more transaction threads,

wherein all of the one or more rows in the unload bitmap are open when the at least one page is allocated, and

wherein at least one of the rows in the unload bitmap is closed when the at least one row is reserved by the transaction thread.