Unified architecture for hybrid database storage using fragments

• US 9,286,336 B2
• Filed: 03/12/2013
• Issued: 03/15/2016
• Est. Priority Date: 03/12/2013
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP SE
• Inventors: Schreter, Ivan, Thomsen, Dirk, Florendo, Colin, French, Blaine
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): BUI, THUY T
• Application Number:
  US13/797,676
• Publication Number:
  US 20140279961A1
• Time in Patent Office: 1,099 Days
• Field of Search: 707/693
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0661 Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/0673 Single storage device

Abstract

Data records of a data set can be stored in multiple main part fragments retained in on-disk storage. Each fragment can include a number of data records that is equal to or less than a defined maximum fragment size. Using a compression that is optimized for each fragment, each fragment can be compressed. After reading at least one of the fragments into main system memory from the on-disk storage, an operation can be performed on the fragment or fragments while the in the main system memory.

Claims

1. A computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:

storing, in a plurality of main part fragments retained in on-disk storage, a plurality of data records that comprise a data set, each fragment comprising a number of data records that is equal to or less than a defined maximum fragment size;

compressing each fragment of the plurality of fragments using a compression that is optimized for each fragment;

reading at least one of the plurality of fragments into main system memory from the on-disk storage; and

performing an operation on the at least one of the plurality of fragments while the at least one of the plurality of fragments is in the main system memory;

executing a merge of an existing delta part with the data set, the merge comprising;

copying one or more uncommitted rows from the existing delta part to a new delta part;

creating a temporary, in-memory data structure containing data from an existing main fragment and committed rows from the existing delta;

re-sorting the temporary, in-memory data structure to allow achievement of maximal compression of the data records; and

storing data from the re-sorted, temporary, in-memory data structure to a new main fragment;

storing cleanup pointers at a start of the merge, the cleanup pointers indicating start and stop locations for the temporary data structure.

2. The computer program product as in claim 1, wherein the defined maximum fragment size is smaller than an available amount of main system memory such that each fragment can be loaded into the main system memory.

3. The computer program product as in claim 1, wherein each fragment of the plurality of fragments comprises a local row index and a mapping of the local row index to RowID values in a global RowID column for the data set, the RowID values in the RowID column being unique, logical references to the data records of the data set, the local row index comprising physical locations of the data records in each fragment of the plurality of fragments.

4. The computer program product as in claim 3, wherein the local row index comprises a compressed local RowID column included with its associated fragment.

5. The computer program product as in claim 1, wherein each of the plurality of main part fragments comprises at least one row set arranged in a temporal order.

6. A system comprising:

at least one processor; and

a machine-readable medium storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising;

storing, in a plurality of main part fragments retained in on-disk storage, a plurality of data records that comprise a data set, each fragment comprising a number of data records that is equal to or less than a defined maximum fragment size;

compressing each fragment of the plurality of fragments using a compression that is optimized for each fragment;

reading at least one of the plurality of fragments into main system memory from the on-disk storage; and

performing an operation on the at least one of the plurality of fragments while the at least one of the plurality of fragments is in the main system memory;

executing a merge of an existing delta part with the data set, the merge comprising;

copying one or more uncommitted rows from the existing delta part to a new delta part;

creating a temporary, in-memory data structure containing data from an existing main fragment and committed rows from the existing delta;

re-sorting the temporary, in-memory data structure to allow achievement of maximal compression of the data records; and

storing data from the re-sorted, temporary, in-memory data structure to a new main fragment

storing cleanup pointers at a start of the merge, the cleanup pointers indicating start and stop locations for the temporary data structure.

7. The system as in claim 6, wherein the defined maximum fragment size is smaller than an available amount of main system memory such that each fragment can be loaded into the main system memory.

8. The system as in claim 6, wherein each fragment of the plurality of fragments comprises a local row index and a mapping of the local row index to RowID values in a global RowID column for the data set, the RowID values in the RowID column being unique, logical references to the data records of the data set, the local row index comprising physical locations of the data records in each fragment of the plurality of fragments.

9. The system as in claim 6, wherein each of the plurality of main part fragments comprises at least one row set arranged in a temporal order.

10. A computer-implemented method comprising:

storing, in a plurality of main part fragments retained in on-disk storage, a plurality of data records that comprise a data set, each fragment comprising a number of data records that is equal to or less than a defined maximum fragment size;

compressing each fragment of the plurality of fragments using a compression that is optimized for each fragment;

reading at least one of the plurality of fragments into main system memory from the on-disk storage; and

performing an operation on the at least one of the plurality of fragments while the at least one of the plurality of fragments is in the main system memory;

executing a merge of an existing delta part with the data set, the merge comprising;

copying one or more uncommitted rows from the existing delta part to a new delta part;

creating a temporary, in-memory data structure containing data from an existing main fragment and committed rows from the existing delta;

re-sorting the temporary, in-memory data structure to allow achievement of maximal compression of the data records; and

storing data from the re-sorted, temporary, in-memory data structure to a new main fragment;

storing cleanup pointers at a start of the merge, the cleanup pointers indicating start and stop locations for the temporary data structure.

11. The computer-implemented method as in claim 10, wherein the defined maximum fragment size is smaller than an available amount of main system memory such that each fragment can be loaded into the main system memory.

12. The computer-implemented method as in claim 10, wherein each fragment of the plurality of fragments comprises a local row index and a mapping of the local row index to RowID values in a global RowID column for the data set, the RowID values in the RowID column being unique, logical references to the data records of the data set, the local row index comprising physical locations of the data records in each fragment of the plurality of fragments.

13. The computer-implemented method as in claim 10, wherein each of the plurality of main part fragments comprises at least one row set arranged in a temporal order.

14. The computer-implemented method as in claim 10, wherein at least one of the storing, the compressing, the reading, and the performing are performed by a system comprising at least one programmable processor.