Unified architecture for hybrid database storage using fragments
First Claim
Patent Images
1. A computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
- storing, in a plurality of fragments retained in on-disk storage, a plurality of data records that comprise a data set, each fragment of the plurality of fragments comprising a local row index, a mapping of the local row index to RowID values in a global RowID column for the data set, and a number of data records that is equal to or less than a defined maximum fragment size, the RowID values in the RowID column being unique, logical references to the data records of the data set, the local row index comprising physical locations of the data records in the fragment, the defined maximum fragment size being smaller than an available amount of main system memory to facilitate the loading of each fragment of the plurality of fragments into the main system memory;
compressing each fragment of the plurality of fragments using a compression that is optimized for each fragment;
reading at least one of the plurality of fragments into main system memory from the on-disk storage; and
performing an operation on the at least one of the plurality of fragments while the at least one of the plurality of fragments is in the main system memory;
executing a merge of the data set with a change to the data set, the merge comprising;
creating a temporary, in-memory data structure containing data from a fragment of the plurality of fragments and a committed row from the change to the data set, andsorting the temporary, in-memory data structure to maximize compression of the plurality of data records; and
storing data from the sorted, temporary, in-memory data structure to a revised fragment.
1 Assignment
0 Petitions
Accused Products
Abstract
Data records of a data set can be stored in multiple main part fragments retained in on-disk storage. Each fragment can include a number of data records that is equal to or less than a defined maximum fragment size. Using a compression that is optimized for each fragment, each fragment can be compressed. After reading at least one of the fragments into main system memory from the on-disk storage, an operation can be performed on the fragment or fragments while the in the main system memory.
-
Citations
13 Claims
-
1. A computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
-
storing, in a plurality of fragments retained in on-disk storage, a plurality of data records that comprise a data set, each fragment of the plurality of fragments comprising a local row index, a mapping of the local row index to RowID values in a global RowID column for the data set, and a number of data records that is equal to or less than a defined maximum fragment size, the RowID values in the RowID column being unique, logical references to the data records of the data set, the local row index comprising physical locations of the data records in the fragment, the defined maximum fragment size being smaller than an available amount of main system memory to facilitate the loading of each fragment of the plurality of fragments into the main system memory; compressing each fragment of the plurality of fragments using a compression that is optimized for each fragment; reading at least one of the plurality of fragments into main system memory from the on-disk storage; and performing an operation on the at least one of the plurality of fragments while the at least one of the plurality of fragments is in the main system memory; executing a merge of the data set with a change to the data set, the merge comprising; creating a temporary, in-memory data structure containing data from a fragment of the plurality of fragments and a committed row from the change to the data set, and sorting the temporary, in-memory data structure to maximize compression of the plurality of data records; and storing data from the sorted, temporary, in-memory data structure to a revised fragment. - View Dependent Claims (2, 3, 4)
-
-
5. A system comprising:
-
at least one processor; and a machine-readable medium storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising; storing, in a plurality of fragments retained in on-disk storage, a plurality of data records that comprise a data set, each fragment of the plurality of fragments comprising a local row index, a mapping of the local row index to RowID values in a global RowID column for the data set, and a number of data records that is equal to or less than a defined maximum fragment size, the RowID values in the RowID column being unique, logical references to the data records of the data set, the local row index comprising physical locations of the data records in the fragment, the defined maximum fragment size being smaller than an available amount of main system memory to facilitate the loading of each fragment of the plurality of fragments into the main system memory; compressing each fragment of the plurality of fragments using a compression that is optimized for each fragment; reading at least one of the plurality of fragments into main system memory from the on-disk storage; and performing an operation on the at least one of the plurality of fragments while the at least one of the plurality of fragments is in the main system memory; executing a merge of the data set with a change to the data set, the merge comprising; creating a temporary, in-memory data structure containing data from a fragment of the plurality of fragments and a committed row from the change to the data set, and sorting the temporary, in-memory data structure to maximize compression of the plurality of data records; and storing data from the sorted, temporary, in-memory data structure to a revised fragment. - View Dependent Claims (6, 7, 8)
-
-
9. A computer-implemented method comprising:
-
storing, in a plurality of fragments retained in on-disk storage, a plurality of data records that comprise a data set, each fragment of the plurality of fragments comprising a local row index, a mapping of the local row index to RowID values in a global RowID column for the data set, and a number of data records that is equal to or less than a defined maximum fragment size, the RowID values in the RowID column being unique, logical references to the data records of the data set, the local row index comprising physical locations of the data records in the fragment, the defined maximum fragment size being smaller than an available amount of main system memory to facilitate the loading of each fragment of the plurality of fragments into the main system memory; compressing each fragment of the plurality of fragments using a compression that is optimized for each fragment; reading at least one of the plurality of fragments into main system memory from the on-disk storage; and performing an operation on the at least one of the plurality of fragments while the at least one of the plurality of fragments is in the main system memory; executing a merge of the data set with a change to the data set, the merge comprising; creating a temporary, in-memory data structure containing data from a fragment of the plurality of fragments and a committed row from the change to the data set, and sorting the temporary, in-memory data structure to maximize compression of the plurality of data records; and storing data from the sorted, temporary, in-memory data structure to a revised fragment. - View Dependent Claims (10, 11, 12, 13)
-
Specification