Database storage architecture

• US 8,700,674 B2
• Filed: 07/14/2009
• Issued: 04/15/2014
• Est. Priority Date: 07/14/2009
• Status: Expired due to Fees

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Bear, Chuck, Venkatesh, Rajat, Vandiver, Benjamin, Bodagala, Sreenath, Lawande, Shilpa
• Primary Examiner(s): GOFMAN, ALEX N
• Application Number:
  US12/502,890
• Publication Number:
  US 20110016157A1
• Time in Patent Office: 1,736 Days
• Field of Search: 707/804
• US Class Current: 707/802
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays

Abstract

Methods, systems and program products for database storage. In one implementation, data of a projection of a database is stored at least partly in grouped ROS format and partly in column format based on patterns of updating the projection data. The projection data is updated so that the updated projection is stored partly in grouped ROS format and partly in column format.

Claims

1. A computer-implemented method comprising:

determining a size of projection data to be stored in a read-optimized store (ROS) container, wherein the projection data includes rows and columns of data;

determining whether the size of the projection data is greater than a predetermined threshold;

in response to a determination that the size of the projection data is greater than the predetermined threshold, storing the projection data in the ROS container in a column format, wherein, in the column format, each column of the projection data is stored in a separate file; and

in response to a determination that the size of the projection data is less than the predetermined threshold, storing the projection data in the ROS container in a grouped ROS format, wherein, in the grouped ROS format, each of a plurality of blocks formed of respective groups of rows and columns of the projection data is stored in a separate file

wherein storing the projection data in the grouped ROS format includes;

selecting a plurality of columns of the projection data, and

storing rows of the selected columns as blocks of data in the ROS container until all of the rows of the selected columns are stored in the ROS container.

2. The method of claim 1, wherein the projection data is stored persistently.

3. The method of claim 1, wherein the projection data is to be stored in the ROS container when the projection data exceeds an age or an aggregate size.

4. The method of claim 1, further comprising updating the projection data in a write-optimized store before storing the projection data in the ROS container.

5. The method of claim 1, further including selecting an order of the columns in the selected subset of columns to be stored together.

6. The method of claim 5 in which each row'"'"'s first columns are stored together in a single file with at least one other row'"'"'s first columns.

7. The method of claim 1, wherein storing the projection data in the column format comprises:

selecting one column of the projection data,

storing a subset of rows of data of the selected column as a block of data in the ROS container, and

storing additional subsets of rows of data of the selected column as additional blocks of data in the ROS container until all of the rows of data of the selected column are stored in the ROS container.

8. The method of claim 7, wherein the blocks of data of the selected column are stored in separate files.

9. The method of claim 1, further comprising storing rows of the projection data for columns in the grouped ROS format based on a frequency of use of the columns of the rows.

10. The method of claim 1, further comprising updating the projection data in response to a request to write data to the projection.

11. The method of claim 10, wherein the updating is performed in response to processing a database operator to write data to storage.

12. A computer-implemented method comprising:

selecting at least two different storage formats to store data of a database projection based on patterns of updating data of the projection, wherein the at least two different storage formats include a column format and a grouped read-optimized store (ROS) format, wherein the column format is to store each column of the data in a separate file, and the grouped ROS format is to store each of a plurality of blocks formed of respective groups of rows and columns of the data is stored in a separate file;

updating the projection data based on the patterns using the selected storage formats;

determining whether a size of the updated projection data is greater than a predetermined threshold;

in response to a determination that the size of the updated projection data is greater than the predetermined threshold, storing the updated projection data in the column format; and

in response to a determination that the size of the updated projection data is less than the predetermined threshold, storing the updated projection data in the grouped ROS format,

wherein storing the updated projection data in the grouped ROS format includes;

selecting columns of the updated projection data to be stored together, and

storing rows of the selected columns as blocks of data in the store until all of the rows of the selected columns are stored in the store.

13. The method of claim 12, wherein the storage formats further comprise a write-optimized storage format.

14. The method of claim 12, further comprising storing the updated projection data persistently.

15. The method of claim 12, wherein the projection data is stored at a time based on at least one of an age or an aggregate size of the of the projection data.

16. The method of claim 12, further comprising storing the updated projection data in a write-optimized store.

17. The method of claim 12, further comprising:

determining the size of the updated projection data by estimating a storage size of rows of the updated projection data based on a size of previously stored projection rows; and

storing the rows of the updated projection data in the grouped read-optimized store (ROS) format or in the column format based on the estimated storage size.

18. The method of claim 12, wherein each of the blocks of data is stored in a single file.

19. The method of claim 12, wherein storing the updated projection data in the column format includes:

selecting one column of the updated projection data,

storing a plurality of rows of the selected column in a file, and

storing additional rows of the selected column in the file until all of the rows of the selected column are stored in the file.

20. The method of claim 12, further comprising storing rows of the updated projection data in the grouped read-optimized store (ROS) format based on a frequency of use of columns of the rows.

21. The method of claim 12 wherein updating is performed in response to processing a database operator to write data to storage.

22. A non-transitory computer readable medium on which is stored a set of instructions that, when executed, causes a device to:

determine a size of projection data to be stored in a read-optimized store (ROS) container, wherein the projection data includes rows and columns of data;

determine whether the size of the projection data is greater than a predetermined threshold;

in response to a determination that the size of the projection data is greater than the predetermined threshold, store the projection data in the ROS container in a column format, wherein, in the column format, each column of the projection data is stored in a separate file; and

in response to a determination that the size of the projection data is less than the predetermined threshold, store the projection data in the ROS container in the grouped ROS format, wherein, in the grouped ROS format, each block of rows and columns of the projection data is stored in a separate file,

wherein to store the projection data in the grouped ROS format, the set of instructions is to cause the device to;

select a plurality of columns of the projection data, and

store rows of the selected columns as blocks of data in the ROS container until all of the rows of the selected columns are stored in the ROS container.

23. A computer-implemented method comprising:

selecting a plurality of candidate containers to store projection data;

identifying a plurality of containers to merge in the candidate containers;

selecting storage formats to store the projection data, including;

determining whether a size of the projection data is greater than a predetermined threshold;

in response to a determination that the size of the projection data is greater than the predetermined threshold, selecting a column format to store the projection data, wherein, in the column format, each column of the projection data is stored in a separate file; and

in response to a determination that the size of the new rows is less than the predetermined threshold, selecting a grouped read-optimized store (ROS) format to store the projection data, wherein, in the grouped ROS format, each of a plurality of blocks formed of respective groups of rows and columns of the projection data is stored in a separate file; and

merging the identified containers into a new container according to the selected storage formats,

wherein selecting the storage formats is based on an assessment of existing storage allocated to the projection data such that some but not all of projection data is stored in the grouped ROS format,

wherein selecting the grouped ROS format to store the projection data includes;

selecting a plurality of columns of the projection data to be stored together, and

storing rows of the selected columns as blocks of data in the store until all of the rows of the selected columns are stored in the store.

24. The method of claim 23 wherein the selected candidate containers are in a common size range.

25. The method of claim 23 wherein the identified containers are each associated with a start epoch that is smaller than the remaining candidate containers.