REORDERING OF DATABASE RECORDS FOR IMPROVED COMPRESSION

• US 20150347087A1
• Filed: 05/27/2014
• Published: 12/03/2015
• Est. Priority Date: 05/27/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dickie, Garth A., Keller, Jeffrey M.
• Granted Patent: US 9,798,727 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 7/36 Combined merging and sorting

Abstract

According to embodiments of the present invention, apparatus, systems, methods and computer program products for sorting and compressing an unordered set of data records from a structured database are provided. Fields of the unordered set of data records are prioritized based on an impact of those fields to a compression scheme for column-oriented compression. The unordered set of data records are sorted based on the prioritized field(s) with a greatest impact on the performance metric. Data of the sorted data records are compressed according to a compression scheme. In some embodiments, prioritizing the fields may be based on an anticipated level of usage of data within those fields and/or a cost function associated with a performance metric as well as optimization of compression. A performance metric may include a faster computational time, reduced I/O computation, faster scan time, etc.

Claims

1-7. -7. (canceled)

8. A system comprising:

a memory; and

a processor configured to;

prioritize fields of an unordered set of data records from a structured database based on an impact of those fields to a performance metric for accessing data stored in a column-oriented compressed database;

sort the unordered set of data records based on the one or more prioritized field with a greatest impact on the performance metric; and

compress data of the sorted data records according to a compression scheme.

9. The system of claim 8, wherein the processor is configured to prioritize the fields based on an anticipated level of usage of data within those fields.

10. The system of claim 8, wherein the processor is configured to prioritize the fields based on a performance metric as assessed by a cost function, the cost function including minimizing a total amount of time reading and computing on a column as well as the compressed size of data in the column.

11. The system of claim 8, wherein the processor is configured to sort the unordered set of data records based on a prioritized tuple of sort keys.

12. The system of claim 8, wherein the unordered set of data records is from a columnar database or a hybrid columnar database.

13. The system of claim 8, wherein the performance metric is determined based upon historical data.

14. The system of claim 8, wherein the processor is configured to:

update, based upon monitored performance metrics, relative priorities of the various fields to each other;

decompress a set of ordered compressed records to obtain an unordered set of data records;

sort the unordered set of data records based on the updated field priorities to obtain a second set of sorted data records; and

compress data of the second set of sorted data records according to a compression scheme.

15. A computer program product for providing information to a user, comprising a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code, when executed by a processor, causes the processor to:

prioritize fields of an unordered set of data records from a structured database based on an impact of those fields to a performance metric for accessing data stored in a column-oriented compressed database;

sort the unordered set of data records based on the one or more prioritized field with a greatest impact on the performance metric; and

compress data of the sorted data records according to a compression scheme.

16. The computer program product of claim 15, the computer readable program code further configured to cause the processor to prioritize the fields based on an anticipated level of usage of data within those fields.

17. The computer program product of claim 15, the computer readable program code further configured to cause the processor to prioritize the fields based on a performance metric as assessed by a cost function, the cost function including minimizing a total amount of time reading and computing on a column as well as the compressed size of data in the column.

18. The computer program product of claim 15, the computer readable program code further configured to cause the processor to sort the unordered set of data records based on a prioritized tuple of sort keys.

19. The computer program product of claim 15, wherein the performance metric is determined based upon historical data.

20. The computer program product of claim 15, the computer readable program code further configured to cause the processor to:

update, based upon monitored performance metrics, relative priorities of the various fields to each other;

decompress a set of ordered compressed records to obtain an unordered set of data records;

sort the unordered set of data records based on the updated field priorities to obtain a second set of sorted data records; and

compress data of the second set of sorted data records according to a compression scheme.