Optimization of metadata via lossy compression

• US 10,366,068 B2
• Filed: 12/18/2014
• Issued: 07/30/2019
• Est. Priority Date: 12/18/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Finlay, Ian R., Garcia-Arellano, Christian M., Storm, Adam J., Zuzarte, Calisto P.
• Primary Examiner(s): Featherstone, Mark D
• Assistant Examiner(s): Forouharnejad, Faezeh
• Application Number:
  US14/575,425
• Publication Number:
  US 20160179858A1
• Time in Patent Office: 1,685 Days
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/24561 Intermediate data storage techniques for performance improvement
• CPC Class Codes: G06F 16/24573 using data annotations, e.g. user-defined metadata
• CPC Class Codes: H03M 7/3059 Digital compression and data reduction techniques where the original information is represented by a subset or similar information, e.g. lossy compression

Abstract

The method includes identifying at least one of a minimum value, a maximum value, and a Bloom filter value for a row of data in a metadata table, wherein the metadata table contains metadata corresponding to a row of data in a main table. The method includes adjusting at least one of an identified first minimum value to a second minimum value, an identified first maximum value to a second maximum value, and an identified first Bloom filter value to a second Bloom filter value.

Claims

1. A computer program product for adjusting metadata in tables, the computer program product comprising:

one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the program instructions comprising;

program instructions to identify at least one of a first minimum value, a first maximum value, and a first Bloom filter value for a row of data in a metadata table, wherein the metadata table contains metadata corresponding to a row of data in a main table;

program instructions to identify at least one of a second minimum value that is a lower value than the first minimum value, a second maximum value that is higher than the first maximum value, and a second Bloom filter value for a row of data in the metadata table, wherein the second values are frequent values in the metadata table; and

program instructions to adjust at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value in the metadata table, and the identified first Bloom filter value to the second frequent Bloom filter value in the metadata table, wherein adjusting the metadata table compresses the metadata table.

2. The computer program product of claim 1, wherein program instructions to adjust at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value comprises program instructions to:

identify at least one of a global minimum value for all rows of the metadata table and a global maximum value for all rows of the metadata table;

determine whether at least one of the identified first minimum value and the identified first maximum value is within a threshold number of the corresponding instance of the identified global minimum value and the identified global maximum value; and

in response to determining that at least one of the identified first minimum value and the identified first maximum value is within a threshold number of the corresponding instance of the identified global minimum value and the identified global maximum value, adjust at least one of the identified first minimum value and the identified first maximum value to the identified global minimum value and the identified global maximum value.

3. The computer program product of claim 1, wherein program instructions to adjust at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value comprises program instructions to:

identify at least one of a lower minimum value than the first identified minimum value that meets a threshold number of occurrences in the metadata table and a higher maximum value than the first identified maximum value that meets a threshold number of occurrences in the metadata table;

determine whether at least one of the identified first minimum value and the identified first maximum value is within a threshold number of the corresponding instance of the identified lower minimum value and the identified higher maximum value; and

in response to determining that at least one of the identified first minimum value and the identified first maximum value is within a threshold number of the corresponding instance of the identified lower minimum value and the identified higher maximum value, adjust at least one of the identified first minimum value and the identified first maximum value to the corresponding instance of the identified lower minimum value and the identified higher maximum value.

4. The computer program product of claim 1, wherein program instructions to adjust at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value comprises program instructions to:

identify at least one of an intermittent value within a threshold number of the identified first minimum value and an intermittent value within a threshold number of the identified first maximum value, wherein the intermittent value is between the identified minimum value for the row of data in the metadata table and the identified maximum value for the row of data in the metadata table; and

adjust at least one of the first minimum value to the identified intermittent value, and the first maximum value to the identified intermittent value.

5. The computer program product of claim 1, wherein program instructions to adjust at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value in the metadata table, and the identified first Bloom filter value to the second frequent Bloom filter value comprises program instructions to:

program instructions to generate an in-memory structure, wherein the in-memory structure corresponds to a column of data in the main table and corresponding metadata from the metadata table, and wherein the in-memory structure monitors at least one of a number of times a predicate is used, a type of predicate used, and a frequency of use of the metadata table for a predicate in the corresponding main table; and

in response to generating an in-memory structure, adjust at least one of the identified first minimum value, the identified first maximum value, and the identified Bloom filter value based upon at least one of the number of times a predicate is used, the type of predicate used, and the frequency of use of the metadata table for a predicated in the corresponding main table.

6. The computer program product of claim 1, wherein program instructions to adjust at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value in the metadata table, and the identified first Bloom filter value to the second frequent Bloom filter value comprises program instructions to:

determine whether a number of bits, which are set to zero in the identified first Bloom filter value, is within a threshold number of bits; and

in response to determining that a number of bits, which are set to zero in the identified first Bloom filter value, is within a threshold number of bits, adjust all zero bits for the identified first Bloom filter value to one.

7. The computer program product of claim 1, wherein program instructions adjust at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value in the metadata table, and the identified first Bloom filter value to the second frequent Bloom filter value comprises program instructions to:

identify a frequent Bloom filter value in the metadata table, wherein a frequent Bloom filter value is above a threshold number of occurrences;

determine whether a number of bits, which are set to zero in the identified first Bloom filter value, is within a threshold number of bits; and

in response to determining that a number of bits, which are set to zero in the identified first Bloom filter value, is within a threshold number of bits, adjust all zero bits for the identified first Bloom filter value to one bits to match the identified frequent Bloom filter value.

8. The computer program product of claim 1, wherein program instructions to adjust at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value in the metadata table, and the identified first Bloom filter value to the second frequent Bloom filter value comprises program instructions to:

identify a frequent Bloom filter value in the metadata table, wherein a frequent Bloom filter value is above a threshold number of occurrences and subsumed by the bits set to one in the second bloom filter value;

determine whether a number of bits, which are set to one in the identified first Bloom filter value, is within a threshold number of bits; and

in response to determining that a number of bits, which are set to one in the identified first Bloom filter value, is within a threshold number of bits, adjust one bits for the identified first Bloom filter value to zero bits to match the identified frequent Bloom filter value.

9. A computer system for adjusting metadata in tables, the computer system comprising:

one or more computer processors;

one or more computer readable storage media;

program instructions stored on the computer readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising;

program instructions to identify at least one of a first minimum value, a first maximum value, and a first Bloom filter value for a row of data in a metadata table, wherein the metadata table contains metadata corresponding to a row of data in a main table;

program instructions to identify at least one of a second minimum value that is a lower value than the first minimum value, a second maximum value that is higher than the first maximum value, and a second Bloom filter value for a row of data in the metadata table, wherein the second values are frequent values in the metadata table; and

program instructions to adjust at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value in the metadata table, and the identified first Bloom filter value to the second frequent Bloom filter value in the metadata table, wherein adjusting the metadata table compresses the metadata table.

10. The computer program product of claim 9, wherein program instructions to adjust at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value comprises program instructions to:

identify at least one of a global minimum value for all rows of the metadata table and a global maximum value for all rows of the metadata table;

determine whether at least one of the identified first minimum value and the identified first maximum value is within a threshold number of the corresponding instance of the identified global minimum value and the identified global maximum value; and

in response to determining that at least one of the identified first minimum value and the identified first maximum value is within a threshold number of the corresponding instance of the identified global minimum value and the identified global maximum value, adjust at least one of the identified first minimum value and the identified first maximum value to the identified global minimum value and the identified global maximum value.

11. The computer program product of claim 9, wherein program instructions to adjust at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value comprises program instructions to:

identify at least one of a lower minimum value than the first identified minimum value that meets a threshold number of occurrences in the metadata table and a higher maximum value than the first identified maximum value that meets a threshold number of occurrences in the metadata table;

determine whether at least one of the identified first minimum value and the identified first maximum value is within a threshold number of the corresponding instance of the identified lower minimum value and the identified higher maximum value; and

in response to determining that at least one of the identified first minimum value and the identified first maximum value is within a threshold number of the corresponding instance of the identified lower minimum value and the identified higher maximum value, adjust at least one of the identified first minimum value and the identified first maximum value to the corresponding instance of the identified lower minimum value and the identified higher maximum value.

12. The computer program product of claim 9, wherein program instructions to adjust at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value comprises program instructions to:

identify at least one of an intermittent value within a threshold number of the identified first minimum value and an intermittent value within a threshold number of the identified first maximum value, wherein the intermittent value is between the identified minimum value for the row of data in the metadata table and the identified maximum value for the row of data in the metadata table; and

adjust at least one of the first minimum value to the identified intermittent value, and the first maximum value to the identified intermittent value.

13. The computer program product of claim 9, wherein program instructions to adjust at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value in the metadata table, and the identified first Bloom filter value to the second frequent Bloom filter value comprises program instructions to:

program instructions to generate an in-memory structure, wherein the in-memory structure corresponds to a column of data in the main table and corresponding metadata from the metadata table, and wherein the in-memory structure monitors at least one of a number of times a predicate is used, a type of predicate used, and a frequency of use of the metadata table for a predicate in the corresponding main table; and

in response to generating an in-memory structure, adjust at least one of the identified first minimum value, the identified first maximum value, and the identified Bloom filter value based upon at least one of the number of times a predicate is used, the type of predicate used, and the frequency of use of the metadata table for a predicated in the corresponding main table.

14. The computer program product of claim 9, wherein program instructions to adjust at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value in the metadata table, and the identified first Bloom filter value to the second frequent Bloom filter value comprises program instructions to:

determine whether a number of bits, which are set to zero in the identified first Bloom filter value, is within a threshold number of bits; and

in response to determining that a number of bits, which are set to zero in the identified first Bloom filter value, is within a threshold number of bits, adjust all zero bits for the identified first Bloom filter value to one.

15. The computer program product of claim 9, wherein program instructions to adjust at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value in the metadata table, and the identified first Bloom filter value to the second frequent Bloom filter value comprises program instructions to:

identify a frequent Bloom filter value in the metadata table, wherein a frequent Bloom filter value is above a threshold number of occurrences;

determine whether a number of bits, which are set to zero in the identified first Bloom filter value, is within a threshold number of bits; and

in response to determining that a number of bits, which are set to zero in the identified first Bloom filter value, is within a threshold number of bits, adjust zero bits for the identified first Bloom filter value to one bits to match the identified frequent Bloom filter value.

16. The computer program product of claim 9, wherein program instructions to adjust at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value in the metadata table, and the identified first Bloom filter value to the second frequent Bloom filter value comprises program instructions to:

identify a frequent Bloom filter value in the metadata table, wherein a frequent Bloom filter value is above a threshold number of occurrences and subsumed by the bits set to one in the second bloom filter value;

determine whether a number of bits, which are set to one in the identified first Bloom filter value, is within a threshold number of bits; and

in response to determining that a number of bits, which are set to one in the identified first Bloom filter value, is within a threshold number of bits, adjust one bits for the identified first Bloom filter value to zero bits to match the identified frequent Bloom filter value.