Optimization of metadata via lossy compression

• US 10,372,698 B2
• Filed: 06/12/2015
• Issued: 08/06/2019
• Est. Priority Date: 12/18/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Finlay, Ian R., Garcia-Arellano, Christian M., Storm, Adam J., Zuzarte, Calisto P.
• Primary Examiner(s): Featherstone, Mark D
• Assistant Examiner(s): Forouharnejad, Faezeh
• Application Number:
  US14/737,715
• Publication Number:
  US 20160179896A1
• Time in Patent Office: 1,516 Days
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/24561 Intermediate data storage techniques for performance improvement
• CPC Class Codes: G06F 16/24573 using data annotations, e.g. user-defined metadata
• CPC Class Codes: H03M 7/3059 Digital compression and data reduction techniques where the original information is represented by a subset or similar information, e.g. lossy compression

Abstract

The method includes identifying at least one of a minimum value, a maximum value, and a Bloom filter value for a row of data in a metadata table, wherein the metadata table contains metadata corresponding to a row of data in a main table. The method includes adjusting at least one of an identified first minimum value to a second minimum value, an identified first maximum value to a second maximum value, and an identified first Bloom filter value to a second Bloom filter value.

Claims

1. A method for adjusting metadata in tables, the method comprising:

identifying, by one or more computer processors, at least one of a first minimum value, a first maximum value, and a first Bloom filter value for a row of data in a metadata table, wherein the metadata table contains metadata corresponding to a row of data in a main table;

identifying, by one or more computer processors, at least one of a second minimum value that is a lower value than the first minimum value, a second maximum value that is higher than the first maximum value, and a second Bloom filter value for a row of data in the metadata table, wherein the second values are frequent values in the metadata table; and

adjusting, by one or more computer processors, at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value in the metadata table, and the identified first Bloom filter value to the second frequent Bloom filter value in the metadata table, wherein adjusting the metadata table compresses the metadata table.

2. The method of claim 1, wherein adjusting at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value comprises:

identifying, by one or more computer processors, at least one of a global minimum value for all rows of the metadata table and a global maximum value for all rows of the metadata table;

determining, by one or more computer processors, whether at least one of the identified first minimum value and the identified first maximum value is within a threshold number of the corresponding instance of the identified global minimum value and the identified global maximum value; and

in response to determining that at least one of the identified first minimum value and the identified first maximum value is within a threshold number of the corresponding instance of the identified global minimum value and the identified global maximum value, adjusting, by one or more computer processors, at least one of the identified first minimum value and the identified first maximum value to the identified global minimum value and the identified global maximum value.

3. The method of claim 1, wherein adjusting at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value comprises:

identifying, by one or more computer processors, at least one of a lower minimum value than the first identified minimum value that meets a threshold number of occurrences in the metadata table and a higher maximum value than the first identified maximum value that meets a threshold number of occurrences in the metadata table;

determining, by one or more computer processors, whether at least one of the identified first minimum value and the identified first maximum value is within a threshold number of the corresponding instance of the identified lower minimum value and the identified higher maximum value; and

in response to determining that at least one of the identified first minimum value and the identified first maximum value is within a threshold number of the corresponding instance of the identified lower minimum value and the identified higher maximum value, adjusting, by one or more computer processors, at least one of the identified first minimum value and the identified first maximum value to the corresponding instance of the identified lower minimum value and the identified higher maximum value.

4. The method of claim 1, wherein adjusting at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value comprises:

identifying, by one or more computer processors, at least one of an intermittent value within a threshold number of the identified first minimum value and an intermittent value within a threshold number of the identified first maximum value, wherein the intermittent value is between the identified minimum value for the row of data in the metadata table and the identified maximum value for the row of data in the metadata table; and

adjusting, by one or more computer processors, at least one of the first minimum value to the identified intermittent value, and the first maximum value to the identified intermittent value.

5. The method of claim 1, wherein adjusting at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value in the metadata table, and the identified first Bloom filter value to the second frequent Bloom filter value comprises:

generating, by one or more computer processors, an in-memory structure, wherein the in-memory structure corresponds to a column of data in the main table and corresponding metadata from the metadata table, and wherein the in-memory structure monitors at least one of a number of times a predicate is used, a type of predicate used, and a frequency of use of the metadata table for a predicate in the corresponding main table; and

in response to generating an in-memory structure, adjusting, by one or more computer processors, at least one of the identified first minimum value, the identified first maximum value, and the identified Bloom filter value based upon at least one of the number of times a predicate is used, the type of predicate used, and the frequency of use of the metadata table for a predicated in the corresponding main table.

6. The method of claim 1, wherein adjusting at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value in the metadata table, and the identified first Bloom filter value to the second frequent Bloom filter value comprises:

determining, by one or more computer processors, whether a number of bits, which are set to zero in the identified first Bloom filter value, is within a threshold number of bits; and

in response to determining that a number of bits, which are set to zero in the identified first Bloom filter value, is within a threshold number of bits, adjusting, by one or more computer processors, all zero bits for the identified first Bloom filter value to one.

7. The method of claim 1, wherein adjusting at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value in the metadata table, and the identified first Bloom filter value to the second frequent Bloom filter value comprises:

identifying, by one or more computer processors, a frequent Bloom filter value in the metadata table, wherein a frequent Bloom filter value is above a threshold number of occurrences;

determining, by one or more computer processors, whether a number of bits which are set to zero in the identified first Bloom filter value is within a threshold number of bits; and

in response to determining that a number of bits, which are set to zero in the identified first Bloom filter value, is within a threshold number of bits, adjusting, by one or more computer processors, zero bits for the identified first Bloom filter value to one bits to match the identified frequent Bloom filter value.

8. The method of claim 1, wherein adjusting at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value in the metadata table, and the identified first Bloom filter value to the second frequent Bloom filter value comprises:

identifying, by one or more computer processors, a frequent Bloom filter value in the metadata table, wherein a frequent Bloom filter value is above a threshold number of occurrences and subsumed by the bits set to one in the second bloom filter value;

determining, by one or more computer processors, whether a number of bits, which are set to one in the identified first Bloom filter value, is within a threshold number of bits; and

in response to determining that a number of bits, which are set to one in the identified first Bloom filter value, is within a threshold number of bits, adjusting, by one or more computer processors, zero bits for the identified first Bloom filter value to one bits to match the identified frequent Bloom filter value.

9. A method for compressing metadata in tables, the method comprising:

identifying, by one or more computer processors, at least one of a minimum value, a maximum value, and a Bloom filter value for a row of data in a metadata table, wherein the metadata table contains metadata corresponding to a row of data in a main table;

generating, by one or more computer processors, an in-memory structure, wherein the in-memory structure corresponds to a column of data in the main table and corresponding metadata from the metadata table, and wherein the in-memory structure monitors at least one of a number of times a predicate is used, a type of predicate used, and a frequency of use of the metadata table for a predicate in the corresponding main table;

identifying, by one or more computer processors, at least one of a second minimum value that is a lower value than the first minimum value, a second maximum value that is higher than the first maximum value, and a second Bloom filter value for a row of data in the metadata table, wherein the second values are frequent values in the metadata table; and

adjusting, by one or more computer processors, at least one of the identified first minimum value to the second frequent and lower minimum value in the metadata table, the identified first maximum value to the second frequent and higher maximum value in the metadata table, and the identified first Bloom filter value to the second frequent Bloom filter value in the metadata table, wherein adjusting the metadata table compresses the metadata table.