Automatic Consistent Sampling For Data Analysis

• US 20140012819A1
• Filed: 09/06/2013
• Published: 01/09/2014
• Est. Priority Date: 07/19/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gorelik, Alexander
• Granted Patent: US 8,892,525 B2
• Time in Patent Office: Days
• US Class Current: 707/690
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity

Abstract

A method, computer program product, and system for analyzing data within one or more databases, comprising selecting one or more databases for analysis, each database comprising one or more database objects comprising one or more data values, applying a function to each data value in each database object within the one or more databases, where the function produces function values limited to a predetermined range, identifying for analysis the data values producing a certain function value within the predetermined range to form a sampled data set, and analyzing the sampled data set to determine relationships between the database objects within and across the one or more databases.

Claims

1. A computer-implemented method of analyzing data within one or more databases, comprising:

selecting one or more databases for analysis, each database comprising one or more database objects comprising one or more data values;

applying a function to each data value in each database object within the one or more databases, wherein the function produces function values limited to a predetermined range;

identifying for analysis the data values producing a certain function value within the predetermined range to form a sampled data set; and

analyzing the sampled data set to determine relationships between the database objects within and across the one or more databases.

3. The method of claim 1, wherein the data values in each database object are arranged in columns, and further comprising:

identifying for analysis the data values that do not produce the certain function value to form an unsampled data set, unless the data values are in a high cardinality column; and

analyzing the unsampled data set with the sampled data set to determine relationships between the database objects within and across the one or more databases.

4. The method of claim 3, wherein a column is a high cardinality column if a number of data values in the column that produce the certain function value exceeds a predetermined cardinality threshold.

5. The method of claim 3, wherein a column is a high cardinality column it a number of unique data values in the column divided by the number of data values in the column exceeds a predetermined selectivity threshold.

7. The method of claim 1, wherein the database objects are tables.

2. The method of claim I, wherein said analysis further comprises determining one or more primary key-foreign key relationships between the database objects within and across the one or more databases.

6. The method of claim wherein the function is a hash function.