Handling data sets

• US 8,666,998 B2
• Filed: 06/30/2011
• Issued: 03/04/2014
• Est. Priority Date: 09/14/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Nelke, Sebastian, Saillet, Yannick, Seifert, Jens, Oberhofer, Martin A
• Primary Examiner(s): LE, DEBBIE M
• Application Number:
  US13/173,383
• Publication Number:
  US 20120066214A1
• Time in Patent Office: 978 Days
• Field of Search: 707/687, 707/690, 707/758, 707/749, 707/776, 707/780
• US Class Current: 707/758
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 17/10 Complex mathematical operations function generation by table look-up G06F1/03; evaluation of elementary functions by calculation G06F7/544

Abstract

A method, system and computer program product provides a first characteristic associated with a first data set and a single data value, and a second characteristic associated with a second data set; and calculates at least one of: 1) the similarity of the first data set with the second data set based on the first and second characteristics, 2) the similarity of the first data set with the single data value based on the first characteristic and the single data value, 3) confidence indicating how well the first characteristic reflects properties of the first data set based on the first characteristic, and 4) confidence indicating how well the similarity of the first data set with the single data value reflects properties of the single data value based on the first characteristic and the single data value.

Claims

1. A computer implemented method comprising:

determining a first characteristic for a first data set of a plurality of data elements, wherein the first characteristic includes a first set of metric values indicating properties for the plurality of data elements within the first data set;

comparing said first data set with at least one of a second data set of a plurality of data elements and a single data value by at least one of the following;

determining a second characteristic for the second data set, wherein the second characteristic includes a second set of metric values indicating properties for the plurality of data elements within the second data set, and calculating a similarity of said first data set with said second data set based on a comparison of said first and second characteristics; and

determining a third characteristic for the single data value, wherein the third characteristic includes a third set of metric values indicating properties for the single data value, and calculating a similarity of said first data set with said single data value based on a comparison of said first characteristic and said third characteristic.

2. The computer implemented method according to claim 1, wherein:

each respective metric value in said first set of metric values is calculated by applying a respective metric algorithm on the first data set, and

each metric algorithm includes steps for;

calculating at least one similarity value based on a respective metric value in said first set of metric values and at least one of the following;

a value of the same metric algorithm applied to said single data value and a value of the same metric algorithm applied to said second data set, and

calculating at least one confidence value based on said respective metric value in said first set of metric values and at least one of the following;

a value of the same metric algorithm applied to said single data value and a value of the same metric algorithm applied to said second data set.

3. The computer implemented method according to claim 2, wherein each metric algorithm utilizes at least:

a first similarity function to calculate a first similarity value based on said respective metric value of said first data set and a metric value of the same metric algorithm applied to said second data set, and

a first confidence function to calculate a first confidence value indicating how well said metric algorithm reflects properties of said first data set based on said first data set.

4. The computer implemented method according to claim 3, wherein each metric algorithm utilizes at least:

a second similarity function to calculate a second similarity value based on said respective metric value of said first data set and a metric value of the same metric algorithm applied to said single data value, and

a second confidence function to calculate a second confidence value indicating how well said metric algorithm reflects properties of said single data value based on said respective metric value of said first data set.

5. The computer implemented method according to claim 4, further comprising:

calculating a weighted average of similarity values provided by said respective metric algorithms for determining at least one of the following;

said similarity of said first data set with said second data set, and said similarity of said first data set with said single data value,

using at least said first confidence value as a weight for determining similarity of said first data set with said second data set, and

using at least said second confidence value as a weight for determining similarity of said first data set with said single data value.

6. The computer implemented method according to claim 5, wherein metric algorithms associated with said first characteristic contain at least one of the following:

general statistic metric algorithm, most frequent formats metric algorithm, most frequent words metric algorithm, n-grams metric algorithm, and metric algorithm based on neural networks.

7. The computer implemented method according to claim 6, wherein common metric algorithms and the respective metric values are used for the similarity calculation if said first characteristic associated with said first data set and said second characteristics associated with said second data set contain different metric algorithms.

8. The computer implemented method according to claim 1, further comprising storing said first characteristic in a metadata repository.

9. The computer implemented method according to claim 5, further comprising determining based on said first and second characteristics a domain of at least one of the following:

said first data set and said second data set.

10. The computer implemented method according to claim 9, further comprising:

receiving information about the domain of said first data set, and

determining whether said second data set belongs to the same domain based on at least one of the following;

said determined similarity and said determined confidence.

11. The computer implemented method according to claim 10, further comprising at least one of the following steps:

determining that said single data value is inconsistent with said domain of said first data set based on said second similarity value, if said second similarity value does not fulfill a predefined condition, and

determining that said single data value is consistent with said domain of said first data set based on said second similarity value, if said second similarity value fulfills a predefined condition.

12. The computer implemented method according to claim 1, comprising detecting said first data set as a mapping candidate in a data transformation based on said first characteristic.

13. A tangible computer program product comprising a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code configured to:

determine a first characteristic for a first data set of a plurality of data elements, wherein the first characteristic includes a first set of metric values indicating properties for the plurality of data elements within the first data set;

compare said first data set with at least one of a second data set of a plurality of data elements and a single data value by at least one of the following;

determining a second characteristic for the second data set, wherein the second characteristic includes a second set of metric values indicating properties for the plurality of data elements within the second data set, and calculating a similarity of said first data set with said second data set based on a comparison of said first and second characteristics; and

determining a third characteristic for the single data value, wherein the third characteristic includes a third set of metric values indicating properties for the single data value, and calculating a similarity of said first data set with said single data value based on a comparison of said first characteristic and said third characteristic.

14. The tangible computer program product according to claim 13, wherein:

each respective metric value in said first set of metric values is calculated by applying a respective metric algorithm on the first data set, and

each metric algorithm includes steps configured to;

calculate at least one similarity value based on a respective metric value in said first set of metric values and at least one of the following;

a value of the same metric algorithm applied to said single data value and a value of the same metric algorithm applied to said second data set, and

calculate at least one confidence value based on said respective metric value in said first set of metric values and at least one of the following;

a value of the same metric algorithm applied to said single data value and a value of the same metric algorithm applied to said second data set.

15. The tangible computer program product according to claim 14, wherein each metric algorithm utilizes at least:

a first similarity function to calculate a first similarity value based on said respective metric value of said first data set and a metric value of the same metric algorithm applied to said second data set, and

a first confidence function to calculate a first confidence value indicating how well said metric algorithm reflects properties of said first data set based on said first data set.

16. The tangible computer program product according to claim 15, wherein each metric algorithm utilizes at least:

a second similarity function to calculate a second similarity value based on said respective metric value of said first data set and a metric value of the same metric algorithm applied to said single data value, and

a second confidence function to calculate a second confidence value indicating how well said metric algorithm reflects properties of said single data value based on said respective metric value of said first data set.

17. The tangible computer program product according to claim 16, wherein the computer readable program code is further configured to:

calculate a weighted average of similarity values provided by said respective metric algorithms for determining at least one of the following;

said similarity of said first data set with said second data set, and said similarity of said first data set with said single data value,

utilize at least said first confidence value as a weight for determining similarity of said first data set with said second data set, and

utilize at least said second confidence value as a weight for determining similarity of said first data set with said single data value.

18. A system comprising a computer system including at least one processor configured to:

determine a first characteristic for a first data set of a plurality of data elements, wherein the first characteristic includes a first set of metric values indicating properties for the plurality of data elements within the first data set;

compare said first data set with at least one of a second data set of a plurality of data elements and a single data value by at least one of the following;

determining a second characteristic for the second data set, wherein the second characteristic includes a second set of metric values indicating properties for the plurality of data elements within the second data set, and calculating a similarity of said first data set with said second data set based on a comparison of said first and second characteristics; and

determining a third characteristic for the single data value, wherein the third characteristic includes a third set of metric values indicating properties for the single data value, and calculating a similarity of said first data set with said single data value based on a comparison of said first characteristic and said third characteristic.

19. The system according to claim 18, wherein:

each respective metric value in said first set of metric values includes metric values calculated by applying a respective metric algorithm on the first data set, and

each metric algorithm includes steps configured to;

calculate at least one similarity value based on a respective metric value in said first set of metric values and at least one of the following;

a value of the same metric algorithm applied to said single data value and a value of the same metric algorithm applied to said second data set, and

calculate at least one confidence value based on said respective metric value in said first set of metric values and at least one of the following;

a value of the same metric algorithm applied to said single data value and a value of the same metric algorithm applied to said second data set.

20. The system according to claim 19, wherein each metric algorithm utilizes at least:

a first similarity function to calculate a first similarity value based on said respective metric value of said first data set and a metric value of the same metric algorithm applied to said second data set, and

a first confidence function to calculate a first confidence value indicating how well said metric algorithm reflects properties of said first data set based on said first data set.

21. The system according to claim 20, wherein each metric algorithm utilizes at least:

a second similarity function to calculate a second similarity value based on said respective metric value of said first data set and a metric value of the same metric algorithm applied to said single data value, and

a second confidence function to calculate a second confidence value indicating how well said metric algorithm reflects properties of said single data value based on said respective metric value of said first data set.

22. The system according to claim 21, wherein the at least one processor is further configured to:

calculate a weighted average of similarity values provided by said respective metric algorithms for determining at least one of the following;

said similarity of said first data set with said second data set, and said similarity of said first data set with said single data value,

utilize at least said first confidence value as a weight for determining similarity of said first data set with said second data set, and

utilize at least said second confidence value as a weight for determining similarity of said first data set with said single data value.