Interestingness of data

• US 8,843,498 B2
• Filed: 09/13/2012
• Issued: 09/23/2014
• Est. Priority Date: 06/29/2011
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Shyr, Jing-Yun, Spisic, Damir, Wright, Raymond, Xu, Jing, Zhang, Xueying
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): Pham, Michael
• Application Number:
  US13/614,335
• Publication Number:
  US 20130007003A1
• Time in Patent Office: 740 Days
• Field of Search: None
• US Class Current: 707/741
• CPC Class Codes: G06F 16/2228 Indexing structures

Abstract

Provided are techniques for analyzing fields. Statistical metrics for each field in a data set are received. A general interestingness index is generated for each field using one or more combination functions that aggregate standardized interestingness sub-indexes. One or more fields are identified as interesting for further analysis using the general interestingness index. One or more expert recommendations for field transformations are constructed for the identified one or more fields.

Claims

1. A method for analyzing fields, comprising:

receiving statistical metrics for each field in a data set;

generating a general interestingness index for each field using one or more combination functions that aggregate standardized interestingness sub-indexes;

identifying one or more fields as interesting for further analysis using the general interestingness index;

constructing one or more expert recommendations for field transformations for the identified one or more fields, wherein the expert recommendations for field transformations indicate how the identified one or more fields in the data set should be updated; and

updating the identified one or more fields in the data set based on the field transformations.

2. The method of claim 1, wherein the general interestingness index comprises an interestingness index for nominal and ordinal fields and an interestingness index for interval and ratio fields.

3. The method of claim 1, further comprising:

calculating the statistical metrics for each field in the data set, wherein the statistical metrics comprise statistics for categorical fields and scale fields.

4. The method of claim 1, further comprising:

calculating multiple interestingness sub-indexes for each field in the data set based on the statistical metrics for that field.

5. The method of claim 4, further comprising:

standardizing the multiple interestingness sub-indexes for each field using one or more scoring functions.

6. The method of claim 1, further comprising:

interpreting at least one of the general interestingness index, the interestingness sub-indexes, and the statistical metrics for each field to construct expert insights for each field.

7. The method of claim 1, further comprising:

presenting the identified one or more fields;

in response to input selecting a field, presenting one or more expert recommendations for field transformations for the selected field; and

receiving input selecting one or more of the field transformations from the expert recommendations, wherein the selected field is updated with the selected one or more field transformations.

8. The method of claim 7, further comprising:

performing additional analysis on the updated field.