Interestingness of data

• US 8,880,532 B2
• Filed: 06/29/2011
• Issued: 11/04/2014
• Est. Priority Date: 06/29/2011
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Shyr, Jing-Yun, Spisic, Damir, Wright, Raymond, Xu, Jing, Zhang, Xueying
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): Pham, Michael
• Application Number:
  US13/172,707
• Publication Number:
  US 20130006998A1
• Time in Patent Office: 1,224 Days
• Field of Search: 707/736
• US Class Current: 707/741
• CPC Class Codes: G06F 16/2228 Indexing structures

Abstract

Provided are techniques for analyzing fields. Statistical metrics for each field in a data set are received. A general interestingness index is generated for each field using one or more combination functions that aggregate standardized interestingness sub-indexes. One or more fields are identified as interesting for further analysis using the general interestingness index. One or more expert recommendations for field transformations are constructed for the identified one or more fields.

Claims

1. A system for analyzing fields, comprising:

a processor; and

storage coupled to the processor, wherein the storage stores a computer program, and wherein the processor executes the computer program to operations, the operations comprising;

receiving statistical metrics for each field in a data set;

generating a general interestingness index for each field using one or more combination functions that aggregate standardized interestingness sub-indexes;

identifying one or more fields as interesting for further analysis using the general interestingness index;

constructing one or more expert recommendations for field transformations for the identified one or more fields, wherein the expert recommendations for field transformations indicate how the identified one or more fields in the data set should be updated; and

updating the identified one or more fields in the data set based on the field transformations.

2. The system of claim 1, wherein the general interestingness index comprises an interestingness index for nominal and ordinal fields and an interestingness index for interval and ratio fields.

3. The system of claim 1, wherein the operations further comprise:

calculating the statistical metrics for each field in the data set, wherein the statistical metrics comprise statistics for categorical fields and scale fields.

4. The system of claim 1, wherein the operations further comprise:

calculating multiple interestingness sub-indexes for each field in the data set based on the statistical metrics for that field.

5. The system of claim 4, wherein the operations further comprise:

standardizing the multiple interestingness sub-indexes for each field using one or more scoring functions.

6. The system of claim 1, wherein the operations further comprise:

interpreting at least one of the general interestingness index, the interestingness sub-indexes, and the statistical metrics for each field to construct expert insights for each field.

7. The system of claim 1, wherein the operations further comprise:

presenting the identified one or more fields;

in response to input selecting a field, presenting one or more expert recommendations for field transformations for the selected field; and

receiving input selecting one or more of the field transformations from the expert recommendations, wherein the selected field is updated with the selected one or more field transformations.

8. The system of claim 7, wherein the operations further comprise:

performing additional analysis on the updated field.

9. A computer program product for analyzing fields, the computer program product comprising:

a computer readable storage medium having computer readable program code embodied therewith, wherein the computer readable program code, when executed by a processor of a computer, is configured to perform;

receiving statistical metrics for each field in a data set;

generating a general interestingness index for each field using one or more combination functions that aggregate standardized interestingness sub-indexes;

identifying one or more fields as interesting for further analysis using the general interestingness index;

constructing one or more expert recommendations for field transformations for the identified one or more fields, wherein the expert recommendations for field transformations indicate how the identified one or more fields in the data set should be updated; and

updating the identified one or more fields in the data set based on the field transformations.

10. The computer program product of claim 9, wherein the general interestingness index comprises an interestingness index for nominal and ordinal fields and an interestingness index for interval and ratio fields.

11. The computer program product of claim 9, the computer readable program code, when executed by the processor of the computer, is configured to perform:

calculating the statistical metrics for each field in the data set, wherein the statistical metrics comprise statistics for categorical fields and scale fields.

12. The computer program product of claim 9, the computer readable program code, when executed by the processor of the computer, is configured to perform:

calculating multiple interestingness sub-indexes for each field in the data set based on the statistical metrics for that field.

13. The computer program product of claim 12, the computer readable program code, when executed by the processor of the computer, is configured to perform:

standardizing the multiple interestingness sub-indexes for each field using one or more scoring functions.

14. The computer program product of claim 9, the computer readable program code, when executed by the processor of the computer, is configured to perform:

interpreting at least one of the general interestingness index, the interestingness sub-indexes, and the statistical metrics for each field to construct expert insights for each field.

15. The computer program product of claim 9, the computer readable program code, when executed by the processor of the computer, is configured to perform:

presenting the identified one or more fields;

in response to input selecting a field, presenting one or more expert recommendations for field transformations for the selected field; and

receiving input selecting one or more of the field transformations from the expert recommendations, wherein the selected field is updated with the selected one or more field transformations.

16. The computer program product of claim 15, the computer readable program code, when executed by the processor of the computer, is configured to perform:

performing additional analysis on the updated field.