Automatic discovery of analysis scripts for a dataset

• US 10,229,171 B2
• Filed: 01/11/2016
• Issued: 03/12/2019
• Est. Priority Date: 01/11/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ekambaram, Vijay, Kumar, Pratyush, Prakash, Ashok Pon Kumar Sree
• Primary Examiner(s): Meng, Jau Shya
• Application Number:
  US14/992,851
• Publication Number:
  US 20170199917A1
• Time in Patent Office: 1,156 Days
• Field of Search: None
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 8/70 Software maintenance or management
• CPC Class Codes: G06F 9/445 Program loading or initiating bootstrapping G06F9/4401; security arrangements for program loading or initiating G06F21/57
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

A method of automatic discovery of analysis scripts for a dataset, the method including: utilizing at least one processor to execute computer code that performs the steps of: receiving, at a script searching tool, an input dataset; searching, in a script repository, a plurality of datasets having analysis scripts associated therewith; the searching comprising finding, based on a feature of the input dataset, one or more datasets of the plurality of datasets having the feature; identifying, based on the one or more datasets of the plurality of datasets having the feature, one or more associated analysis scripts; and returning, via a user interface, a result listing of the one or more associated analysis scripts. Other aspects are described and claimed.

Claims

1. A method of automatic discovery of analysis scripts for a dataset, the method comprising:

utilizing at least one processor to execute computer code that performs the steps of;

receiving, at a script searching tool, an input dataset;

extracting at least one feature from the input dataset;

searching, in a script repository, a plurality of datasets having analysis scripts associated therewith, wherein each analysis script comprises program code that analyzes a dataset in a predetermined manner per the program code of the analysis script;

said searching comprising finding, using the at least one feature of the input dataset to search the plurality of datasets, one or more datasets of the plurality of datasets having the at least one feature;

identifying, based on the one or more datasets of the plurality of datasets having the feature, one or more associated analysis scripts, wherein each of the one or more associated analysis scripts comprise at least one analysis script corresponding to at least one of the one or more datasets having the feature;

returning, via a user interface, a result listing of the one or more associated analysis scripts; and

analyzing the input dataset using an analysis script selected by a user from the one or more associated analysis scripts within the result listing.

2. The method of claim 1, comprising prioritizing results of the result listing.

3. The method of claim 2, wherein the prioritizing comprises promoting an analysis script included in the result listing based on a factor selected from the group consisting of a user input, a user profile and a sample dataset.

4. The method of claim 3, wherein the user profile comprises information selected from the group consisting of:

a user search history and one or more datasets stored in a user client repository.

5. The method of claim 1, wherein the extracted feature comprises a feature selected from the group consisting of a dataset column name, a dataset file name, a dataset file header, a dataset file structure, and a dataset type.

6. The method of claim 1, wherein the feature of the input dataset comprises a relational feature.

7. The method of claim 6, wherein the relational feature is selected from the group consisting of a file name pattern, a parallel dataset, and a forked repository.

8. The method of claim 1, wherein the searching comprises employing a trained model to search based on the feature of the input dataset.

9. The method of claim 8, comprising:

receiving user feedback in response to the result listing; and

updating the trained model based on the user feedback.

10. An apparatus for automatic discovery of analysis scripts for a dataset, the apparatus comprising:

at least one processor; and

a computer readable storage medium having computer readable program code embodied therewith and executable by the at least one processor to;

receive, at a script searching tool, an input dataset;

extract at least one feature from the input dataset;

search, in a script repository, a plurality of datasets having analysis scripts associated therewith, wherein each analysis script comprises program code that analyzes a dataset in a predetermined manner per the program code of the analysis script;

said searching comprising finding, using the at least one feature of the input dataset to search the plurality of datasets, one or more datasets of the plurality of datasets having the at least one feature;

identify, based on the one or more datasets of the plurality of datasets having the feature, one or more associated analysis scripts, wherein each of the one or more associated analysis scripts comprise at least one analysis script corresponding to at least one of the one or more datasets having the feature;

return, via a user interface, a result listing of the one or more associated analysis scripts; and

analyzes the input dataset using an analysis script selected by a user from the one or more associated analysis scripts within the result listing.

11. A computer program product for automatic discovery of analysis scripts for a dataset, the computer program product comprising:

a computer readable storage medium having computer readable program code embodied therewith and executable by at least one processor to;

receive, at a script searching tool, an input dataset;

extract at least one feature from the input dataset;

search, in a script repository, a plurality of datasets having analysis scripts associated therewith, wherein each analysis script comprises program code that analyzes a dataset in a predetermined manner per the program code of the analysis script;

said searching comprising finding, using the at least one feature of the input dataset to search the plurality of datasets, one or more datasets of the plurality of datasets having the at least one feature;

identify, based on the one or more datasets of the plurality of datasets having the feature, one or more associated analysis scripts, wherein each of the one or more associated analysis scripts comprise at least one analysis script corresponding to at least one of the one or more datasets having the feature;

return, via a user interface, a result listing of the one or more associated analysis scripts; and

analyzes the input dataset using an analysis script selected by a user from the one or more associated analysis scripts within the result listing.

12. The computer program product of claim 11, further comprising prioritizing results of the result listing.

13. The computer program product of claim 12, wherein the prioritizing comprises promoting an analysis script included in the result listing based on a factor selected from the group consisting of a user input, a user profile and a sample dataset.

14. The computer program product of claim 13, wherein the user profile comprises information selected from the group consisting of:

a user search history and one or more datasets stored in a user client repository.

15. The computer program product of claim 11, wherein the extracted feature comprises a feature selected from the group consisting of a dataset column name, a dataset file name, a dataset file header, a dataset file structure, and a dataset type.

16. The computer program product of claim 11, wherein the searching comprises employing a trained model to search based on the feature of the input dataset.

17. The computer program product of claim 12, comprising:

receiving user feedback in response to the result listing; and

updating the trained model based on the user feedback.

18. A method of automatic discovery of analysis scripts for a dataset, the method comprising:

utilizing at least one processor to execute computer code that performs the steps of;

receiving, at a script searching tool, an input dataset;

extracting at least one feature from the input dataset, said extracting comprising use of one or more features identified from a user specific dataset repository;

searching, in a script repository, a plurality of datasets having analysis scripts associated therewith, wherein each analysis script comprises program code that analyzes a dataset in a predetermined manner per the program code of the analysis script;

said searching comprising finding, using the extracted feature of the input dataset to search the plurality of datasets, one or more datasets of the plurality of datasets having the extracted feature;

identifying, based on the one or more datasets of the plurality of datasets having the extracted feature, one or more associated analysis scripts, wherein each of the one or more associated analysis scripts comprise at least one analysis script corresponding to at least one of the one or more datasets having the extracted feature;

prioritizing, using user specific information, the one or more associated analysis scripts;

returning, via a user interface, a result listing comprising the one or more associated analysis scripts;

said results list being organized according to the prioritizing;

wherein said results list is customized by consultation of the user specific information; and

analyzing the input dataset using an analysis script selected by a user from the one or more associated analysis scripts within the results list.