Bootstrapping the data lake and glossaries with ‘dataset joins’ metadata from existing application patterns

• US 10,572,494 B2
• Filed: 09/29/2017
• Issued: 02/25/2020
• Est. Priority Date: 03/26/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dantressangle, Patrick, Laws, Simon, Lee, Adrian C. F., Wooldridge, Peter
• Primary Examiner(s): Ortiz Ditren, Belix M
• Application Number:
  US15/719,903
• Publication Number:
  US 20180025061A1
• Time in Patent Office: 879 Days
• Field of Search: None
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases

Abstract

A method to search for at least one relationship pattern in a plurality of runtime artifacts is provided. The method may include detecting at least one data manipulation statement in the plurality of runtime artifacts. The method may also include extracting at least one relationship clause from the detected at least one data manipulation statement. The method may further include parsing the extracted at least one relationship clause. The method may include generating at least one normalized syntax tree based on the parsed at least one relationship clause. The method may also include performing a classification and a snippet discovery on the generated at least one normalized syntax tree.

Claims

1. A method to search for at least one relationship pattern in a plurality of runtime artifacts, the method comprising:

detecting at least one data manipulation statement in the plurality of runtime artifacts;

extracting at least one relationship clause from the detected at least one data manipulation statement;

parsing the extracted at least one relationship clause; and

generating at least one normalized syntax tree based on the parsed at least one relationship clause.

2. The method of claim 1, further comprising:

performing a classification and a snippet discovery on the generated at least one normalized syntax tree;

clustering a plurality of common relationship clauses according to a plurality of linkages that link a plurality of data sources together; and

storing the plurality of common relationship clauses and the plurality of linkages.

3. The method of claim 2, wherein the plurality of common relationship clauses and the plurality of linkages are stored according to at least one of a class, a cluster or a syntax tree.

4. The method of claim 2, further comprising:

proposing a list comprising the clustered common relationship clauses to a user for validation; and

providing a search capability to retrieve at least one probable relationship between at least one of a plurality of data elements, a plurality of business terms, and a plurality of data elements and business terms, wherein the retrieved at least one probable relationship includes a sorted list of a most common predicate used.

5. The method of claim 1, wherein the plurality of runtime artifacts is associated with a data source comprising at least one of an ETL, a database view, a database SQL procedure, a batch file, a plurality of reporting tool metadata, a metadata server, a program, and a script.

6. The method of claim 2, wherein performing a classification comprises classifying a plurality of common relationship clauses according to at least one of a uni-directional classification, a bi-directional classification, and a mapping classification.

7. The method of claim 2, wherein the clustering the plurality of common relationship clauses is performed using a plurality of analytic algorithms.

8. A computer system to search for at least one relationship pattern in a plurality of runtime artifacts, the computer system comprising:

one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of performing a method comprising;

detecting at least one data manipulation statement in the plurality of runtime artifacts;

extracting at least one relationship clause from the detected at least one data manipulation statement;

parsing the extracted at least one relationship clause; and

generating at least one normalized syntax tree based on the parsed at least one relationship clause.

9. The computer system of claim 8, further comprising:

performing a classification and a snippet discovery on the generated at least one normalized syntax tree;

clustering a plurality of common relationship clauses according to a plurality of linkages that link a plurality of data sources together; and

storing the plurality of common relationship clauses and the plurality of linkages.

10. The computer system of claim 9, wherein the plurality of common relationship clauses and the plurality of linkages are stored according to at least one of a class, a cluster or a syntax tree.

11. The computer system of claim 9, further comprising:

proposing a list comprising the clustered common relationship clauses to a user for validation; and

providing a search capability to retrieve at least one probable relationship between at least one of a plurality of data elements, a plurality of business terms, and a plurality of data elements and business terms, wherein the retrieved at least one probable relationship includes a sorted list of a most common predicate used.

12. The computer system of claim 8, wherein the plurality of runtime artifacts is associated with a data source comprising at least one of an ETL, a database view, a database SQL procedure, a batch file, a plurality of reporting tool metadata, a metadata server, a program, and a script.

13. The computer system of claim 9, wherein performing a classification comprises classifying a plurality of common relationship clauses according to at least one of a uni-directional classification, a bi-directional classification, and a mapping classification.

14. The computer system of claim 9, wherein the clustering the plurality of common relationship clauses is performed using a plurality of analytic algorithms.

15. A computer program product, to search for at least one relationship pattern in a plurality of runtime artifacts, the computer program product comprising:

one or more computer-readable storage devices and program instructions stored on at least one of the one or more tangible storage devices, the program instructions executable by a processor, the program instructions comprising;

program instructions to detect at least one data manipulation statement in the plurality of runtime artifacts;

program instructions to extract at least one relationship clause from the detected at least one data manipulation statement;

program instructions to parse the extracted at least one relationship clause; and

program instructions to generate at least one normalized syntax tree based on the parsed at least one relationship clause.

16. The computer program product of claim 15, further comprising:

program instructions to perform a classification and a snippet discovery on the generated at least one normalized syntax tree;

program instructions to cluster a plurality of common relationship clauses according to a plurality of linkages that link a plurality of data sources together; and

program instructions to store the plurality of common relationship clauses and the plurality of linkages.

17. The computer program product of claim 16, wherein the plurality of common relationship clauses and the plurality of linkages are stored according to at least one of a class, a cluster or a syntax tree.

18. The computer program product of claim 16, further comprising:

program instructions to propose a list comprising the clustered common relationship clauses to a user for validation; and

program instructions to provide a search capability to retrieve at least one probable relationship between at least one of a plurality of data elements, a plurality of business terms, and a plurality of data elements and business terms, wherein the retrieved at least one probable relationship includes a sorted list of a most common predicate used.

19. The computer program product of claim 15, wherein the plurality of runtime artifacts is associated with a data source comprising at least one of an ETL, a database view, a database SQL procedure, a batch file, a plurality of reporting tool metadata, a metadata server, a program, and a script.

20. The computer program product of claim 16, wherein performing a classification comprises classifying a plurality of common relationship clauses according to at least one of a uni-directional classification, a bi-directional classification, and a mapping classification.