MANAGING LINEAGE INFORMATION

• US 20160019286A1
• Filed: 07/20/2015
• Published: 01/21/2016
• Est. Priority Date: 07/18/2014
• Status: Active Grant

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: AB Lnitio Technology LLC
• Inventors: Larson, Brond, Bach, Edward, Oberdorf, Richard
• Granted Patent: US 10,175,974 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/116 Details of conversion of file system types or formats
• CPC Class Codes: G06F 16/119 Details of migration of file systems migration mechanisms in storage systems G06F3/0647
• CPC Class Codes: G06F 16/17 Details of further file system functions
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 8/10 Requirements analysis; Specification techniques
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: G06F 8/70 Software maintenance or management
• CPC Class Codes: G06F 9/44505 Configuring for program initiating, e.g. using registry, configuration files
• CPC Class Codes: G06F 9/4494 data driven
• CPC Class Codes: G06F 9/54 Interprogram communication
• CPC Class Codes: G06Q 10/0633 Workflow analysis

Abstract

Managing lineage information includes: receiving lineage information representing one or more lineage relationships among two or more data processing programs and two or more logical datasets; receiving one or more runtime artifacts, each runtime artifact including information related to a previous execution of a data processing program of the two or more data processing programs; and analyzing the one or more runtime artifacts and the lineage information to determine one or more candidate modifications to the lineage information.

Claims

1. A method for managing lineage information, the method including:

receiving lineage information representing one or more lineage relationships among two or more data processing programs and two or more logical datasets;

receiving one or more runtime artifacts, each runtime artifact including information related to a previous execution of a data processing program of the two or more data processing programs; and

analyzing the one or more runtime artifacts and the lineage information to determine one or more candidate modifications to the lineage information.

2. The method of claim 1 wherein the one or more candidate modifications include a candidate modification that adds a new indirect lineage relationship between a data processing program of the two or more data processing programs and a logical dataset of the two or more logical datasets.

3. The method of claim 1 wherein the one or more candidate modifications include a first candidate modification that adds a new direct lineage relationship between a data processing program of the two or more data processing programs and a logical dataset of the two or more logical datasets.

4. The method of claim 3 wherein analyzing the runtime artifacts and the lineage information includes analyzing logs of previous executions of the two or more data processing programs to determine physical datasets read from or written to by the two or more data processing programs.

5. The method of claim 4 wherein analyzing the runtime artifacts and the lineage information further includes identifying two distinct logical datasets of the two or more logical datasets that are represented in the lineage information and are associated with the same physical dataset.

6. The method of claim 5 wherein the first candidate modification includes creation of the new lineage relationship between the two distinct logical datasets.

7. The method of claim 5 wherein the first candidate modification includes creation of the new lineage relationship including merging the two distinct logical datasets into a new combined logical dataset.

8. The method of claim 1 wherein each data processing program of the two or more data processing programs is an instance of a generic data processing program instantiated according to a set of one or more parameter values.

9. The method of claim 8 wherein analyzing the one or more runtime artifacts and the lineage information includes:

analyzing one or more logs of previous executions of a first data processing program of the two or more data processing programs to determine a first parameter set used in a first instantiation of the first data processing program according to a first set of one or more parameter values,

selecting at least some parameters from the first parameter set, and

determining that the first instantiation of the first data processing program is not represented in the lineage information based on a generic version of the first data processing program and the at least some parameters.

10. The method of claim 9 wherein selecting at least some parameters from the first parameter set includes selecting parameters based on information received from a user.

11. The method of claim 9 wherein selecting at least some parameters from the first parameter set includes selecting parameters based on one or more predefined rules.

12. The method of claim 11 wherein a first rule of the one or more predefined rules specifies that parameters with parameter values in the form of a date are excluded from the selected parameters.

13. The method of claim 11 wherein a first rule of the one or more predefined rules specifies that a parameter with a parameter value that is transformed in the logic of a generic data processing program is included in the selected parameters.

14. The method of claim 9 wherein the one or more candidate modifications to the lineage information includes a first candidate modification that adds a new lineage relationship between the first data processing program of the two or more data processing programs and a logical dataset of the two or more logical datasets.

15. Software for managing lineage information stored in a non-transitory form on a computer-readable medium, the software including instructions for causing a computing system to:

receive lineage information representing one or more lineage relationships among two or more data processing programs and two or more logical datasets;

receive one or more runtime artifacts, each runtime artifact including information related to a previous execution of a data processing program of the two or more data processing programs; and

analyze the one or more runtime artifacts and the lineage information to determine one or more candidate modifications to the lineage information.

16. A computing system for managing lineage information, the computing system including:

an input device or port configured to receive lineage information representing one or more lineage relationships among two or more data processing programs and two or more logical datasets and one or more runtime artifacts, each runtime artifact including information related to a previous execution of a data processing program of the two or more data processing programs; and

at least one processor configured to analyze the one or more runtime artifacts and the lineage information to determine one or more candidate modifications to the lineage information.

17. A computing system for managing lineage information, the computing system including:

means for receiving lineage information representing one or more lineage relationships among two or more data processing programs and two or more logical datasets and one or more runtime artifacts, each runtime artifact including information related to a previous execution of a data processing program of the two or more data processing programs; and

means for analyzing the one or more runtime artifacts and the lineage information to determine one or more candidate modifications to the lineage information.