SYSTEM FOR AUTOMATICALLY TRACKING DATA THROUGH A PLURALITY OF DATA SOURCES AND PLURALITY OF CONVERSIONS

• US 20160314143A1
• Filed: 09/07/2015
• Published: 10/27/2016
• Est. Priority Date: 04/25/2015
• Status: Active Grant

• Current Assignee: Phillip Hiroshige
• Original Assignee: Phillip Hiroshige
• Inventors: Hiroshige, Phillip
• Granted Patent: US 10,140,348 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

A networked computing system where data from multiple sources is automatically tracked through a plurality of conversion steps at the data row and element level. The system allows input of data conversion metadata. The system automatically applies a version number to the conversion metadata. The system includes a data conversion engine. The data conversion engine ensures that processing metadata versions are in sync with versions of converted data. The conversion engine creates version synchronization at the data row and element level thus allowing data tracking at the data row and element level. The system includes a database that stores conversion metadata and execution attributes and a graphical user interface that allows the user access to the conversion metadata.

Claims

1. A method of data tracking a source dataset comprising a plurality of data elements in a computer system which has a memory and a processor, the method comprising the steps of:

storing data conversion instructions for one or more data elements as conversion metadata associated with the source dataset in the memory;

creating a unique version number with the processor based on the conversion metadata;

creating a conversion program from the conversion instructions of the conversion metadata;

running the conversion program to perform a data conversion with the processor on one or more data elements of the source dataset to form a target dataset comprising one or more converted data elements; and

associating and storing the unique version number with the conversion metadata, with the conversion program, and with the target dataset with the processor in the memory.

2. The method of claim 1, wherein a unique version number is created when the conversion metadata is modified.

3. The method of claim 2, wherein the unique version number created when the conversion metadata is modified is associated and stored with the conversion metadata, with the conversion program, and with the target dataset with the processor in the memory.

4. The method of claim 1, further comprising the step of storing the source dataset, conversion metadata with the associated unique version number, and the target dataset with the associated unique version number as a data build.

5. The method of claim 1, wherein the conversion program creates execution attributes.

6. The method of claim 3, wherein the execution attributes are stored in the data build.

7. The method of claim 4, wherein the target dataset of a first data build is used as a second source dataset to create a second data build comprising the second source dataset, a second conversion metadata with a second associated unique version number, and a second target dataset with the second associated unique version number.

8. The method of claim 5, wherein the first data build and second data build are stored as a data flow.

9. The method of claim 6, wherein the data flow comprises lineage data describing the relationship between the first data build and the second data build.

10. The method of claim 9, wherein a graphic user interface is configured to present summary information of the data builds in a data flow, and wherein the graphic user interface may be used to select an item of summary information regarding a data build, and the conversion metadata associated with the summary information of that data build may then be presented to the user by the graphic user interface.

11. A method of data tracking a source dataset comprising a plurality of data elements in a computer system which has a memory and a processor, the method comprising the steps of:

storing data conversion instructions for one or more data elements as conversion metadata associated with the source dataset in the memory;

creating a unique version number with the processor based on the conversion metadata;

creating a conversion program from the conversion instructions of the conversion metadata;

running the conversion program to perform a data conversion with the processor on one or more data elements of the source dataset to form a target dataset comprising one or more converted data elements; and

associating and storing the unique version number with the conversion metadata, with the conversion program, and with each converted data element with the processor in the memory.

12. The method of claim 11, wherein a unique version number is created when the conversion metadata is modified and, wherein the unique version number created when the conversion metadata is modified is associated and stored with the conversion metadata, with the conversion program, and with each converted data element with the processor in the memory.

13. The method of claim 12, further comprising the step of associating and storing the unique version number with the target dataset.

14. The method of claim 13, further comprising the step of storing the source dataset, conversion metadata with the associated unique version number, and the target dataset with the associated unique version number as a data build.

15. The method of claim 13, wherein the conversion program creates execution attributes.

16. The method of claim 15, wherein the execution attributes are stored in the data build.

17. The method of claim 16, wherein the target dataset of a first data build is used as a second source dataset to create a second data build comprising the second source dataset, a second conversion metadata with a second associated unique version number, and a second converted dataset with the second associated unique version number.

18. The method of claim 17, wherein the first data build and second data build are stored as a data flow.

19. The method of claim 18, wherein the data flow comprises lineage data describing the relationship between the first data build and the second data build.

20. The method of claim 19, wherein a graphic user interface is configured to present summary information of the data builds in a data flow, and wherein the graphic user interface may be used to select an item of summary information regarding a data build, and the conversion metadata associated with the summary information of that data build may then be presented to the user by the graphic user interface.