DATA PROVENANCE AND DATA PEDIGREE TRACKING

• US 20180129712A1
• Filed: 11/09/2016
• Published: 05/10/2018
• Est. Priority Date: 11/09/2016
• Status: Abandoned Application

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Inventors: Mankovskii, Serguei
• Application Number:
  US15/346,958
• Publication Number:
  US 20180129712A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: H04L 65/60 Network streaming of media packets

Abstract

A data provenance and pedigree tracking system may collect, store, and process monitoring data collected by correlators. Monitoring data collected by correlators are events that associate data pedigree, usage rules, and provenance events. Data monitoring may be performed on the data processing and storage functions invoked when performing data analytics for example. The system can determine, maintain and persist association among components, events, rules etc. that contributed to generating a data object result. For example, a data provenance and pedigree tracking system can calculate the total cost of processing the data by adding the processing cost of each component.

Claims

1. A method comprising:

in response to a request from a processing component to access source data,

a data identifier (ID) correlator generating a source data ID, wherein the data ID correlator is loaded in a storage component; and

transmitting the source data in association with the source data ID to the processing component;

the processing component processing the source data to generate result data;

a process ID correlator that is loaded in the processing component,

generating a process ID;

transmitting the result data in association with the process ID and the source data ID to a streaming manager;

transmitting the process ID in association with the processing component ID to the streaming manager; and

the streaming manager linking the association of the result data with the process ID and the source data ID with the association of the processing component ID with the process ID.

2. The method of claim 1, further comprising transmitting the result data in association with the process ID and the source data ID to the storage component.

3. The method of claim 1, further comprising:

generating the request to access source data based on a client request received by a data set processing system that includes the processing component; and

the streaming manager linking a client request ID associated with the client request to the source data ID.

4. The method of claim 1, further comprising:

the data ID correlator,

generating a result data ID for the result data; and

transmitting the process ID in association with the result data ID to the streaming manager; and

the streaming manager generating one or more table records that associate the result data ID with the source data ID and the process ID.

5. The method of claim 1, further comprising:

the data ID correlator,

associating the source data ID with an owner ID, wherein the owner ID is further associated with usage rules for the source data; and

transmitting the source data ID and the owner ID to the streaming manager.

6. The method of claim 1, further comprising:

the data ID correlator and the process ID correlator monitoring a metric of each component of a provenance unit and a pedigree unit respectively from a plurality of metrics; and

the data ID correlator and the process ID correlator transmitting the metric along with the source data ID or result data ID and the process ID to the streaming manager.

7. The method of claim 1, wherein the data ID correlator and the process ID correlator comprise bytecode injected instrumentation.

8. One or more non-transitory machine-readable storage media comprising program code for managing data, the program code to:

in response to a request from a processing component to access source data,

generate a source data ID; and

transmit the source data in association with the source data ID to the processing component;

process the source data to generate result data;

generate a process ID;

transmit the result data in association with the process ID and the source data ID to a streaming manager;

transmit the process ID in association with the processing component ID to the streaming manager; and

link, within relational tables of the streaming manager, the association of the result data with the process ID and the source data ID with the association of the processing component ID with the process ID.

9. The machine-readable storage media of claim 8, wherein the program code further comprises program code to transmit the result data in association with the process ID and the source data ID to the storage component.

10. The machine-readable storage media of claim 8, wherein the program code further comprises program code to:

generate the request to access source data based on a client request received by a data set processing system that includes the processing component; and

link, within the streaming manager, a client request ID associated with the client request to the source data ID.

11. The machine-readable storage media of claim 8, wherein the program code further comprises program code to:

generate a result data ID for the result data; and

transmit the process ID in association with the result data ID to the streaming manager; and

generate one or more table records that associate the result data ID with the source data ID and the process ID.

12. The machine-readable storage media of claim 8, wherein the program code further comprises program code to:

associate the source data ID with an owner ID, wherein the owner ID is further associated with usage rules for the source data; and

transmit the source data ID and the owner ID to the streaming manager.

13. The machine-readable storage media of claim 8, wherein the program code further comprises program code to:

monitor a metric of each component of a provenance unit and a pedigree unit respectively from a plurality of metrics; and

transmit the metric along with the source data ID or result data ID and the process ID to the streaming manager.

14. An apparatus comprising:

a processor; and

a machine-readable medium having program code executable by the processor to cause the apparatus to,

in response to a request from a processing component to access source data,

generate a source data ID; and

transmit the source data in association with the source data ID to the processing component;

process the source data to generate result data;

generate a process ID;

transmit the result data in association with the process ID and the source data ID to a streaming manager;

transmit the process ID in association with the processing component ID to the streaming manager; and

link, within the streaming manager, the association of the result data with the process ID and the source data ID with the association of the processing component ID with the process ID.

15. The apparatus of claim 14, wherein the program code further comprises program code executable by the processor to cause the apparatus to transmit the result data in association with the process ID and the source data ID to the storage component.

16. The apparatus of claim 14, wherein the program code further comprises program code executable by the processor to cause the apparatus to:

generate the request to access source data based on a client request received by a data set processing system that includes the processing component; and

link, within the streaming manager, a client request ID associated with the client request to the source data ID.

17. The apparatus of claim 14, wherein the program code further comprises program code executable by the processor to cause the apparatus to:

generate a result data ID for the result data; and

transmit the process ID in association with the result data ID to the streaming manager; and

generate one or more table records that associate the result data ID with the source data ID and the process ID.

18. The apparatus of claim 14, wherein the program code further comprises program code executable by the processor to cause the apparatus to:

associate the source data ID with an owner ID, wherein the owner ID is further associated with usage rules for the source data; and

transmit the source data ID and the owner ID to the streaming manager.

19. The apparatus of claim 14, wherein the program code further comprises program code executable by the processor to cause the apparatus to:

monitor a metric of each component of a provenance unit and a pedigree unit respectively from a plurality of metrics; and

transmit the metric along with the source data ID or result data ID and the process ID to the streaming manager.

20. The apparatus of claim 14, wherein said generating a source data ID and transmitting the source data in association with the source data ID to the processing component data are performed by a data ID correlator that comprises bytecode injected instrumentation.