Data asset reconstruction

• US 9,886,353 B2
• Filed: 12/19/2014
• Issued: 02/06/2018
• Est. Priority Date: 12/19/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Halberstadt, Benjamin, Nizri, Ortal, Sharf, Erel, Steinhart, Shlomo
• Primary Examiner(s): Mobin, Hasanul
• Application Number:
  US14/576,241
• Publication Number:
  US 20160179630A1
• Time in Patent Office: 1,145 Days
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

According to an aspect, data asset reconstruction includes receiving a data lineage for a data asset, where the data lineage identifies a data source connected to the data asset by an intermediate process. It is determined whether the intermediate process can be used to reconstruct a lost data asset. Based on determining that the intermediate process can be used to reconstruct the lost data asset, reconstructing the lost data asset with the intermediate process and the data source.

Claims

1. A system for data asset reconstruction, comprising:

a memory having computer readable instructions; and

a processor for executing the computer readable instructions, the computer readable instructions including;

receiving a data lineage for a data asset, wherein the data lineage identifies a data source connected to the data asset by an intermediate process;

determining whether the data source is downstream or upstream from a lost data asset;

based on determining that the data source is downstream from the lost data asset;

determining whether the intermediate process can be used to reconstruct the lost data asset; and

based on determining that the intermediate process can be used to reconstruct the lost data asset, including determining that the intermediate process is reversible, reconstructing the lost data asset by reversing the intermediate process from the data source to yield the lost data asset,

wherein the intermediate process is reversible if the intermediate process is capable of being run in reverse to create the lost data asset as output using the data source as input; and

based on determining that the data source is upstream from the lost data asset;

determining whether the intermediate process can be used to reconstruct the lost data asset; and

based on determining that the intermediate process can be used to reconstruct the lost data asset, including determining that the intermediate process is re-runnable, reconstructing the lost data asset by re-running the intermediate process from the data source to yield the lost data asset,

wherein the intermediate process is re-runnable if the intermediate process did not change the data source or if the intermediate process is a known one-to-one function.

2. The system recited in claim 1, wherein:

the data source is upstream from the lost data asset in the data lineage.

3. The system recited in claim 1, wherein:

the data source is downstream from the data asset in the data lineage.

4. The system recited in claim 1, wherein the lost data asset is a first lost data asset, the data source is a first data source, the intermediate process is a first intermediate process, the data lineage is a first data lineage, and the computer instructions further include:

identifying a second lost data asset of the first data source upon which reconstruction of the first lost data asset depends;

receiving a second data lineage for the second lost data asset, wherein the second data lineage identifies at least a second data source connected to the second lost data asset by a second intermediate process; and

reconstructing the second lost data asset with the second intermediate process and the second data source prior to reconstructing the first lost data asset.

5. The system recited in claim 1, wherein the computer instructions further include reconstructing a portion of the lost data asset even if a full version of the lost data asset cannot be reconstructed.

14. The system recited in claim 1, wherein reversing the intermediate process from the data source comprises:

based on determining that a function was used as part of the intermediate process to generate the data source, a compensating function of the function is applied to the data source; and

based on determining that more than one function was used as part of the intermediate process to generate the data source, the order of the more than one functions is reversed to establish a new order and the compensating function for each of the more than one function are applied to the data source in the new order.

6. A method for data asset reconstruction comprising:

obtaining, by a processor, a data lineage for a data asset at a memory location, wherein the data lineage identifies a data source related to the data asset by an intermediate process;

determining whether the data source is downstream or upstream from a lost data asset;

based on determining that the data source is downstream from the lost data asset;

analyzing, by the processor, the intermediate process to determine whether it can be used to reconstruct the lost data asset; and

based on determining that the intermediate process can be used to reconstruct the lost data asset, including determining that the intermediate process is reversible, reconstructing, the processor, the lost data asset by reversing the intermediate process from the data source to yield the lost data asset,

wherein the intermediate process is reversible if the intermediate process is capable of being run in reverse to create the lost data asset as output using the data source as input; and

based on determining that the data source is upstream from the lost data asset;

determining whether the intermediate process can be used to reconstruct the lost data asset; and

based on determining that the intermediate process can be used to reconstruct the lost data asset, including determining that the intermediate process is re-runnable, reconstructing the lost data asset by re-running the intermediate process from the data source to yield the lost data asset,

wherein the intermediate process is re-runnable if the intermediate process did not change the data source or if the intermediate process is a known one-to-one function.

7. The method recited in claim 6, wherein:

the data source is upstream from the data asset in the data lineage.

8. The method recited in claim 6, wherein:

the data source is downstream from the data asset in the data lineage.

9. The method recited in claim 8, wherein reversing the intermediate process from the data source comprises:

based on determining that a function was used as part of the intermediate process to generate the data source, a compensating function of the function is applied to the data source, wherein the compensating function reverses processing performed by the function; and

based on determining that more than one function was used as part of the intermediate process to generate the data source, the order of the more than one functions is reversed to establish a new order and the compensating function for each of the more than one function are applied to the data source in the new order.

10. The method recited in claim 6, wherein the lost data asset is a first lost data asset, the data source is a first data source, the intermediate process is a first intermediate process, the data lineage is a first data lineage further comprising:

identifying a second lost data asset of the first data source upon which reconstruction of the first lost data asset depends;

receiving a second data lineage for the second lost data asset, wherein the second data lineage identifies at least a second data source connected to the second lost data asset by a second intermediate process; and

reconstructing the second lost data asset with the second intermediate process and the second data source prior to reconstructing the first lost data asset.

11. A computer program product comprising:

a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to perform a method comprising;

receiving a data lineage for a data asset, wherein the data lineage identifies a data source connected to the data asset by an intermediate process;

determining whether the data source is downstream or upstream from a lost data asset;

based on determining that the data source is downstream from the lost data asset;

determining whether the intermediate process can be used to reconstruct the lost data asset; and

based on determining that the intermediate process can be used to reconstruct the lost data asset, including determining that the intermediate process is reversible, reconstructing the lost data asset by reversing the intermediate process from the data source to yield the lost data asset,

wherein the intermediate process is reversible if the intermediate process is capable of being run in reverse to create the lost data asset as output using the data source as input; and

based on determining that the data source is upstream from the lost data asset;

determining whether the intermediate process can be used to reconstruct the lost data asset; and

based on determining that the intermediate process can be used to reconstruct the lost data asset, including determining that the intermediate process is re-runnable, reconstructing the lost data asset by re-running the intermediate process from the data source to yield the lost data asset,

wherein the intermediate process is re-runnable if the intermediate process did not change the data source or if the intermediate process is a known one-to-one function.

12. The computer program product of claim 11, wherein:

the data source is upstream from the lost data asset in the data lineage.

13. The computer program product of claim 11, wherein

the data source is downstream from the data asset in the data lineage.