Performing a computation using provenance data

• US 10,664,316 B2
• Filed: 03/30/2017
• Issued: 05/26/2020
• Est. Priority Date: 01/31/2017
• Status: Active Grant

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Inventors: Murugan, Muthukumar, Bhattacharya, Suparna, Voigt, Douglas L., Tom, Ancy Sarah
• Primary Examiner(s): Onat, Umut
• Application Number:
  US15/473,894
• Publication Number:
  US 20180217883A1
• Time in Patent Office: 1,153 Days
• Field of Search: None
• CPC Class Codes: G06F 9/5033 considering data affinity

Abstract

Example implementations relate to performing computations using provenance data. An example implementation includes storing first lineage data of a first dataset and provenance data of an application operating on the first dataset in a storage system. A computing resource may determine whether second lineage data of a second dataset meets a similarity criterion with the first lineage data of the first dataset. A computation on the second dataset may be performed using the provenance data of the application, and an insight of the second dataset may be generated from the performed computation.

Claims

1. A method comprising:

storing first lineage data of a first dataset and provenance data of an application at an application layer operating on the first dataset in a storage system;

determining, by a computing resource collocated within the storage system, that second lineage data of a second dataset meets a similarity criterion with the first lineage data of the first dataset;

in response to the determination, performing a proactive computation on the second dataset using the provenance data of the application, wherein the proactive computation is at least a partial processing of the second dataset for generating an insight performed without being prompted to do so by a request from the application layer; and

generating the insight of the second dataset from the performed computation.

2. The method of claim 1, further comprising storing the insight of the second dataset and third lineage data of the generated insight of the second dataset in the storage system.

3. The method of claim 1, wherein the provenance data of the application comprises a series of workflow operations of the application, or the location of an insight of the first dataset generated by the application workflow operating on the first dataset.

4. The method of claim 1, wherein the first lineage data comprises an origin of the first dataset, a workflow applied to the first dataset, or a storage path of the first dataset.

5. The method of claim 1, wherein the application is within an application layer of a computing system.

6. The method of claim 5, further comprising:

receiving a communication associated with the insight of the second dataset from the application layer; and

wherein the insight of the second dataset is provided to the application responsive to receiving the communication.

7. The method of claim 6, wherein the received communication is a request of the application to perform a computation on the second dataset to generate the insight of the second dataset.

8. The method of claim 1, further comprising:

performing a proactive computation on a third dataset using the provenance data of the application; and

merging an insight of the third dataset with the insight of the second dataset.

9. The method of claim 1, wherein the provenance data of the application is a series of workflow operations, and further comprising assigning a cost value to each of the workflow operations in the series upon performing the computation on the second dataset.

10. The method of claim 9, wherein the series of workflow operations in total exceeds a cost threshold, and wherein the insight of the second dataset is an output of the series of workflow operations partially computed such that the cost threshold is not exceeded.

11. The method of claim 1, further comprising:

assigning a confidence value to the third lineage data of the generated insight of the second dataset, wherein the confidence value corresponds to a determined value of the third lineage data; and

sorting the insight of the second dataset in a storage tier corresponding to the confidence value of the third lineage data.

12. The method of claim 11, wherein the determined value of the third lineage data is a frequency at which a request to generate the insight of the second dataset occurs.

13. A non-transitory computer readable medium having instructions executable by a processor to:

within a storage system,

store first lineage data of a first dataset and provenance data of an application operating on the first dataset;

predict that the application operating on the first dataset will attempt to operate on a second dataset by determining that second lineage data of the second dataset meets a similarity criterion with the first lineage data of the first dataset;

in response to the prediction, perform a proactive computation on the second dataset using the provenance data of the application, wherein the computation comprises operating on multiple data sample portions of the second dataset, and wherein the proactive computation is at least a partial processing of the second dataset for generating an insight performed without being prompted to do so by a request from the application;

identify a data sample portion of the multiple data sample portions that satisfies a second criterion; and

store the insight of the second dataset generated from the performed computation, wherein the insight is the output of the performed computation on the identified data sample portion.

14. The non-transitory computer readable medium of claim 13, wherein the provenance data of the application comprises an error metric of the application.

15. The non-transitory computer readable medium of claim 14, wherein the second criterion is meeting an error threshold according to the error metric of the application.

16. The non-transitory computer readable medium of claim 15, wherein the multiple data sample portions comprises a first data sample portion determined using a first sampling strategy, and a second data sample portion determined using a second sampling strategy different from the first sampling strategy.

17. The non-transitory computer readable medium of claim 16, wherein the second criterion is a data sample portion of the multiple data sample portions having a generated insight of the data sample portion with the lowest error rate among insights generated by the multiple data sample portions.

18. The non-transitory computer readable medium of claim 15, wherein the second criterion is a data sample portion of the multiple data sample portions being the smallest data sample portion among the multiple data sample portions satisfying the error threshold.

19. A storage system comprising:

a storage resource for storing first lineage data of a first dataset and provenance data of an application at an application layer operating on the first dataset;

a processor; and

a non-transitory machine-readable storage medium comprising instructions executable by the processor to;

predict that the application operating on the first dataset will attempt to operate on a second dataset by determining that second lineage data of a second dataset meets a similarity criterion with the first lineage data of the first dataset;

in response to the prediction, perform a proactive computation on the second dataset using the provenance data of the application, wherein the proactive computation is at least a partial processing of the second dataset for generating an insight performed without being prompted to do so by a request from the application layer; and

store the insight of the second dataset generated from the performed computation.

20. The storage system of claim 19, further comprising a task receptor to:

receive a communication associated with the first insight; and

provide the insight of the second dataset to the application responsive to the received communication.