Metadata-based data valuation

• US 10,528,522 B1
• Filed: 03/17/2016
• Issued: 01/07/2020
• Est. Priority Date: 03/17/2016
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Todd, Stephen, Singh, Anand, Vijendra, Sudhir, Krishnakumar, Narayanan, Ye, Qing
• Primary Examiner(s): Owyang, Michelle N
• Application Number:
  US15/072,557
• Time in Patent Office: 1,391 Days
• Field of Search: 707600-899
• CPC Class Codes: G06F 16/14 Details of searching files based on file metadata
• CPC Class Codes: G06F 16/164 File meta data generation
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06Q 10/10 Office automation; Time management

Abstract

At least one application data set stored in a data repository is obtained. The application data set is analyzed to generate at least one metadata node. The at least one metadata node is combined with at least one other related node to form a hierarchical data structure. One or more valuation algorithms are executed against the hierarchical data structure to calculate a value for the data set represented in the hierarchical data structure.

Claims

1. A method, comprising:

obtaining at least one application data set stored in a data repository, the application data set including source data generated by a source type program, end-user deliverable data generated by a destination type program and intermediate data in between the source data and the end-user deliverable data generated by an intermediate type program;

analyzing the source data to generate one or more source metadata attributes, the one or more source metadata attributes describing the source data in a native form;

performing computations on the source data to generate one or more intermediate metadata attributes, the one or more intermediate metadata attributes describing the intermediate data;

performing computations on at least one of the intermediate data and the source data to generate one or more end-user deliverable metadata attributes, the one or more end-user deliverable metadata attributes describing end-user deliverable data;

populating the at least one or more source metadata attributes into at least one source metadata node, the at least one or more intermediate metadata attributes into at least one intermediate metadata node, and the one or more end-user deliverable attributes into the at least one metadata end user node;

combining the at least one source metadata node, the at least one intermediate metadata node and the at least one end-user metadata node to form a hierarchical data structure, the hierarchical data structure comprising source valuation nodes, intermediate valuation nodes, and end-user valuation nodes, wherein the one or more source metadata attributes populate the source valuation nodes, the one or more intermediate metadata attributes populate the intermediate valuation nodes, and the end-user deliverable metadata attributes populate the end-user valuation nodes, and further wherein one or more source valuation nodes point to at least two intermediate valuation nodes, and one or more intermediate valuation nodes point to one or more end-user valuation nodes;

assigning values to the valuation nodes of the hierarchical data structure;

determining a value for the application data set based on the values assigned to at least a subset of the valuation nodes of the hierarchical data structure, wherein the subset includes source valuation nodes, intermediate valuation nodes and end-user valuation nodes;

wherein determining the value includes executing one or more valuation algorithms against at least the subset of the hierarchical data structure, including assigning a value to a given valuation node;

wherein the value of the given valuation node is based at least in part on a product of a weight attributed to the given node and a value of the valuation node that functions as a root node to the given node;

wherein assigning values further comprises assigning values by traversing the hierarchical data structure from the source valuation nodes to the intermediate valuation nodes to the end-user valuation nodes; and

wherein the steps are implemented via at least one processing device operatively coupled to the data repository.

2. The method of claim 1, further comprising defining one or more relationships between the valuation nodes of the hierarchical data structure.

3. The method of claim 1, wherein:

the one or more source metadata attributes that populate the source valuation nodes comprise one or more of;

source data table definitions, source data view definitions, source data field definitions, and unstructured source data terms;

the one or more intermediate metadata attributes that populate the intermediate valuation nodes comprise one or more of;

intermediate data table definitions, intermediate data view definitions, intermediate data field definitions, and unstructured intermediate data terms; and

the one or more end-user deliverable metadata attributes that populate the end-user valuation nodes comprise one or more of;

end-user data table definitions, end-user data view definitions, end-user data field definitions, and unstructured end-user data terms.

4. The method of claim 1, wherein the one or more end-user deliverable metadata attributes comprise business type metadata, and the one or more source metadata attributes and the one or more intermediate metadata attributes comprise technical type metadata.

5. The method of claim 1, wherein a given source valuation node is assigned a value equal to a number of valuation nodes to which the given source valuation node contributes divided by the total number of intermediate valuation nodes and end-user valuation nodes.

6. The method of claim 1, wherein assigning values further comprises assigning values by an end user assigning values to at least a portion of the valuation nodes in the structure.

7. The method of claim 1, further comprising modifying the hierarchical data structure by at least one of adding a node, deleting a node, and updating a node.

8. The method of claim 1, wherein the value of the application data set is calculated as a function of at least one of when the value is calculated and from where calculation of the value is requested.

9. The method of claim 1, further comprising:

identifying, as a candidate data set from the data repository, an application data set that has a data valuation at or below a given valuation threshold; and

at least one of deleting one or more candidate data sets from the data repository, and locking one or more candidate data sets to prevent deletion from the data repository.

10. An article of manufacture comprising a processor-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by one or more processing devices implement the steps of:

obtaining at least one application data set stored in a data repository, the application data set including source data generated by a source type program, end-user deliverable data generated by a destination type program and intermediate data in between the source data and the end-user deliverable data generated by an intermediate type program;

analyzing the source data to generate one or more source metadata attributes, the one or more source metadata attributes describing the source data in a native form;

performing computations on the source data to generate one or more intermediate metadata attributes, the one or more intermediate metadata attributes describing the intermediate data;

performing computations on at least one of the intermediate data and the source data to generate one or more end-user deliverable metadata attributes, the one or more end-user deliverable metadata attributes describing end-user deliverable data;

populating the at least one or more source metadata attributes into at least one source metadata node, the at least one or more intermediate metadata attributes into at least one intermediate metadata node, and the one or more end-user deliverable attributes into the at least one metadata end user node;

combining the at least one source metadata node, the at least one intermediate metadata node and the at least one end-user metadata node to form a hierarchical data structure, the hierarchical data structure comprising source valuation nodes, intermediate valuation nodes, and end-user valuation nodes, wherein the one or more source metadata attributes populate the source valuation nodes, the one or more intermediate metadata attributes populate the intermediate valuation nodes, and the end-user deliverable metadata attributes populate the end-user valuation nodes, and further wherein one or more source valuation nodes point to at least two intermediate valuation nodes, and one or more intermediate valuation nodes point to one or more end-user valuation nodes;

assigning values to the valuation nodes of the hierarchical data structure;

determining a value for the application data set based on the values assigned to at least a subset of the valuation nodes of the hierarchical data structure, wherein the subset includes source valuation nodes, intermediate valuation nodes and end-user valuation nodes;

wherein determining the value includes executing one or more valuation algorithms against at least the subset of the hierarchical data structure, including assigning a value to a given valuation node;

wherein the value of the given valuation node is based at least in part on a product of a weight attributed to the given node and a value of the valuation node that functions as a root node to the given node; and

wherein assigning values further comprises assigning values by traversing the hierarchical data structure from the source valuation nodes to the intermediate valuation nodes to the end-user valuation nodes.

11. The article of manufacture of claim 10, wherein a given source valuation node is assigned a value equal to a number of valuation nodes to which the given source valuation node contributes divided by the total number of intermediate valuation nodes and end-user valuation nodes.

12. The article of manufacture of claim 10, wherein:

the one or more source metadata attributes that populate the source valuation nodes comprise one or more of;

source data table definitions, source data view definitions, source data field definitions, and unstructured source data terms;

the one or more intermediate metadata attributes that populate the intermediate valuation nodes comprise one or more of;

intermediate data table definitions, intermediate data view definitions, intermediate data field definitions, and unstructured intermediate data terms; and

the one or more end-user deliverable metadata attributes that populate the end-user valuation nodes comprise one or more of;

end-user data table definitions, end-user data view definitions, end-user data field definitions, and unstructured end-user data terms.

13. The article of manufacture of claim 10, wherein the value of the application data set is calculated as a function of at least one of when the value is calculated and from where calculation of the value is requested.

14. The article of manufacture of claim 10, wherein the one or more end-user deliverable metadata attributes comprise business type metadata, and the one or more source metadata attributes and the one or more intermediate metadata attributes comprise technical type metadata.

15. A system comprising:

one or more processors operatively coupled to one or more memories configured to;

obtain at least one application data set stored in a data repository, the application data set including source data generated by a source type program, end-user deliverable data generated by a destination type program and intermediate data in between the source data and the end-user deliverable data generated by an intermediate type program;

analyze the source data to generate one or more source metadata attributes, the one or more source metadata attributes describing the source data in a native form;

perform computations on the source data to generate one or more intermediate metadata attributes, the one or more intermediate metadata attributes describing the intermediate data;

perform computations on at least one of the intermediate data and the source data to generate one or more end-user deliverable metadata attributes, the one or more end-user deliverable metadata attributes describing end-user deliverable data;

populate the at least one or more source metadata attributes into at least one source metadata node, the at least one or more intermediate metadata attributes into at least one intermediate metadata node, and the one or more end-user deliverable attributes into the at least one metadata end user node;

combine the at least one source metadata node, the at least one intermediate metadata node with the at least one end-user metadata node to form a hierarchical data structure, the hierarchical data structure comprising source valuation nodes, intermediate valuation nodes, and end-user valuation nodes, wherein the one or more source metadata attributes populate the source valuation nodes, the one or more intermediate metadata attributes populate the intermediate valuation nodes, and the end-user deliverable metadata attributes populate the end-user valuation nodes, and further wherein one or more source valuation nodes point to at least two intermediate valuation nodes, and one or more intermediate valuation nodes point to one or more end-user valuation nodes;

assign values to the valuation nodes of the hierarchical data structure;

determine a value for the application data set based on the values assigned to at least a subset of the valuation nodes of the hierarchical data structure, wherein the subset includes source valuation nodes, intermediate valuation nodes and end-user valuation nodes;

wherein determining the value includes executing one or more valuation algorithms against at least the subset of the hierarchical data structure, including assigning a value to a given valuation node;

wherein the value of the given valuation node is based at least in part on a product of a weight attributed to the given node and a value of the valuation node that functions as a root node to the given node;

wherein assigning values further comprises assigning values by traversing the hierarchical data structure from the source valuation nodes to the intermediate valuation nodes to the end-user valuation nodes.

16. The system of claim 15, wherein a given source valuation node is assigned a value equal to a number of valuation nodes to which the given source valuation node contributes divided by the total number of intermediate valuation nodes and end-user valuation nodes.

17. The system of claim 15, wherein:

the one or more source metadata attributes that populate the source valuation nodes comprise one or more of;

source data table definitions, source data view definitions, source data field definitions, and unstructured source data terms;

the one or more intermediate metadata attributes that populate the intermediate valuation nodes comprise one or more of;

intermediate data table definitions, intermediate data view definitions, intermediate data field definitions, and unstructured intermediate data terms; and

the one or more end-user deliverable metadata attributes that populate the end-user valuation nodes comprise one or more of;

end-user data table definitions, end-user data view definitions, end-user data field definitions, and unstructured end-user data terms.

18. The system of claim 15, wherein the one or more end-user deliverable metadata attributes comprise business type metadata, and the one or more source metadata attributes and the one or more intermediate metadata attributes comprise technical type metadata.

19. The system of claim 15, wherein the value of the application data set is calculated as a function of at least one of when the value is calculated and from where calculation of the value is requested.

20. The system of claim 15, further comprising modifying the hierarchical data structure by at least one of adding a node, deleting a node, and updating a node.