Automated identification and classification of critical data elements

• US 10,303,690 B1
• Filed: 11/23/2016
• Issued: 05/28/2019
• Est. Priority Date: 11/23/2016
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Todd, Stephen, Singh, Anand, Latulippe, Barbara
• Primary Examiner(s): Dwivedi, Mahesh H
• Application Number:
  US15/359,916
• Time in Patent Office: 916 Days
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2457 with adaptation to user needs

Abstract

A data governance method comprises the following steps. Data elements from data assets associated with an enterprise are obtained. One or more of the data elements are identified as one or more critical data elements based on a level of criticality computed for each of the one or more data elements. In illustrative embodiments, the level of criticality is based on one or more of: a cardinality computed for the one or more data elements; a business relevance criterion computed for the one or more data elements; and an indirect cross-data lake correlation criterion computed for the one or more data elements.

Claims

1. A method comprising:

obtaining data elements from data assets associated with an enterprise;

identifying one or more of the data elements as one or more critical data elements based on a level of criticality computed for each of the one or more data elements;

wherein the level of criticality is based on a cardinality computed for the one or more data elements and wherein the cardinality of a data element is computed as the total number of nodes in a data lineage map that have consumed the data element or a derivation of the data element;

wherein the level of criticality is based on a correlation criterion computed for the one or more data elements;

wherein the correlation criterion is based on indirect correlation identified between a given data element and portions of the enterprise for which the data element is potentially critical the indirect correlation being calculated through application of a correlation algorithm against the given data element and another data element that is not connected to the given data element in the data lineage map; and

wherein the obtaining and identifying are implemented by one or more processing devices each comprising a processor coupled to a memory.

2. The method of claim 1, further comprising:

generating a rationale for identifying each of the one or more data elements as one or more critical data elements; and

storing the rationale in an accessible data structure referenced by the one or more critical data elements.

3. The method of claim 2, wherein the generated rationale is based on a standardized rationale criteria.

4. The method of claim 1, wherein a data element is identified as a critical data element when its computed cardinality is greater than or equal to a given threshold value.

5. The method of claim 1, wherein the cardinality of a data element is computed as the number of end-user nodes in a data lineage map that have consumed the data element or a derivation of the data element.

6. The method of claim 1, wherein the level of criticality is based on a relevance criterion computed for the one or more data elements.

7. The method of claim 6, wherein the relevance criterion is based on whether or not the data element is directly accessible to an entity of importance to the enterprise.

8. The method of claim 6, wherein the relevance criterion is based on which portions of the enterprise are consuming the data element.

9. The method of claim 6, wherein the relevance criterion is based on how many portions of the enterprise are consuming the data element.

10. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device to:

obtain data elements from data assets associated with an enterprise; and

identify one or more of the data elements as one or more critical data elements based on a level of criticality computed for each of the one or more data elements, the level of criticality being based on a cardinality computed for the one or more data elements, the cardinality of a data element being computed as the total number of nodes in a data lineage map that have consumed the data element or a derivation of the data element;

wherein the level of criticality is based on a correlation criterion of an indirect correlation identified between a given data element and portions of the enterprise for which the data element is potentially critical, the indirect correlation being calculated through application of a correlation algorithm against the given data element and another data element that is not connected to the given data element in the data lineage map.

11. The computer program product of claim 10, wherein the program code when executed by the at least one processing device causes the at least one processing device to also:

generate a rationale for identifying each of the one or more data elements as one or more critical data elements; and

store the rationale in an accessible data structure referenced by the one or more critical data elements.

12. An apparatus comprising:

at least one processing platform accessible to a plurality of user devices over at least one network;

wherein the processing platform implements a critical element manager for data assets of an enterprise, and wherein the critical data element manager is configured to;

obtain data elements from the data assets associated with the enterprise; and

identify one or more of the data elements as one or more critical data elements based on a level of criticality computed for each of the one or more data elements, the level of criticality being based on a cardinality computed for the one or more data elements, the cardinality of the data element being computed as the total number of nodes in a data lineage map that have consumed the data element or a derivation of the data element;

wherein the processing platform is implemented by one or more processing devices each comprising a processor coupled to a memory wherein the level of criticality is based on a correlation criterion of an indirect correlation identified between a given data element and portions of the enterprise for which the data element is potentially critical, the indirect correlation being calculated through application of a correlation algorithm against the given data element and another data element that is not connected to the given data element in the data lineage map.

13. The apparatus of claim 12, wherein the critical data element manager is further configured to:

generate a rationale for identifying each of the one or more data elements as one or more critical data elements; and

store the rationale in an accessible data structure referenced by the one or more critical data elements.

14. The apparatus of claim 13, wherein the generated rationale is based on a standardized rationale criteria.

15. The apparatus of claim 12, wherein the level of criticality is based on a relevance criterion computed for the one or more data elements.

16. The apparatus of claim 12, wherein the level of criticality is based on an indirect correlation criterion computed for the one or more data elements.