Intelligent data munging

• US 10,347,019 B2
• Filed: 08/31/2016
• Issued: 07/09/2019
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Accenture Global Solutions Limited (Accenture PLC)
• Original Assignee: Accenture Global Solutions Limited (Accenture PLC)
• Inventors: Puri, Colin Anil, Deb, Sanghamitra
• Primary Examiner(s): Casanova, Jorge A
• Application Number:
  US15/253,480
• Publication Number:
  US 20170060931A1
• Time in Patent Office: 1,042 Days
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 10/063 Operations research, analysis or management
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs

Abstract

According to examples, intelligent data munging may include ascertaining data that is to be transformed, and determining, based on an analysis of the ascertained data, a sample of the ascertained data. Intelligent data munging may further include enriching the sample of the ascertained data, determining features of the enriched sample of the ascertained data, and determining, based on the features, a transformation to be applied to the enriched sample of the ascertained data to transform the enriched sample of the ascertained data from a first format to a second format. Further, intelligent data munging may include validating the determined transformation, and generating, based on the validation of the determined transformation, a script that is to be applied to the ascertained data to transform the ascertained data from the first format to the second format.

Claims

1. An intelligent data munging system comprising:

a data loader, executed by at least one hardware processor, to

ascertain data that is to be transformed, and

determine, based on an analysis of the ascertained data, a sample of the ascertained data, wherein the sample of the ascertained data is less than the ascertained data;

a data iterator, executed by the at least one hardware processor, to enrich the sample of the ascertained data;

a data transformer, executed by the at least one hardware processor, to

determine features of the enriched sample of the ascertained data, and

determine, based on the features of the enriched sample of the ascertained data, at least one transformation to be applied to the enriched sample of the ascertained data to transform the enriched sample of the ascertained data from a first format to a second format, wherein the at least one transformation is determined from a plurality of available transformations;

a data munging validator, executed by the at least one hardware processor, to validate the at least one determined transformation by analyzing a similarity of a transformation applied to previously transformed data; and

a script generator, executed by the at least one hardware processor, to generate, based on the validation of the at least one determined transformation, at least one script that is to be applied to the ascertained data to transform the ascertained data from the first format to the second format.

2. The intelligent data munging system according to claim 1, further comprising:

a results generator, executed by the at least one hardware processor, to generate a display including the ascertained data and transformed data that is generated based on application of the at least one script to the ascertained data.

3. The intelligent data munging system according to claim 1, wherein the data transformer is to determine, based on the features of the enriched sample of the ascertained data, the at least one transformation to be applied to the enriched sample of the ascertained data to transform the enriched sample of the ascertained data from the first format to the second format by

learning, based on previously determined features that are similar to the features of the enriched sample of the ascertained data, the at least one transformation, and

determining, from the learning of the at least one transformation, the at least one transformation to be applied to the enriched sample of the ascertained data to transform the enriched sample of the ascertained data from the first format to the second format.

4. The intelligent data munging system according to claim 1, wherein the data transformer is to determine features of the enriched sample of the ascertained data by

determining, based on a semantic analysis of the enriched sample of the ascertained data, a semantic profile of the enriched sample of the ascertained data,

determining, based on a syntactic analysis of the enriched sample of the ascertained data, a syntactic profile of the enriched sample of the ascertained data, and

determining, based on the semantic profile and the syntactic profile of the enriched sample of the ascertained data, the features of the enriched sample of the ascertained data.

5. The intelligent data munging system according to claim 4, wherein the data transformer is to determine, based on the semantic analysis of the enriched sample of the ascertained data, the semantic profile of the enriched sample of the ascertained data by

identifying tokens from the enriched sample of the ascertained data,

normalizing the tokens by analyzing the tokens relative to at least one of

a meaning of each of the tokens, and

an abbreviation related to each of the tokens, and

determining, based on an analysis of at least one of an ontology, a glossary, and a function classifier, a class vector that represents the semantic profile of each of the normalized tokens, wherein the class vector represents concepts associated with the normalized tokens.

6. The intelligent data munging system according to claim 4, wherein the data transformer is to determine, based on the syntactic analysis of the enriched sample of the ascertained data, the syntactic profile of the enriched sample of the ascertained data by

identifying tokens from the enriched sample of the ascertained data,

normalizing the tokens by analyzing the tokens relative to at least one of

a meaning of each of the tokens, and

an abbreviation related to each of the tokens, and

determining, based on an analysis of at least one of an ontology, a glossary, and a function classifier, an instance vector that represents the syntactic profile of each of the normalized tokens, wherein the instance vector represents patterns associated with the normalized tokens.

7. The intelligent data munging system according to claim 1, wherein the data iterator is to enrich the sample of the ascertained data by

identifying missing parts of the sample of the ascertained data, and

completing the missing parts of the sample of the ascertained data based on at least one of historical data, and mathematical manipulations associated with the sample of the ascertained data.

8. The intelligent data munging system according to claim 1, wherein the at least one transformation includes adding a human-readable label to a code in the sample of the ascertained data, wherein the human-readable label represents a meaning of the code.

9. The intelligent data munging system according to claim 1, wherein the data munging validator is to validate the at least one determined transformation by

determining whether historical data is similar to the enriched sample of the ascertained data,

in response to a determination that the historical data is similar to the enriched sample of the ascertained data, determining whether at least one transformation applied to the historical data is similar to the at least one determined transformation, and

in response to a determination that the at least one transformation applied to the historical data are similar to the at least one determined transformation, indicating a validity of the at least one determined transformation.

10. A method for intelligent data munging, the method comprising:

ascertaining, by at least one hardware processor, data that is to be transformed;

determining, by the at least one hardware processor, based on an analysis of the ascertained data, a sample of the ascertained data, wherein the sample of the ascertained data is less than the ascertained data;

enriching, by the at least one hardware processor, the sample of the ascertained data;

determining, by the at least one hardware processor, features of the enriched sample of the ascertained data;

determining, by the at least one hardware processor, based on the features of the enriched sample of the ascertained data, at least one transformation to be applied to the enriched sample of the ascertained data to transform the enriched sample of the ascertained data from a first format to a second format, wherein the at least one transformation is determined from a plurality of available transformations;

validating, by the at least one hardware processor, the at least one determined transformation by analyzing a similarity of a transformation applied to previously transformed data; and

generating, by the at least one hardware processor, based on the validation of the at least one determined transformation, at least one script that is to be applied to the ascertained data to transform the ascertained data from the first format to the second format.

11. The method according to claim 10, wherein validating the at least one determined transformation further comprises:

determining whether historical data is similar to the enriched sample of the ascertained data;

in response to a determination that the historical data is similar to the enriched sample of the ascertained data, determining whether at least one transformation applied to the historical data is similar to the at least one determined transformation; and

in response to a determination that the at least one transformation applied to the historical data are similar to the at least one determined transformation, indicating a validity of the at least one determined transformation.

12. The method according to claim 10, wherein determining, based on the features of the enriched sample of the ascertained data, the at least one transformation to be applied to the enriched sample of the ascertained data to transform the enriched sample of the ascertained data from the first format to the second format, further comprises:

learning, based on previously determined features that are similar to the features of the enriched sample of the ascertained data, the at least one transformation; and

determining, from the learning of the at least one transformation, the at least one transformation to be applied to the enriched sample of the ascertained data to transform the enriched sample of the ascertained data from the first format to the second format.

13. The method according to claim 10, wherein determining the features of the enriched sample of the ascertained data further comprises:

determining, based on a semantic analysis of the enriched sample of the ascertained data, a semantic profile of the enriched sample of the ascertained data;

determining, based on a syntactic analysis of the enriched sample of the ascertained data, a syntactic profile of the enriched sample of the ascertained data; and

determining, based on the semantic profile and the syntactic profile of the enriched sample of the ascertained data, the features of the enriched sample of the ascertained data.

14. The method according to claim 13, wherein determining, based on the semantic analysis of the enriched sample of the ascertained data, the semantic profile of the enriched sample of the ascertained data further comprises:

identifying tokens from the enriched sample of the ascertained data;

normalizing the tokens by analyzing the tokens relative to at least one of

a meaning of each of the tokens, and

an abbreviation related to each of the tokens; and

determining, based on an analysis of at least one of an ontology, a glossary, and a function classifier, a class vector that represents the semantic profile of each of the normalized tokens, wherein the class vector represents concepts associated with the normalized tokens.

15. The method according to claim 13, wherein determining, based on the syntactic analysis of the enriched sample of the ascertained data, the syntactic profile of the enriched sample of the ascertained data further comprises:

identifying tokens from the enriched sample of the ascertained data;

normalizing the tokens by analyzing the tokens relative to at least one of

a meaning of each of the tokens, and

an abbreviation related to each of the tokens; and

determining, based on an analysis of at least one of an ontology, a glossary, and a function classifier, an instance vector that represents the syntactic profile of each of the normalized tokens, wherein the instance vector represents patterns associated with the normalized tokens.

16. The method according to claim 10, wherein enriching the sample of the ascertained data further comprises:

identifying missing parts of the sample of the ascertained data; and

completing the missing parts of the sample of the ascertained data based on at least one of historical data, and mathematical manipulations associated with the sample of the ascertained data.

17. A non-transitory computer readable medium having stored thereon machine readable instructions to provide intelligent data munging, the machine readable instructions, when executed, cause a processor to:

ascertain data that is to be transformed;

determine, based on an analysis of the ascertained data, a sample of the ascertained data, wherein the sample of the ascertained data is less than the ascertained data;

enrich the sample of the ascertained data;

determine features of the enriched sample of the ascertained data;

determine, based on the features of the enriched sample of the ascertained data, at least one transformation to be applied to the enriched sample of the ascertained data to transform the enriched sample of the ascertained data from a first format to a second format, wherein the at least one transformation is determined from a plurality of available transformations;

validate the at least one determined transformation by analyzing a similarity of a transformation applied to previously transformed data;

generate, based on the validation of the at least one determined transformation, at least one script that is to be applied to the ascertained data to transform the ascertained data from the first format to the second format;

generate, based on application of the script to the ascertained data, a transformed version of the ascertained data; and

identify, based on the transformation of the ascertained data, an anomaly in the ascertained data.

18. The non-transitory computer readable medium of claim 17, wherein the machine readable instructions to determine the features of the enriched sample of the ascertained data, when executed, further cause the processor to:

determine, based on a semantic analysis of the enriched sample of the ascertained data, a semantic profile of the enriched sample of the ascertained data by

identifying tokens from the enriched sample of the ascertained data,

normalizing the tokens by analyzing the tokens relative to at least one of

a meaning of each of the tokens, and

an abbreviation related to each of the tokens, and

determining, based on an analysis of at least one of an ontology, a glossary, and a function classifier, a class vector that represents the semantic profile of each of the normalized tokens, wherein the class vector represents concepts associated with the normalized tokens;

determine, based on a syntactic analysis of the enriched sample of the ascertained data, a syntactic profile of the enriched sample of the ascertained data; and

determine, based on the semantic profile and the syntactic profile of the enriched sample of the ascertained data, the features of the enriched sample of the ascertained data.

19. The non-transitory computer readable medium of claim 17, wherein the machine readable instructions to determine the features of the enriched sample of the ascertained data, when executed, further cause the processor to:

determine, based on a semantic analysis of the enriched sample of the ascertained data, a semantic profile of the enriched sample of the ascertained data;

determine, based on a syntactic analysis of the enriched sample of the ascertained data, a syntactic profile of the enriched sample of the ascertained data by

identifying tokens from the enriched sample of the ascertained data,

normalizing the tokens by analyzing the tokens relative to at least one of

a meaning of each of the tokens, and

an abbreviation related to each of the tokens, and

determining, based on an analysis of at least one of an ontology, a glossary, and a function classifier, an instance vector that represents the syntactic profile of each of the normalized tokens, wherein the instance vector represents patterns associated with the normalized tokens; and

determine, based on the semantic profile and the syntactic profile of the enriched sample of the ascertained data, the features of the enriched sample of the ascertained data.

20. The non-transitory computer readable medium of claim 17, wherein the machine readable instructions to enrich the sample of the ascertained data, when executed, further cause the processor to:

identify missing parts of the sample of the ascertained data; and

complete the missing parts of the sample of the ascertained data based on at least one of historical data, and mathematical manipulations associated with the sample of the ascertained data.