Normalizing ingested data sets based on fuzzy comparisons to known data sets

• US 9,529,863 B1
• Filed: 12/21/2015
• Issued: 12/27/2016
• Est. Priority Date: 12/21/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: Apptio (International Business Machines Corporation)
• Inventors: Gindin, Jay Randolf, Mihaylov, Svilen, McLachlan, Paul Damien
• Primary Examiner(s): PARK, GRACE A
• Application Number:
  US14/977,368
• Time in Patent Office: 372 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06Q 40/00 Finance; Insurance; Tax strategies; Processing of corporate or income taxes

Abstract

Embodiments are directed towards normalizing ingested data sets based on fuzzy comparisons to known data sets. Raw data sets that each include raw records may be provided to an ingestion engine. Ingestion rules and known data sets may be provided based on the raw records. The ingestion engine may be employed to iteratively execute the ingestion rules. A comparison of the raw records to the known data sets may be performed. Contents of the raw records may be transformed into model record values and stored in model records. A score value that indicates a confidence level that the model records are correct may be provided. An association of the one or more ingestion rules used to transform the raw record contents into the model record values for each of the one or more model records may be added to a data model.

Claims

1. A method for ingesting data for a data model using a network computer that employs one or more processors to execute instructions that perform actions, comprising:

providing one or more raw data sets to an ingestion engine, wherein each raw data set includes one or more raw records;

providing one or more ingestion rules associated with one or more confidence scores and one or more known data sets based on a type of the one or more raw records;

employing the ingestion engine to iteratively execute the one or more ingestion rules, performing further actions, including;

providing a comparison of one or more portions of the one or more raw records to the one or more known data sets;

transforming contents of the one or more raw records into one or more model record values based on the comparison to the one or more known data sets;

storing the one or more model record values in one or more model records;

providing a score value that indicates a confidence level that the one or more model records are correct based on the one or more confidence scores; and

storing an association of the one or more ingestion rules used to transform the raw record contents into the model record values stored in the one or more model records; and

when the score value that indicates the confidence level of the one or more model records is less than a threshold value, performing further actions, including;

providing a user-interface to interactively edit the one or more raw records or the one or more ingestion rules, wherein the edited one or more ingestion rules produce an increase change or a decrease change in the one or more confidence scores, wherein the one or more changed confidence scores are employed to provide the score value; and

storing the one or more model records in a data store, wherein the one or more model records are added to the data model.

2. The method of claim 1, further comprising, providing one or more suggested model record values for the one or more model records based on the one or more raw records and the one or more known data sets.

3. The method of claim 1, wherein the one or more known data sets are provided from one or more external sources.

4. The method of claim 1, wherein providing the comparison of the one or more portions of the one or more raw records to the one or more known data sets further comprises, communicating the one or more portions of the one or more raw records to an external service to perform the comparison.

5. The method of claim 1, further comprising:

when two or more of the one or more ingestion rules matches a raw record, performing further actions;

providing at least one model record for each of the matched two or more ingestion rules; and

allocating one or more resource values corresponding to the raw record to the at least one model records based on weighting values that are associated with the two or more ingestion rules.

6. The method of claim 1, wherein providing the user-interface that enables interactive modification of the one or more model records further comprises:

storing a historical record of the modifications made to the one or more model records; and

generating a report that includes a source and a type of one or more errors that triggered the interactive modification session.

7. The method of claim 1, wherein providing the one or more raw data sets to an ingestion engine, comprises further actions, including:

caching at least a portion of the one or more raw data sets when network communication is disabled; and

providing the cached at least portion of the one or more raw data sets when network communication is enabled.

8. A system for ingesting data for a data model, comprising:

a network computer, comprising;

a transceiver that communicates over the network;

a memory that stores at least instructions; and

a processor device that executes instructions that perform actions, including;

providing one or more raw data sets to an ingestion engine, wherein each raw data set includes one or more raw records;

providing one or more ingestion rules associated with one or more confidence scores and one or more known data sets based on a type of the one or more raw records;

employing the ingestion engine to iteratively execute the one or more ingestion rules, performing further actions, including;

providing a comparison of one or more portions of the one or more raw records to the one or more known data sets;

transforming contents of the one or more raw records into one or more model record values based on the comparison to the one or more known data sets;

storing the one or more model record values in one or more model records;

providing a score value that indicates a confidence level that the one or more model records are correct based on the one or more confidence scores; and

storing an association of the one or more ingestion rules used to transform the raw record contents into the model record values stored in the one or more model records;

when the score value that indicates the confidence level of the one or more model records is less than a threshold value, performing further actions, including;

providing a user-interface to interactively edit the one or more raw records or the one or more ingestion rules, wherein the edited one or more ingestion rules produce an increase change or a decrease change in the one or more confidence scores, wherein the one or more changed confidence scores are employed to provide the score value; and

storing the one or more model records in a data store, wherein the one or more model records are added to the data model; and

a client computer, comprising;

a transceiver that communicates over the network;

a memory that stores at least instructions; and

a processor device that executes instructions that perform actions, including;

providing the user-interface to a user; and

providing one or more user interactions to the ingestion engine.

9. The system of claim 8, further comprising, providing one or more suggested model record values for the one or more model records based on the one or more raw records and the one or more known data sets.

10. The system of claim 8, wherein the one or more known data sets are provided from one or more external sources.

11. The system of claim 8, wherein providing the comparison of the one or more portions of the one or more raw records to the one or more known data sets further comprises, communicating the one or more portions of the one or more raw records to an external service to perform the comparison.

12. The system of claim 8, wherein the network computer processor device executes instructions that perform actions, further comprising:

when two or more of the one or more ingestion rules matches a raw record, performing further actions;

providing at least one model record for each of the matched two or more ingestion rules; and

allocating one or more resource values corresponding to the raw record to the at least one model records based on weighting values that are associated with the two or more ingestion rules.

13. The system of claim 8, wherein providing the user-interface that enables interactive modification of the one or more model records further comprises:

storing a historical record of the modifications made to the one or more model records; and

generating a report that includes a source and a type of one or more errors that triggered the interactive modification session.

14. The system of claim 8, wherein providing the one or more raw data sets to an ingestion engine, comprises further actions, including:

caching at least a portion of the one or more raw data sets when network communication is disabled; and

providing the cached at least portion of the one or more raw data sets when network communication is enabled.

15. A processor readable non-transitory storage media that includes instructions for ingesting data for a data model, wherein execution of the instructions by a hardware processor performs actions, comprising:

providing one or more raw data sets to an ingestion engine, wherein each raw data set includes one or more raw records;

providing one or more ingestion rules associated with one or more confidence scores and one or more known data sets based on a type of the one or more raw records;

employing the ingestion engine to iteratively execute the one or more ingestion rules, performing further actions, including;

providing a comparison of one or more portions of the one or more raw records to the one or more known data sets;

transforming contents of the one or more raw records into one or more model record values based on the comparison to the one or more known data sets;

storing the one or more model record values in one or more model records;

providing a score value that indicates a confidence level that the one or more model records are correct based on the one or more confidence scores; and

storing an association of the one or more ingestion rules used to transform the raw record contents into the model record values stored in the one or more model records;

when the score value that indicates the confidence level of the one or more model records is less than a threshold value, performing further actions, including;

providing a user-interface to interactively edit the one or more raw records or the one or more ingestion rules, wherein the edited one or more ingestion rules produce an increase change or a decrease change in the one or more confidence scores, wherein the one or more changed confidence scores are employed to provide the score value; and

storing the one or more model records in a data store, wherein the one or more model records are added to the data model.

16. The media of claim 15, further comprising, providing one or more suggested model record values for the one or more model records based on the one or more raw records and the one or more known data sets.

17. The media of claim 15, wherein the one or more known data sets are provided from one or more external sources.

18. The media of claim 15, wherein providing the comparison of the one or more portions of the one or more raw records to the one or more known data sets further comprises, communicating the one or more portions of the one or more raw records to an external service to perform the comparison.

19. The media of claim 15, further comprising:

when two or more of the one or more ingestion rules matches a raw record, performing further actions;

providing at least one model record for each of the matched two or more ingestion rules; and

allocating one or more resource values corresponding to the raw record to the at least one model records based on weighting values that are associated with the two or more ingestion rules.

20. The media of claim 15, wherein providing the user-interface that enables interactive modification of the one or more model records further comprises:

storing a historical record of the modifications made to the one or more model records; and

generating a report that includes a source and a type of one or more errors that triggered the interactive modification session.

21. The media of claim 15, wherein providing the one or more raw data sets to an ingestion engine, comprises further actions, including:

caching at least a portion of the one or more raw data sets when network communication is disabled; and

providing the cached at least portion of the one or more raw data sets when network communication is enabled.

22. A network computer for ingesting data for a data model, comprising:

a transceiver that communicates over the network;

a memory that stores at least instructions; and

a processor device that executes instructions that perform actions, including;

providing one or more raw data sets to an ingestion engine, wherein each raw data set includes one or more raw records;

providing one or more ingestion rules associated with one or more confidence scores and one or more known data sets based on a type of the one or more raw records;

employing the ingestion engine to iteratively execute the one or more ingestion rules, performing further actions, including;

providing a comparison of one or more portions of the one or more raw records to the one or more known data sets;

transforming contents of the one or more raw records into one or more model record values based on the comparison to the one or more known data sets;

storing the one or more model record values in one or more model records;

providing a score value that indicates a confidence level that the one or more model records are correct based at least on the one or more confidence scores;

storing an association of the one or more ingestion rules used to transform the raw record contents into the model record values stored in the one or more model records;

when the score value that indicates the confidence level of the one or more model records is less than a threshold value, performing further actions, including;

providing a user-interface to interactively edit the one or more raw records or the one or more ingestion rules, wherein the edited one or more ingestion rules produce an increase change or a decrease change in the one or more confidence scores, wherein the one or more changed confidence scores are employed to provide the score value; and

storing the one or more model records in a data store, wherein the one or more model records are added to the data model.

23. The network computer of claim 22, further comprising, providing one or more suggested model record values for the one or more model records based on the one or more raw records and the one or more known data sets.

24. The network computer of claim 22, wherein the one or more known data sets are provided from one or more external sources.

25. The network computer of claim 22, wherein providing the comparison of the one or more portions of the one or more raw records to the one or more known data sets further comprises, communicating the one or more portions of the one or more raw records to an external service to perform the comparison.

26. The network computer of claim 22, further comprising:

when two or more of the one or more ingestion rules matches a raw record, performing further actions;

providing at least one model record for each of the matched two or more ingestion rules; and

allocating one or more resource values corresponding to the raw record to the at least one model records based on weighting values that are associated with the two or more ingestion rules.

27. The network computer of claim 22, wherein providing the user-interface that enables interactive modification of the one or more model records further comprises:

storing a historical record of the modifications made to the one or more model records; and

generating a report that includes a source and a type of one or more errors that triggered the interactive modification session.

28. The network computer of claim 22, wherein providing the one or more raw data sets to an ingestion engine, comprises further actions, including:

caching at least a portion of the one or more raw data sets when network communication is disabled; and

providing the cached at least portion of the one or more raw data sets when network communication is enabled.