System and method for automating data normalization using text analytics

• US 20060112110A1
• Filed: 11/23/2004
• Published: 05/25/2006
• Est. Priority Date: 11/23/2004
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hehenberger, Michael, Maymir-Ducharme, Fred A.
• Granted Patent: US 7,822,768 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing

Abstract

A system, method and program product for normalizing, sanitizing and disambiguating structured data. Structured data includes data stored in a database management system (DBMA), as well labeled files (e.g., XML data). An automated data enhancement processing system is provided, comprising: a system for ingesting data structured in at least one predefined database format; and a set of text analytics processes that treat the ingested data as unstructured, and generate normalized data represented and indexed by consistent, structured metadata.

Claims

1. An automated data enhancement processing system for processing database data, comprising:

a system for ingesting data structured in at least one predefined format; and

a set of text analytics processes that treat the ingested data as unstructured to generate normalized data represented by consistent and structured metadata.

2. The system of claim 1, wherein the system for ingesting data ingests data in at least two disparate database formats.

3. The system of claim 1, wherein the set of text analytics processes includes processes for sanitizing, normalizing and disambiguating the ingested data.

4. The system of claim 1, wherein the set of text analytics processes includes processes selected from the group consisting of:

translating, people extracting, place extracting, entity type extracting, name tokenizing, organization tokenizing, entity tokenizing, relevancy ranking, polysemi filtering, geo-spatial analyzing, and table/list extracting.

5. The system of claim 1, wherein the set of text analytics processes includes language processes selected from the group consisting of:

parts-of-speech tagging, thesaurus matching, and statistical machine translation matching.

6. The system of claim 1, wherein the set of text analytics processes includes polymorphism analysis.

7. The system of claim 1, wherein the normalized data is stored in a data warehouse.

8. A program product stored on a recordable medium, which when executed by a computer, normalizes database data representation and metadata, comprising:

program code configured for ingesting data structured in at least one predefined database format; and

program code providing a set of text analytics processes that treats the ingested data as unstructured, and generates normalized data represented by a consistent and structured metadata.

9. The program product of claim 8, wherein the program code configured for ingesting data ingests data in at least two disparate database formats.

10. The program product of claim 8, wherein the set of text analytics processes includes processes for sanitizing, normalizing and disambiguating the ingested data.

11. The program product of claim 8, wherein the set of text analytics processes includes processes selected from the group consisting of:

translating, people extracting, place extracting, entity type extracting, name tokenizing, organization tokenizing, entity tokenizing, relevancy ranking, polysemi filtering, polymorphism analyzing, geo-spatial analyzing, and table/list extracting.

12. The program product of claim 8, wherein the set of text analytics processes includes a natural language processor.

13. The program product of claim 8, wherein the generated normalized data is indexed.

14. The program product of claim 8, further comprising program code configured for storing the normalized data in a data warehouse.

15. A method for normalizing database data, comprising:

ingesting data structured in at least one predefined database format; and

performing a set of text analytics processes on the ingested data to generate normalized data represented by a consistent and structured metadata, wherein each of the text analytics processes treat the ingested data as unstructured.

16. The method of claim 15, wherein the step of ingesting data ingests data in at least two disparate database formats.

17. The method of claim 15, wherein the set of text analytics processes includes processes for sanitizing, normalizing and disambiguating the ingested data.

18. The method of claim 15, wherein the set of text analytics processes includes processes selected from the group consisting of:

translating, people extracting, place extracting, entity type extracting, name tokenizing, organization tokenizing, entity tokenizing, relevancy ranking, polysemi filtering, polymorphism analyzing, geo-spatial analyzing, and table/list extracting.

19. The method of claim 15, wherein the set of text analytics processes includes a natural language processor.

20. The method of claim 15, wherein the generated normalized data is indexed.

21. The method of claim 15, further comprising the step of storing the normalized data in a data warehouse.

22. A method for deploying an automated data enhancement processing system for normalizing database data, comprising:

providing a computer infrastructure being operable to;

ingest data structured in at least one predefined database format; and

perform a set of text analytics processes on the ingested data to generate normalized data represented by a consistent, indexed and structured metadata, wherein each of the text analytics processes treat the ingested data as unstructured.