System and method for automating data normalization using text analytics

• US 7,822,768 B2
• Filed: 11/23/2004
• Issued: 10/26/2010
• Est. Priority Date: 11/23/2004
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hehenberger, Michael, Maymir-Ducharme, Fred A.
• Primary Examiner(s): Mofiz; Apu M
• Assistant Examiner(s): Jacob; Ajith
• Application Number:
  US10/995,575
• Publication Number:
  US 20060112110A1
• Time in Patent Office: 2,163 Days
• Field of Search: 707/102, 707/776
• US Class Current: 707/776
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing

Abstract

A system, method and program product for normalizing, sanitizing and disambiguating structured data. Structured data includes data stored in a database management system (DBMA), as well labeled files (e.g., XML data). An automated data enhancement processing system is provided, comprising: a system for ingesting data structured in at least one predefined database format; and a set of text analytics processes that treat the ingested data as unstructured, and generate normalized data represented and indexed by consistent, structured metadata.

Claims

1. An automated data enhancement processing system for processing database data, comprising a processor and a memory, the memory including:

a system for ingesting database data structured in at least two disparate predefined database formats used by at least two different database management systems (DBMS),

wherein the structured data are elements of at least one relational database, the elements representing values of well-defined entities;

a set of automated text analytics processes that, when applied to the ingested database data, treat the structured ingested database data as unstructured data to generate normalized data described by consistent and structured metadata,

wherein the set of text analytics processes apply a shared set of metadata tags to the structured database data to describe the normalized data;

a system for processing the normalized data, wherein the processing includes translating and expanding the normalized data into a consistent structure which is ready for mining and OLAP (online analytical processing) applications; and

a system for generating mining and OLAP results using the normalized data.

2. The system of claim 1, wherein the set of text analytics processes includes processes for sanitizing, normalizing and disambiguating the ingested data, wherein

the sanitizing includes identifying and eliminating misspellings, ranges, redundancies, and other errors,

the normalizing includes converting data to a consistent format, and

the disambiguating includes eliminating redundancies, anomalies, and polysemi.

3. The system of claim 1, wherein the set of text analytics processes includes processes selected from the group consisting of:

translating, people extracting, place extracting, entity type extracting, name tokenizing, organization tokenizing, entity tokenizing, relevancy ranking, polysemi filtering, geo-spatial analyzing, and table/list extracting.

4. The system of claim 1, wherein the set of text analytics processes includes language processes selected from the group consisting of:

parts-of-speech tagging, thesaurus matching, and statistical machine translation matching.

5. The system of claim 1, wherein the set of text analytics processes includes polymorphism analysis.

6. The system of claim 1, wherein the normalized data is stored in a data warehouse.

7. A program product stored on a recordable medium, which when executed by a computer, normalizes and processes database data representation and metadata, comprising:

program code configured for ingesting database data structured in at least two disparate predefined database formats used by at least two different database management systems (DBMS),

wherein the structured data are elements of at least one relational database, the elements representing values of well-known entities;

program code providing a set of automated text analytics processes that, when applied to the ingested database data, treat the ingested database data as unstructured data, and generates normalized data described by consistent and structured metadata,

wherein the set of text analytics processes apply a shared set of metadata tags to describe the normalized data;

program code for processing the normalized data, wherein the processing includes translating and expanding the normalized data into a consistent structure which is ready for mining and OLAP (online analytical processing) applications; and

program code for generating mining and OLAP results using the normalized data.

8. The program product of claim 7, wherein the set of text analytics processes includes processes for sanitizing, normalizing and disambiguating the ingested data, wherein

the sanitizing includes identifying and eliminating misspellings, ranges, redundancies, and other errors,

the normalizing includes converting data to a consistent format, and

the disambiguating includes eliminating redundancies, anomalies, and polysemi.

9. The program product of claim 7, wherein the set of text analytics processes includes processes selected from the group consisting of:

translating, people extracting, place extracting, entity type extracting, name tokenizing, organization tokenizing, entity tokenizing, relevancy ranking, polysemi filtering, polymorphism analyzing, geo-spatial analyzing, and table/list extracting.

10. The program product of claim 7, wherein the set of text analytics processes includes a natural language processor.

11. The program product of claim 7, wherein the generated normalized data is indexed.

12. The program product of claim 7, further comprising program code configured for storing the normalized data in a data warehouse.

13. A method for normalizing and processing database data, comprising:

ingesting database data structured in at least two disparate predefined database formats used by at least two different database management systems (DBMS),

wherein the structured data are elements of at least one relational database, the elements representing values of well-defined entities;

performing a set of automated text analytics processes on the ingested database data to generate normalized data described by a consistent and structured metadata,

wherein each of the text analytics processes treat the ingested database data as unstructured data, and

wherein the set of text analytics processes apply a shared set of metadata tags to describe the normalized data;

processing the normalized data, wherein the processing includes translating and expanding the normalized data into a consistent structure which is ready for mining and OLAP (online analytical processing) applications; and

generating mining and OLAP results using the normalized data.

14. The method of claim 13, wherein the set of text analytics processes includes processes for sanitizing, normalizing and disambiguating the ingested data, wherein

the sanitizing includes addressing identifying and eliminating misspellings, ranges, redundancies, and other errors,

the normalizing includes converting data to a consistent format, and

the disambiguating includes eliminating redundancies, anomalies, and polysemi.

15. The method of claim 13, wherein the set of text analytics processes includes processes selected from the group consisting of:

translating, people extracting, place extracting, entity type extracting, name tokenizing, organization tokenizing, entity tokenizing, relevancy ranking, polysemi filtering, polymorphism analyzing, geo-spatial analyzing, and table/list extracting.

16. The method of claim 13, wherein the set of text analytics processes includes a natural language processor.

17. The method of claim 13, wherein the generated normalized data is indexed.

18. The method of claim 13, further comprising the step of storing the normalized data in a data warehouse.

19. A method for deploying an automated data enhancement processing system for normalizing and processing database data, comprising:

providing a computer infrastructure being operable to;

ingest database data structured in at least two disparate predefined database formats used by at least two different database management systems (DBMS),

wherein the structured data are elements of at least one relational database, the elements representing values of well-defined entities;

perform a set of automated text analytics processes on the ingested database data to generate normalized data described by a consistent, indexed and structured metadata,

wherein each of the text analytics processes treat the ingested data as unstructured data, and

wherein the set of text analytics processes apply a shared set of metadata tags to describe the normalized data;

process the normalized data, wherein the processing includes translating and expanding the normalized data into a consistent structure which is ready for mining and OLAP (online analytical processing) applications; and

generate mining and OLAP results using the normalized data.