Method and system for processing structured data and unstructured data

• US 7,668,849 B1
• Filed: 12/09/2005
• Issued: 02/23/2010
• Est. Priority Date: 12/09/2005
• Status: Active Grant

• Current Assignee: Bmmsoft Incorporated
• Original Assignee: Bmmsoft Incorporated
• Inventors: Narancic, Perry J., Krneta, Paul
• Primary Examiner(s): Wong; Don
• Assistant Examiner(s): XUE, BIN
• Application Number:
  US11/298,261
• Time in Patent Office: 1,537 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation

Abstract

A method for processing data is provided. In this method, unstructured data and structured data are captured and the unstructured data is associated with the structured data. After capture, the unstructured data and the structured data are correlated to define a link between the unstructured data and the structured data. The unstructured data and the structured data then are stored in a data structure based on the link. A system for processing data also is described.

Claims

1. A computer implemented method for processing data, the method comprising:

(a) executing, on a data processing, programming for providing a plurality of data processing modules, said data processing modules comprising;

an e-mail capture and parsing engine for intercepting, copying, and processing e-mails transmitted from a gateway to an email server, said processing comprising dividing a copy of an email into sections including at least a header section and a body section, and dividing the header section into sections comprising one or more of sender email address, sender name, recipient email address, recipient name, summary of email contents, date the email was sent, or time the email was sent;

an email reload engine for loading data associated with archived or stored emails;

a web crawler engine for Internet crawling and capturing Internet web pages;

a document gathering engine for capturing data from external sources comprising one or more of an application data warehouse, application server, or file system;

one or more data staging areas for temporary storage of data associated with said email capture and parsing engine, said web crawler engine, and said document gathering engine;

a text extraction and parsing engine for receiving data from said data staging areas, extracting structured data from unstructured data, and correlating extracted structured data and associated unstructured data to define a link between said structured data and said associated unstructured data;

a data holding area for temporary storage of said structured data and said unstructured data from said text extraction and parsing engine;

a data loading engine for loading and storing into a database management system said structured data and said unstructured data from said data holding area based on said link;

an email account management engine for bypassing said text extraction and parsing engine and directly copying structured data from said email server to said database management system;

(b) using said plurality of data processing modules, carrying out operations comprising;

capturing unstructured data from an unstructured data source and structured data from a structured data source, the unstructured data being associated with the structured data, wherein the unstructured data source and the structured data source are each associated with at least an email, the email including a header;

parsing the header into at least a sending email address, a receiving email address, a date and time of transmission associated with the email, and a carbon copy email address;

evaluating the email using the email capture and parsing engine, wherein the email capture and parsing engine generates a summary of content associated with the email, the sending email address, the receiving email address, the date and the time of transmission associated with the email, the carbon copy email address, and a summary used to classify the email;

correlating the unstructured data and the structured data to establish a link between the unstructured data and the structured data, wherein the link integrates the unstructured data and the structured data; and

storing the unstructured data and the structured data in a first data structure, wherein the unstructured data is stored in an unstructured portion of the first data structure, wherein the structured data is stored in a structured portion of the first data structure, wherein the link is preserved in the first data structure, and wherein the storing of the unstructured data and the structured data enables access of the unstructured data and the structured data from the first data structure.

2. The method of claim 1, wherein the unstructured data is captured from an email server.

3. The method of claim 1, wherein the structured data is captured from a data warehouse.

4. The method of claim 1, further comprising copying the unstructured data and the structured data from the first data structure to a second data structure.

5. The method of claim 1, wherein the first data structure is stored in a remote database management system.

6. A computer readable storage medium having computer instructions executable on a computer for carrying out operations, comprising:

(a) providing a plurality of data processing modules, said data processing modules comprising;

an e-mail capture and parsing engine for intercepting, copying, and processing e-mails transmitted from a gateway to an email server, said processing comprising dividing a copy of an email into sections including at least a header section and a body section, and dividing the header section into sections comprising one or more of sender email address, sender name, recipient email address, recipient name, summary of email contents, date the email was sent, or time the email was sent;

an email reload engine for loading data associated with archived or stored emails;

a web crawler engine for Internet crawling and capturing Internet web pages;

a document gathering engine for capturing data from external sources comprising one or more of an application data warehouse, application server, or file system;

one or more data staging areas for temporary storage of data associated with said email capture and parsing engine, said web crawler engine, and said document gathering engine;

a text extraction and parsing engine configured for receiving data from said data staging areas, extracting structured data from unstructured data, and correlating extracted structured data and associated unstructured data to define a link between said structured data and said associated unstructured data;

a data holding area for temporary storage of said structured data and said unstructured data from said text extraction and parsing engine;

a data loading engine for loading and storing into a database management system said structured data and said unstructured data from said data holding area based on said link;

an email account management engine configured for bypassing said text extraction and parsing engine and directly copying structured data from said email server to said database management system; and

(b) using said plurality of data processing modules, carrying out operations comprising;

capturing unstructured data from an unstructured data source and a first structured data from a structured data source, a first portion of the unstructured data being associated with the first structured data, wherein the unstructured data source and the structured data source are each associated with at least an email, the email including a header;

parsing the header into at least a sending email address, a receiving email address, a date and time of transmission associated with the email, and a carbon copy email address;

evaluating the email using the email capture and parsing engine, wherein the email capture and parsing engine generates a summary of content associated with the email, the sending email address, the receiving email address, the date and the time of transmission associated with the email, the carbon copy email address, and a summary used to classify the email;

extracting a second structured data from the first portion of the unstructured data;

correlating the first and second structured data to establish a first link between the first and second structured data, wherein the first link integrates the first and second structured data; and

storing the first and second structured data in a structured portion of a data structure, wherein the first link is preserved in the data structure.

7. The computer readable storage medium of claim 6, further comprising computer instructions for:

correlating a second portion of the unstructured data and the second structured data to define a second link between the second portion of the unstructured data and the second structured data; and

storing the second portion of the unstructured data in an unstructured portion of the data structure, the second link being preserved in the data structure.

8. The computer readable storage medium of claim 6, further comprising storing the first and second structured data to enable access to the first and second structured data from the data structure.

9. The computer readable storage medium of claim 6, wherein extracting the second structured data from the first portion of the unstructured data further comprises translating character images from the first portion of the unstructured data into text stored in the second structured data.

10. The computer readable storage medium of claim 6, wherein extracting the second structured data from the first portion of the unstructured data further comprises interpreting digitized speech from the first portion of the unstructured data into text stored in the second structured data.

11. A computer implemented system for processing data associated with an e-mail, the system comprising:

(a) a data processing unit;

(b) programming, executable by said data processing unit, for providing a plurality of data processing modules, said data processing modules comprising;

an e-mail capture and parsing engine for intercepting, copying, and processing e-mails transmitted from a gateway to an email server, said processing comprising dividing a copy of an email into sections including at least a header section and a body section, and dividing the header section into sections comprising one or more of sender email address, sender name, recipient email address, recipient name, summary of email contents, date the email was sent, or time the email was sent;

an email reload engine for loading data associated with archived or stored emails;

a web crawler engine for Internet crawling and capturing Internet web pages;

a document gathering engine for capturing data from external sources comprising one or more of an application data warehouse, application server, or file system;

one or more data staging areas configured for temporary storage of data associated with said email capture and parsing engine, said web crawler engine, and said document gathering engine;

a text extraction and parsing engine for receiving data from the data staging areas, extracting structured data from unstructured data, and correlating extracted structured data and associated unstructured data and a to define a link between said structured data and said associated unstructured data;

a data holding area configured for temporary storage of said structured and said unstructured data from said text extraction and parsing engine;

a data loading engine for loading and storing into a database management system said structured data and said unstructured data from said data holding area based on said link;

an email account management engine for bypassing said text extraction and parsing engine and directly copying structured data from said email server to said database management system;

(c) wherein said plurality of data processing modules are configured for carrying out operations comprising;

receiving unstructured data and structured data from a plurality of sources comprising an unstructured data source and a structured data source, the unstructured data being associated with the structured data, wherein the unstructured data source and the structured data source are each associated with at least an email, the email including a parsing the header into at least a sending email address, a receiving email address, a date and time of transmission associated with the email, and a carbon copy email address;

evaluating the email using an email capture and parsing engine, wherein the email capture and parsing engine is configured to generate a summary of content associated with the email, the sending email address, the receiving email address, the date and the time of transmission associated with the email, the carbon copy email address, and a summary used to classify the email;

correlating the unstructured data and the structured data to establish a link between the unstructured data and the structured data, wherein the link integrates the unstructured data and the structured data; and

storing the unstructured data and the structured data in a data structure based on the link, wherein the unstructured data is stored in an unstructured portion of the data structure and wherein the structured data is stored in a structured portion of the data structure.

12. The system of claim 11, wherein the storing the unstructured data and the structured data further comprises enabling access of the unstructured data and the structured data from the data structure.

13. The system of claim 11, wherein the data structure stores the unstructured data and the structured data associated with an email.

14. The system of claim 13, wherein the data structure stores the structured data associated with an email header.

15. The system of claim 13, wherein the data structure stores the structured data associated with an email body.

16. The system of claim 13, wherein the data structure stores the unstructured data associated with an email attachment.

17. The system of claim 11, wherein the data structure is stored in a remote database management system.

18. The system of claim 11, wherein the data structure stores the unstructured data and the structured data associated with an electronic document.

19. The system of claim 11, wherein the data processing module further comprises computer instructions for translating a portion of the unstructured data into another language before storing the portion in the data structure.

20. The system of claim 11, wherein the structured portion of the data structure stores data in Extensible Markup Language format.