System for monitoring global online opinions via semantic extraction

• US 8,352,412 B2
• Filed: 02/27/2009
• Issued: 01/08/2013
• Est. Priority Date: 02/27/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Alba, Alfredo, Bhagwan, Varun, Grandison, Tyrone W. A., Gruhl, Daniel F., Pieper, Jan H.
• Primary Examiner(s): RIFKIN, BEN M
• Application Number:
  US12/394,646
• Publication Number:
  US 20100223226A1
• Time in Patent Office: 1,411 Days
• Field of Search: 706/55
• US Class Current: 706/55
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

A system for transforming domain specific unstructured data into structured data including an intake platform controlled by feed back from a control platform. The intake platform includes an intake acquisition module for acquiring data building baseline data related to a domain and problem of interest, an intake pre-processing module, an intake language module, an intake application descriptors module, and an intake adjudication module. The control platform includes a control data acquisition module, a control data consistency collator, a control auditor, a control event definition and policy repository, an error resolver, and an output that outputs results of the workflow into structured data enabled to be used in data analytics.

Claims

1. A system for transforming domain specific unstructured data into structured data, said system comprising:

a digital processing apparatus executing programs of machine-readable instructions;

an intake platform operating through said digital processing apparatus, said intake platform comprising;

an intake acquisition module acquiring structured data, semi-structured data, and associated metadata related to a domain and problem of interest, said intake acquisition module developing baseline data related to said domain and problem of interest from said structured data, semi-structured data, and associated metadata;

an intake pre-processing module receiving structured and unstructured content related to said domain and problem of interest;

an intake language module providing word equivalents to words within said structured and unstructured content according to said domain and problem of interest;

an intake application descriptors module providing definitions of key descriptors within said domain and problem of interest; and

an intake adjudication module processing said structured and unstructured content using said baseline data, said word equivalents, and said key descriptors to develop a workflow for classifying said structured and unstructured content for said domain and problem of interest; and

a control platform comprising;

a control data acquisition module identifying data acquisition and data analysis errors relating to said receiving of said structured and unstructured content and said classifying of said structured and unstructured content;

a control data consistency collator analyzing states of said data within said workflow to identify state errors;

a control auditor monitoring sources of said structured and unstructured content and monitoring said data within said workflow to identify source and processing errors;

a control event definition and policy repository maintaining policies for resolving said data acquisition and data analysis errors, said state errors, and said source and processing errors; and

an error resolver resolving said data acquisition and data analysis errors, said state errors, and said source and processing errors according to said policies; and

an output operating through said digital processing apparatus, said intake output outputting results of said workflow, said results of said workflow comprising said structured and unstructured content classified into structured data enabled to be used in data analytics.

2. The system of claim 1, wherein said intake acquisition module is directed to one of websites, forums, feeds and proprietary sources, which provide one of user opinions, behavior and preferences.

3. The system of claim 2, wherein one of said user opinions, behavior and preferences will be supplemented by one of user demographics, domain and application relevant structured and semi-structured repositories are used to provide said baseline data.

4. The system of claim 1, wherein said control data acquisition module provides reporting on one of standard data acquisition layer errors, protocol related error codes, data source response times, semantic extraction failure codes, parsing error codes, data processing error codes, data output integrity checks, data corruption error codes and data consistency.

5. The system of claim 1, wherein said control data consistency collator executes state assertion and associated source accessibility state assertion.

6. The system of claim 1, wherein said control data consistency collator provides one of standardized reporting of acquired data volumes and output data volumes and standardized output integrity checks and standardized acquisition integrity checks, provide a functional state monitoring per processing unit, provide data throughput per processing unit, provide data volume analysis, and provide analytics processing capacity high watermark.

7. The system of claim 1, wherein said control auditor further ensures one of data presence per source, wherein the data presence per source is minimum data volumes expected per source over a specified time increment, data presence per modality, such that there are data consistency checks on expected modalities on a per source basis, data flow volumes, including scalability of the data store and scalability of the cube producing subsystem, data store integrity, including the insertion data flow and throughput, utilization high watermark, capacity and functional state, and post processing data consistency checks.

8. The system of claim 1, wherein said error resolver further comprises a component feedback component that executes advice of said control auditor based on issues having no solutions.

9. A programmable non-transitory storage medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus to perform operations of transforming domain specific unstructured and broken language data into structured data for enabling custom data analytics, the operations comprising:

acquiring structured data, semi-structured data, and associated metadata related to a domain and problem of interest;

developing baseline data related to said domain and problem of interest from said structured data, semi-structured data, and associated metadata;

receiving structured and unstructured content related to said domain and problem of interest;

providing word equivalents to words within said structured and unstructured content according to a domain and problem of interest;

providing definitions of key descriptors within said domain and problem of interest;

processing said structured and unstructured content using said baseline data, said word equivalents, and said key descriptors to develop a workflow for classifying said structured and unstructured content for said domain and problem of interest;

identifying data acquisition and data analysis errors relating to said receiving of said structured and unstructured content and said classifying of said structured and unstructured content;

analyzing states of said data within said workflow to identify state errors;

monitoring sources of said structured and unstructured content and monitoring said data within said workflow to identify source and processing errors;

maintaining policies for resolving said data acquisition and data analysis errors, said state errors, and said source and processing errors;

resolving said data acquisition and data analysis errors, said state errors, and said source and processing errors according to said policies; and

outputting results of said workflow, said results of said workflow comprising said structured and unstructured content classified into structured data enabled to be used in data analytics.

10. The programmable non-transitory storage medium of claim 9, wherein said acquiring is further to one of websites, forums and feeds which provide one of user opinions, behavior and preferences.

11. The programmable non-transitory storage medium of claim 10, wherein one of said user opinions, behavior and preferences will be supplemented by one of user demographics, domain and application relevant structured and semi-structured repositories are used to provide said baseline data.

12. The programmable non-transitory storage medium of claim 9, wherein said identifying data acquisition and data analysis errors further provides reporting on one of standard data acquisition layer errors, protocol related error codes, data source response times, semantic extraction failure codes, parsing error codes, data processing error codes, data output integrity checks, data corruption error codes and data consistency.

13. The programmable non-transitory storage medium of claim 9, wherein said analyzing states of said data further executes state assertion and associated source accessibility state assertion.

14. The programmable non-transitory storage medium of claim 9, wherein said analyzing states of said data further provides one of standardized reporting of acquired data volumes and output data volumes and standardized output integrity checks and standardized acquisition integrity checks, provide a functional state monitoring per processing unit, provide data throughput per processing unit, provide data volume analysis, and provide analytics processing capacity high watermark.

15. The programmable non-transitory storage medium of claim 9, wherein said monitoring sources further ensures one of data presence per source, wherein the data presence per source is minimum data volumes expected per source over a specified time increment, data presence per modality, such that there are data consistency checks on expected modalities on a per source basis, data flow volumes, including scalability of the data store and scalability of the cube producing subsystem, data store integrity, such as the insertion data flow and throughput, utilization high watermark, capacity and functional state, and post processing data consistency checks.

16. The programmable non-transitory storage medium of claim 9, wherein said resolving said data acquisition and data analysis errors further executes advice of said control auditor based on issues having no solutions.

17. A programmable non-transitory storage medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus to perform operations of transforming domain specific unstructured and broken language data into structured data for enabling custom data analytics, the operations comprising:

acquiring structured data, semi-structured data, and associated metadata related to a domain and problem of interest;

developing baseline data related to said domain and problem of interest from said structured data, semi-structured data, and associated metadata;

receiving structured and unstructured content related to said domain and problem of interest;

providing word equivalents to words within said structured and unstructured content according to a domain and problem of interest;

providing definitions of key descriptors within said domain and problem of interest;

processing said structured and unstructured content using said baseline data, said word equivalents, and said key descriptors to develop a workflow for classifying said structured and unstructured content for said domain and problem of interest;

identifying data acquisition and data analysis errors relating to said receiving of said structured and unstructured content and said classifying of said structured and unstructured content;

analyzing states of said data within said workflow to identify state errors;

monitoring sources of said structured and unstructured content and monitoring said data within said workflow to identify source and processing errors;

maintaining policies for resolving said data acquisition and data analysis errors, said state errors, and said source and processing errors;

resolving said data acquisition and data analysis errors, said state errors, and said source and processing errors according to said policies; and

outputting results of said workflow, said results of said workflow comprising said structured and unstructured content classified into structured data enabled to be used in data analytics,

wherein said acquiring is further to one of websites, forums and feeds which provide one of user opinions, behavior and preferences, and said user opinions, behavior and preferences are be supplemented by one of user demographics, domain and application relevant structured and semi-structured repositories are used to provide said baseline data.