Methods for information extraction, search, and structured representation of text data

• US 10,146,751 B1
• Filed: 12/07/2015
• Issued: 12/04/2018
• Est. Priority Date: 12/31/2014
• Status: Expired due to Fees

• Current Assignee: Linfo IP LLC (Pueblo Nuevo LLC)
• Original Assignee: Guangsheng Zhang
• Inventors: Zhang, Guangsheng
• Primary Examiner(s): GUERRA-ERAZO, EDGAR X
• Application Number:
  US14/961,882
• Time in Patent Office: 1,093 Days
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 40/117 Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates

Abstract

System and methods for creating structured or semi-structured representations of information extracted from unstructured text data sources are described. In some embodiments, without requiring a predefined target data structure, the methods identify the grammatical and semantic attributes and context information in a text content, and create object-properties association data as knowledge and information extracted from the unstructured data, and represent such information in a structured or semi-structured format to facilitate search and trend analysis. In some other embodiments, the methods identify the types of information contained in the unstructured data, and for a pre-defined target information type, the methods identify the context and content of the portion of the text that represents the target information type, and extract the text, attach a tag or label to the extracted text, and store or display the data in a database table format or xml format for further pattern and trend analysis. Applications of the present system and methods include effectively analyzing user-generated contents such as customer feedback, reviews, comments, technical support forum messages, resume or job description documents, and other types of text contents.

Claims

1. A computer-implemented method for detecting hidden information in an unstructured data source, and generating a new data object representing the detected information, comprising:

defining a first attribute for identifying a target term in a text content, wherein the first attribute comprises a semantic or syntactic attribute, wherein the semantic attribute includes being or representing a name of an object or a property associated with an object;

defining a target term as a term that is associated with the first attribute;

identifying a second attribute as an additional attribute associated with the target term, wherein the second attribute comprises a semantic or syntactic attribute;

receiving a text content comprising a plurality of terms each comprising a word or a phrase;

identifying a first term in the text content;

identifying a second term in the context of the first term, wherein the context includes a term having a syntactic or semantic relation with the first term;

identifying whether the second term refers to or represents the second attribute;

if the second term is identified as referring to or representing the second attribute, then determining that the first term is the target term;

generating a first data object comprising a first label associated with the first term, wherein the first label is not an element extracted from the text content, wherein the first attribute and the second attribute are not explicitly indicated in the text content;

associating the first term to a search engine or a database; and

when the first term is associated to a search engine, using the first term as an automatically-generated query to perform a search and determine a relevance for a search result, and displaying the search result in a user interface, wherein the first term is displayed in the user interface for an enhanced user interface functionality of highlighting a key information item that is automatically discovered from an unstructured data source and used for determining the relevance of the search result, wherein the search engine comprises an enterprise search engine, a job-seeking search engine, a recruitment search engine, a text analytics tool, and a data mining search engine;

when the first term is associated to a database, compiling a data source comprising the first term, wherein the data source indicates that the first term is associated with the first label representing the first attribute that is hidden in the raw unstructured data source.

2. The computer-implemented method of claim 1, wherein the first term is determined to be the target term, the method further comprising:

extracting the first term, or storing the first term in a data field associated with the first attribute, or displaying the first term with a tag or label representing the first attribute.

3. The computer-implemented method of claim 1, wherein the second term is identified as referring to or representing the second attribute, the method further comprising:

determining, based on the second attribute, a likelihood value for determining whether the first term is the target term, and selecting the first term as the target term if the likelihood value is above a threshold.

4. The computer-implemented method of claim 1, wherein the second term is identified as referring to or representing the second attribute, the method further comprising:

assigning a term weight value to the first term based on the second attribute, wherein the function of the term weight value includes at least being a term importance value for document summarization, tagging and search result ranking.

5. The computer-implemented method of claim 4, further comprising:

determining a weighting co-efficient value for the second attribute, wherein the term weight value assigned to the first term is determined based on the weighting co-efficient value.

6. The computer-implemented method of claim 1, wherein the text content comprises an article, a comment, an email or message containing user feedback or discussion related to a product or service, wherein the first attribute refers at least to a problem being reported, or a question being asked.

7. The computer-implemented method of claim 1, wherein the text content comprises a resume or job description document, wherein the first attribute refers to a name of an object, wherein the object comprises at least a person, a skill, a position, a company, a time period, an education degree, a publication, or other objects of pertinent types.

8. The computer-implemented method of claim 1, wherein the syntactic relation between the first term and the second term includes at least a head-modifier relation represented by a modifier adjective or noun phrase with a noun phrase, or a modifier phrase and a head phrase connected by a preposition.

9. The computer-implemented method of claim 1, wherein the syntactic relation between the first term and the second term includes at least a head-modifier relation represented by a subject-predicate relation, or a verb-object relation.

10. A computer-implemented method for detecting hidden information and generating a structured data object from an unstructured data source, comprising:

defining a first information type for extracting a text unit from a text content, and creating a first label as a description of the first information type;

defining a second information type as a sub-type of the first information type, and creating a second label as a description of the second information type, wherein the second information type represents a subclass of instances that are associated with the first information type;

receiving a text content as an unstructured data source comprising a plurality of terms, each comprising a word or a phrase;

identifying a text unit comprising a first term in the text content;

identifying a first attribute associated with the first term;

determining whether the text unit is associated with the first information type based on the first attribute;

identifying a second attribute associated with a term in the text unit;

determining whether the text unit is associated with the second information type based on the second attribute;

if the text unit is determined to be associated with the first information type, generating a first data object comprising a first data field;

associating the first data field with the first label, wherein the first label is not an element extracted from the text unit, wherein the first information type is not explicitly indicated in the text unit; and

creating a structured data presentation format as a transformation from an unstructured data source, wherein the structured data presentation format comprises the first data field annotated by the first label;

associating the structured data presentation format to a text data analysis tool, wherein the text data analysis tool comprises an element selected from the group consisting of at least an enterprise call center data analysis tool, a customer support text data analysis tool, a text mining tool;

using the text data analysis tool to conduct a structured query, based on the structured data presentation format, for information originally hidden in an unstructured data source; and

displaying, in a user interface associated with the text data analysis tool, the structured data presentation format as an enhanced information presentation functionality of the user interface for information originally hidden in an unstructured data source.

11. The computer-implemented method of claim 10, wherein the text unit is determined to be associated with the second information type, the method further comprising:

storing the second label in a second data field associated with the first data field, or displaying the second label in association with the text unit in the first data field.

12. The computer-implemented method of claim 10, further comprising:

determining, based on the first attribute, a likelihood value for determining whether the text unit is associated with the first information type;

or

determining, based on the second attribute, a likelihood value for determining whether the text unit is associated with the second information type.

13. The computer-implemented method of claim 10, wherein the text content comprises an article, a comment, an email or message containing user feedback or discussion related to a product or service, wherein the first information type refers to a problem being reported or a request for information, and the second information type refers to a sub-type of the problem or a sub-type of the request for information, wherein the sub-type of the problem or the sub-type of the request for information is related to at least an operating procedure, a product feature or function, or a price.

14. The method of claim 13, wherein text unit representing the request for information includes a question sentence.

15. The method of claim 13, wherein the text unit representing the request for information is not marked by a question mark in a language in which a question mark is normally present in a question sentence, or is not marked by a question word in a language in which a question word is normally present in a question sentence.

16. The method of claim 13, wherein the text unit representing the request for information is in the format of a statement sentence, or of an embedded question.

17. The computer-implemented method of claim 10, wherein steps for identifying the first attribute as being associated with the first term comprise:

obtaining a first dataset comprising a set of terms, each term is associated with the first attribute;

matching the first term with a term in the first dataset;

if a match is found, determining that the first term is associated with the first attribute.

18. The computer-implemented method of claim 10, wherein the steps for identifying the second attribute as being associated with a term in the text unit comprise:

obtaining a second dataset comprising a set of terms, each term is associated with the second attribute;

matching the term in the text unit with a term in the second dataset;

if a match is found, determining that the text unit is associated with the second attribute.

19. The computer-implemented method of claim 10, wherein the steps for identifying the first attribute as being associated with the first term comprise:

identifying a contextual relation between the first term and a term in the context of the first term, wherein the contextual relation includes a syntactic or semantic relation including a head-modifier relation or a subject-predicate relation, or a verb-object relation, wherein the first attribute is identified as being associated with the first term based on the contextual relation.