SOURCE EXPANSION FOR INFORMATION RETRIEVAL AND INFORMATION EXTRACTION

• US 20120078895A1
• Filed: 09/24/2010
• Published: 03/29/2012
• Est. Priority Date: 09/24/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Chu-Carroll, Jennifer, Fan, James J., Zadrozny, Wlodek W., Schlaefer, Nico M.
• Granted Patent: US 8,892,550 B2
• Time in Patent Office: Days
• US Class Current: 707/728
• CPC Class Codes: G06F 16/332 Query formulation
• CPC Class Codes: G06F 16/3325 Reformulation based on results of preceding query
• CPC Class Codes: G06F 16/903 Querying for retrieval from the web G06F16/953

Abstract

System, method and computer program product for 1) preparing queries for retrieving related content based on existing data content. For instance, titles of existing documents or entities extracted from documents can be used as queries. 2) Retrieving content from other repositories of unstructured, semi-structured, or structured data. For instance, web pages can be retrieved using existing search engines. 3) Extracting smaller units of text from the retrieved content. For instance, web pages can be split into coherent paragraphs of text. 4) Judging the quality of the smaller units of text and their relatedness to existing data. For instance, paragraphs can be scored using a statistical model based on lexico-syntactic features and topic models. 5) Synthesizing new sources from high-quality related text. For instance, paragraphs that score above a threshold can be concatenated into a new document.

Claims

1. A method for automatically expanding existing data content that is included in a corpus comprising:

automatically generating search queries to search for content related to existing data, the queries being generated based on existing data content;

automatically retrieving content from one or more data repositories;

automatically extracting units of text from the retrieved content;

automatically determining a relevance of the extracted units of text and their relatedness to the existing data; and

automatically selecting new sources of content and including them in the corpus based on the determined relevance.

2. The method as claimed in claim 1, wherein the existing data content includes one or more documents, said method further comprising:

generating from said one or more documents, a topic name or a topic descriptor corresponding to units extracted from said one or more documents, said generated search queries including one or more;

said topic name or words and phrases extracted from said topic descriptor.

3. The method as claimed in claim 2, further comprising:

generating said topic name and said topic descriptor using one or more dictionaries of known topics, patterns or statistical methods.

4. The method as claimed in claim 2, further comprising:

specifying a projected information need in the form of words, text strings, phrases or ranked lists of words or phrases not in the existing data content; and

,

said generated search queries optionally including words and phrases extracted from said projected information need.

5. The method as claimed in claim 2, wherein said retrieving content includes using search engines to run said search queries against the one or more external data repositories, said content retrieved including one or more text passages or documents.

6. The method as claimed in claim 5, wherein said extracting units of text comprises splitting the retrieved text passages or documents into smaller text units, said splitting using structural markup for demarcating text unit boundaries.

7. The method as claimed in claim 6, wherein said determining the relevance of the text units from said retrieved passages or documents includes scoring each said text unit using a statistical model based on lexico-syntactic feature, said lexico-syntactic feature includes one of:

a topicality feature, a search feature, or a surface feature.

8. The method as claimed in claim 7, wherein said scoring based on a topicality feature includes computing a likelihood ratio of a text unit estimated with topic and background language models, said topic model being estimated from a seed document, and said background language model being estimated from a sample of documents.

9. The method as claimed in claim 7, wherein said scoring based on a topicality feature includes computing a likelihood ratio of a text unit estimated with topic and background language models, said topic model being estimated from text units retrieved for a given topic, and said background language model being estimated from a sample of text units retrieved for different topics.

10. The method as claimed in claim 7, further comprising:

determining, from a text unit, one or more tokens, said scoring based on a topicality feature includes computing an average tf.idf (term frequency—

inverse document frequency) score of said tokens, a “

tf”

score being estimated from a seed document, and an “

idf”

score being estimated from a sample of documents.

11. The method as claimed in claim 7, further comprising, fitting a logistic regression (LR) model using said features to estimate a relevance score of each independent text unit.

12. The method as claimed in claim 7, wherein said determining the relevance of the text units from said retrieved passages or documents includes using graphical models and other statistical models, rule-based approaches and heuristics.

13. The method as claimed in claim 1, further comprising:

automatically extending content of said corpus to include content of text units having scores above a pre-determined threshold.

14. The method as claimed in claim 1, further comprising:

automatically extending content of said corpus to include content of text units using an automatic summarization method.

15. A system for automatically expanding existing data content that is included in a corpus comprising:

memory storage device;

a processor in communication with said memory storage device configured to;

automatically generate search queries to search for content related to existing data, the queries being generated based on existing data content;

automatically retrieve content from one or more data repositories;

automatically extract units of text from the retrieved content;

automatically determine a relevance of the extracted units of text and their relatedness to the existing data; and

automatically select new sources of content and include them in the corpus based on the determined relevance.

16. The system as claimed in claim 15, wherein the existing data content includes one or more documents, said processor further configured to:

generate from said one or more documents, a topic name or a topic descriptor corresponding to units extracted from said one or more documents, said generated search queries including one or more;

said topic name or words and phrases extracted from said topic descriptor.

17. The system as claimed in claim 16, wherein said topic name and said topic descriptor are generated using one or more dictionaries of known topics, patterns or statistical methods.

18. The system as claimed in claim 16, wherein said processor is further configured to:

receive a projected information need in the form of words, text strings, phrases or ranked lists of words or phrases not in the existing data content;

wherein said generated search queries optionally include words and phrases extracted from said projected information need.

19. The system as claimed in claim 15, wherein said processor is further configured to retrieve content by using search engines to run said search queries against the one or more external data repositories, said content retrieved including one or more text passages or documents.

20. The system as claimed in claim 19, wherein said processor is further configured to extract units of text by splitting the retrieved text passages or documents into smaller text units using structural markup for demarcating text unit boundaries.

21. The system as claimed in claim 20, wherein said processor is further configured to determine the relevance of the text units from said retrieved passages or documents by scoring each said text unit using a statistical model based on lexico-syntactic feature, said lexico-syntactic feature includes one of:

a topicality feature, a search feature, or a surface feature.

22. The system as claimed in claim 21, wherein said processor is further configured to compute a score based on a topicality feature by one of:

computing a likelihood ratio of a text unit estimated with topic and background language models, said topic model being estimated from a seed document, and said background language model being estimated from a sample of documents or, said topic model being estimated from text units retrieved for a given topic, and said background language model being estimated from a sample of text units retrieved for different topics.

23. The system as claimed in claim 21, wherein said processor is further configured to determine, from a text unit, one or more tokens, said scoring based on a topicality feature includes computing an average tf.idf (term frequency—

inverse document frequency) score of said tokens, a “

tf”

score being estimated from a seed document, and an “

idf”

score being estimated from a large sample of documents.

24. The system as claimed in claim 15, wherein said processor is configured to automatically generate a new document to include content of text units having score above a pre-determined threshold.

25. The system as claimed in claim 15, wherein said processor is further configured to:

automatically extend content of said corpus to include content of text units using an automatic summarization method.

26. A computer program device for automatically expanding existing data content that is included in a corpus, the computer program device comprising a storage medium readable by a processing circuit and storing instructions run by the processing circuit for performing a method, the method comprising:

automatically generating search queries to search for content related to existing data, the queries being generated based on existing data content;

automatically retrieving content from one or more data repositories;

automatically extracting units of text from the retrieved content;

automatically determining a relevance of the extracted units of text and their relatedness to the existing data; and

automatically selecting new sources of content and including them in the corpus based on the determined relevance.

27. The computer program device as claimed in claim 26, wherein the existing data content includes one or more documents, said method further comprising:

generating from said one or more documents, a topic name or a topic descriptor corresponding to units extracted from said one or more documents, said generated search queries including one or more;

said topic name or words and phrases extracted from said topic descriptor.

28. The computer program device as claimed in claim 27, wherein said topic name and said topic descriptor are generated using one or more dictionaries of known topics, patterns or statistical methods.

29. The computer program device as claimed in claim 26, wherein said determining the relevance of the text units from said retrieved passages or documents includes scoring each said text unit using a statistical model based on lexico-syntactic feature, said lexico-syntactic feature includes one of:

a topicality feature, a search feature, or a surface feature.