Systems, methods and computer program products for fast and scalable proximal search for search queries

• US 8,805,848 B2
• Filed: 05/24/2012
• Issued: 08/12/2014
• Est. Priority Date: 05/24/2012
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bhatia, Sumit, He, Bin, He, Qi, Spangler, William S.
• Primary Examiner(s): Corrielus, Jean M
• Application Number:
  US13/480,429
• Publication Number:
  US 20130318090A1
• Time in Patent Office: 810 Days
• Field of Search: 707/706, 707/710, 707/724, 707/728, 707/737, 707/740, 707/741, 707/751
• US Class Current: 707/741
• CPC Class Codes: G06F 16/31 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/33 Querying

Abstract

Embodiments of the invention provide a system, method and computer program products for information retrieval from multiple documents by proximity searching for search queries. A method includes generating an index for the multiple documents, wherein the index includes words in snippets in the documents. An input search query is processed against the index by searching query terms over the snippets to introduce term proximity information implicitly in the information retrieval. Results of multiple sentence level search operations are combined as output.

Claims

1. A computer program product for information retrieval from multiple documents, the computer program product comprising a tangible storage medium readable by a computer system and storing instructions for execution by the computer system for performing a method comprising:

splitting each document into multiple snippets of words;

indexing each snippet as a separate document;

receiving an input search query including at least one sentence;

processing the search query against the indexes of each of the multiple snippets by searching query terms over each of the multiple snippets to implicitly introduce term proximity information in the information retrieval;

decomposing the search query into sub-queries;

processing each sub-query against the indexes of each of the multiple snippets, sentence by sentence, using all words in each sentence of the sub-query to create an OR-Query of all non-stopwords in the sentence;

returning a fit value for each OR-Query; and

aggregating the fit values to provide a score for every document returned by the OR-Queries.

2. The computer program product of claim 1, wherein a snippet comprises multiple consecutive sentences of an original document.

3. The computer program product of claim 2, wherein processing the search query comprises:

processing the search query against the indexes of each of the multiple snippets, sentence by sentence, using all words in each sentence to create an OR-Query of all non-stopwords in the sentence.

4. The computer program product of claim 3, wherein processing the search query comprises:

returning a fit value for each OR-Query, wherein a fit value represents a similarity metric that measures the amount of word content overlap between two text units; and

aggregating the fit values to provide a score for every document returned by the OR-Queries.

5. The computer program product of claim 4, wherein the similarity is based on measuring similarity between a snippet in the documents and the original input text of the query.

6. The computer program product of claim 1, wherein the method comprises:

determining the frequency of each non-stopword in the documents; and

removing from the search those words having a frequency that exceeds a threshold.

7. The computer program product of claim 1, wherein the method comprises:

processing each sub-query against the indexes of each of the multiple snippets for searching sub-query terms over the multiple snippets, implicitly introducing term proximity information in the information retrieval.

8. An information retrieval system, comprising:

an indexer configured for splitting each document into multiple snippets of words and indexing each snippet as a separate document; and

a searching module configured for receiving an input search query and searching the query against the indexes of each of the multiple snippets by searching query terms over each of the multiple snippets to implicitly introduce term proximity information in the information retrieval;

wherein the searching module is further configured to;

search the query against the indexes of each of the multiple snippets, sentence by sentence, using all words in each sentence to create an OR-Query of all non-stopwords in the sentence, and return a fit value for each OR-Query, wherein a fit value represents a similarity metric that measures the amount of word content overlap between two text units; and

the system further comprises an aggregator configured to aggregate the fit values to provide a score for every document returned by the OR-Queries, and a query module configured to decompose the search query into sub-queries.

9. The system of claim 8, comprising:

a document decomposing module configured for splitting each document into the multiple snippets of words;

wherein a snippet comprises multiple consecutive sentences of an original document.

10. The system of claim 9, wherein:

the searching module processes each sub-query against the indexes of each of the multiple snippets for searching sub-query terms over the multiple snippets, implicitly introducing term proximity information in the information retrieval.