System and methodology providing improved information retrieval

• US 7,769,776 B2
• Filed: 06/16/2006
• Issued: 08/03/2010
• Est. Priority Date: 06/16/2006
• Status: Active Grant

• Current Assignee: Sybase Incorporated (SAP SE)
• Original Assignee: Sybase Incorporated (SAP SE)
• Inventors: Tabraham, John
• Primary Examiner(s): Wong; Don
• Assistant Examiner(s): Le; Michael
• Application Number:
  US11/309,082
• Publication Number:
  US 20070294220A1
• Time in Patent Office: 1,509 Days
• Field of Search: 707/3, 707/770
• US Class Current: 707/770
• CPC Class Codes: G06F 16/3331 Query processing

Abstract

System and methodology for performing Bayesian-based distributed query processing is provided that solves the problem of how to get each server participating in a Bayesian distributed search system to return the same accurate relevance score for different documents. By performing calculations in a two-step process, accurate Bayesian calculation results are obtained whilst distributing the document indexing and query processing.

Claims

1. An improved method which operates in a computer system comprising a central server and one or more remote servers and which performs distributed query processing, the method comprising:

receiving at the central server a search request specifying one or more terms for retrieving documents;

at each remote server where documents have been indexed, generating local Bayes calculation data for each term on that server;

at the central server, generating global Bayes calculation data by combining the local Bayes calculation data from each remote server, and allowing the remote servers to accurately score and rank documents as if they were all located on one server on a per query basis;

sending the global Bayes calculation data to each remote server as part of the distributed query processing;

at each remote server, producing a list of matching documents sorted by relevance ranking by performing a local Bayesian query calculation based at least in part on said global Bayes calculation data; and

at the central server, generating a final list of most relevant documents by merging the lists of matching relevant documents from each remote server, each document having an accurate score calculated from the local calculation data and global Bayesian statistics on a per query basis.

2. The method of claim 1, wherein said search request comprises a natural language text query entered by a user.

3. The method of claim 2, further comprising breaking down the natural language text into individual query terms.

4. The method of claim 1, wherein said step of generating local Bayes calculation data for each term includes:

making a light weight request from the central server to each remote server where documents have been indexed, to obtain local Bayes calculation data for each term on that remote server.

5. The method of claim 4, wherein said local Bayes calculation data includes, for a given term and a given remote server, how many documents on the given remote server contain the given term.

6. The method of claim 5, wherein said local Bayes calculation data also includes a total frequency of the given term in all documents on the given remote server.

7. The method of claim 5, wherein said local Bayes calculation data also includes how many documents are indexed on the given remote server.

8. The method of claim 5, wherein said local Bayes calculation data also indicates an average document length of all documents on the given remote server.

9. The method of claim 1, further comprising:

caching the generated global Bayes calculation data, so as to aid similar queries.

10. The method of claim 1, wherein the final list represents a final set sorted based on relevance to the search request.

11. The method of claim 1, further comprising:

storing on a computer-readable medium processor-executable instructions for performing the method.

12. An information retrieval system comprising:

a central hub and one or more remote document stores;

a module, operating at the central hub, for receiving a search request specifying one or more terms for retrieving documents;

a module, operating at remote document stores, for generating local Bayes calculation data for each term for documents indexed at that document store;

a module, operating at the central hub, for generating global Bayes calculation data by combining the local Bayes calculation data from each remote document store, and sending the global Bayes calculation data back to each remote document store, and allowing the remote document stores to accurately score and rank documents as if they were all located on one document store on a per query basis;

a module, operating at remote document stores, for producing a list of matching documents sorted by relevance ranking by performing a local Bayesian query calculation based at least in part on said global Bayes calculation data; and

a module, operating at the central hub, for generating a final list of most relevant documents by merging the lists of matching relevant documents from each remote document store, each document having an accurate score calculated from the local calculation data and global Bayesian statistics on a per query basis.

13. The system of claim 12, wherein said search request comprises a natural language text query entered by a user.

14. The system of claim 13, further comprising breaking down the natural language text into individual query terms.

15. The system of claim 12, wherein said module for generating local Bayes calculation data for each term comprises:

a module for making a light weight request from the central hub to each remote document store where documents have been indexed, to obtain local Bayes calculation data for each term on that remote document store.

16. The system of claim 15, wherein said local Bayes calculation data includes, for a given term and a given remote document store, how many documents on the given remote document store contain the given term.

17. The system of claim 16, wherein said local Bayes calculation data also includes a total frequency of the given term in all documents on the given remote document store.

18. The system of claim 16, wherein said local Bayes calculation data also includes how many documents are indexed on the given remote document store.

19. The system of claim 16, wherein said local Bayes calculation data also indicates an average document length of all documents on the given remote document store.

20. The system of claim 12, further comprising:

a module, operating at the central hub, for caching the generated global Bayes calculation data, so as to aid similar queries.

21. The system of claim 12, wherein the final list represents a final set sorted based on relevance to the search request.

22. An improved method for performing distributed Bayesian query processing, the method comprising:

receiving at a first computer a search request for retrieving documents pursuant to one or more search terms;

based on said search request, sending a lightweight request to other computers where documents have been indexed, for determining local Bayes calculation data for each term on each of the other computers;

generating at the first computer global Bayes calculation data by combining the local Bayes calculation data from each of the other computers;

sending the global Bayes calculation data to each of the other computers as part of the distributed Bayesian query processing;

at each of the other computers, producing a list of matching documents sorted by relevance ranking, by performing a local Bayesian query calculation that takes into account said global Bayes calculation data, and allowing the other computers to accurately score and rank documents as if they were all located on one server computer on a per query basis; and

generating at the first computer a final list of most relevant documents by merging the lists of matching relevant documents from each of the other computers, each document having an accurate score calculated from the local calculation data and global Bayesian statistics on a per query basis.

23. The method of claim 22, wherein said local Bayes calculation data includes, for a given term and a given one of the other computers, how many documents on the given one of the other computers contain the given term.

24. The method of claim 23, wherein said local Bayes calculation data also includes a total frequency of the given term in all documents on the given one of the other computers.

25. The method of claim 23, wherein said local Bayes calculation data also includes how many documents are indexed on the given one of the other computers.

26. The method of claim 23, wherein said local Bayes calculation data also indicates an average document length of all documents on the given one of the other computers.

27. The method of claim 22, further comprising:

caching the generated global Bayes calculation data, so as to aid similar queries.

28. The method of claim 22, wherein the final list represents a final set sorted based on relevance to the search request.

29. The method of claim 22, wherein the first computer is configured as a hub for sending data to the other computers efficiently.

30. The method of claim 22, further comprising:

storing on a computer-readable medium processor-executable instructions for performing the method.