Method and system for retrieving relevant documents from a database

• US 6,240,408 B1
• Filed: 02/10/2000
• Issued: 05/29/2001
• Est. Priority Date: 06/08/1998
• Status: Expired due to Fees

• Current Assignee: KCSL Incorporated
• Original Assignee: KCSL Incorporated
• Inventors: Kaufman, Ilia
• Primary Examiner(s): Black, Thomas G.
• Assistant Examiner(s): Mills, John G.
• Application Number:
  US09/502,186
• Time in Patent Office: 474 Days
• Field of Search: 707/1-206
• US Class Current: 707/730
• CPC Class Codes: G06F 16/30 of unstructured textual data document management systems G06F16/93
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

A method for processing a search query uses the results of a search performed on a high quality, controlled database to assess the relevance of documents retrieved from a search of an uncontrolled public database having documents of highly variable quality. The method includes the steps of parsing the search query and then searching the authoritative database to generate authoritative database results. The search query is also used to search the public database, thereby generating public database results. The quality or relevance of the public database results are then quantified on the basis of the authoritative database results, thereby generating a quality index. The results from both the authoritative and the public databases are then ranked on the basis of this quality index.

Claims

1. A method for ranking a plurality of documents on the basis of the similarity of each of the plurality of documents to a user-query, said method comprising the steps of

parsing the user-query, thereby generating a query-word and a distribution of the query-word in the user-query,

assessing an importance of the query-word on the basis of

the frequency with which the query-word occurs in an authoritative database having at-least-one-authoritative-document, the at-least-one-authoritative-document having at-least-one-authoritative-document-sentence, and

the distribution of the query-word in the user-query,

evaluating a similarity of the at-least-one-authoritative-document to the user-query on the basis of a distribution of the query-word in the at-least-one-authoritative-document,

evaluating a similarity of a public document from a public database to the user-query on the basis of a distribution of the query-word in the public document, the public document having at-least-one-public-document-sentence,

evaluating a similarity of the at-least-one-authoritative-document-sentence to the user-query on the basis of the frequency with which the query-word occurs in the at-least-one-authoritative-document-sentence,

evaluating a similarity of the at-least-one-public-document-sentence to the user-query on the basis of the frequency with which the query-word occurs in the at-least-one-public-document-sentence,

ranking the at-least-one-public document relative to the at-least-one-authoritative document on the basis of the similarity of the at-least-one-authoritative-document to the user-query, the similarity of the public document to the user-query, the similarity of the at-least-one-authoritative-document-sentence to the user-query, and the similarity of the at-least-one-public-document-sentence to the user-query.

2. The method of claim 1 further comprising the steps of

evaluating the similarity of the at-least-one-authoritative-document-sentence to the user-query on the basis of a location of the at-least-one-authoritative-document-sentence in the at-least-one-authoritative-document,

evaluating the similarity of the at-least-one-public-document-sentence to the user-query on the basis of a location of the at-least-one-public-document-sentence relative to the public document.

3. The method of claim 1 further comprising the step of generating a derivative query-word, the derivative query-word being selected on the basis of an association with the query-word.

4. The method of claim 3 further comprising the step of

assessing an importance of the derivative query-word on the basis of

the frequency with which the derivative query-word occurs in an authoritative database having at-least-one-authoritative-document, the at-least-one-authoritative-document having at-least-one-authoritative-document-sentence, and

the distribution of the derivative query-word in the user-query.

5. The method of claim 4 further comprising the step of diminishing the importance of the derivative query-word relative to the importance of the query-word.

6. The method of claim 1 wherein the step of assessing the importance of the query-word comprises the steps of

determining the overall concentration of the query-word in the authoritative database,

determining the concentration of the query-word in the user-query,

determining a query-conductance indicative of the importance of the query-word, the query-conductance varying with the concentration of the query-word in the user-query and varying inversely with the overall concentration of the query-word in the authoritative database.

7. The method of claim 1 wherein the step of evaluating the similarity of the at-least-one-authoritative document to the user-query comprises the steps of

determining the concentration of the query-word in the at-least-one-authoritative document,

assigning a document-conductance to the at-least-one-authoritative document, the document-conductance being selected to vary with the concentration of the query-word in the at-least-one-authoritative document.

8. The method of claim 1 wherein the step of evaluating the similarity of the public document to the user-query comprises the steps of

determining the concentration of the query-word in the public document,

assigning a document-conductance to the public document, the document-conductance being selected to vary with the concentration of the query-word in the public document.

9. The method of claim 1 wherein the step of evaluating the similarity of the at-least-one-authoritative-document-sentence to the user-query comprises the steps of

determining the overall concentration of the query-word in the authoritative database,

determining the concentration of the query-word in the at-least-one-authoritative-document-sentence

assigning a position-independent sentence similarity to the at-least-one-authoritative-document-sentence, the position-independent sentence similarity being responsive to the ratio of

the overall concentration of the query-word in the authoritative database to the concentration of the query-word in the at-least-one-authoritative-document-sentence.

10. The method of claim 1 wherein the step of evaluating the similarity of the at-least-one-public-document-sentence to the user-query comprises the steps of

determining the overall concentration of the query-word in the authoritative database,

determining the concentration of the query-word in the at-least-one-public-document-sentence

assigning a position-independent sentence similarity to the at-least-one-public-document-sentence, the position-independent sentence similarity being responsive to the ratio of

the overall concentration of the query-word in the public database to the concentration of the query-word in the at-least-one-public-document-sentence.

11. A computer-readable medium containing software for ranking a plurality of documents on the basis of the similarity of each of the plurality of documents to a user-query, the software comprising instructions for executing the steps of

parsing the user-query, thereby generating a query-word and a distribution of the query-word in the user-query,

assessing an importance of the query-word on the basis of

the frequency with which the query-word occurs in an authoritative database having at-least-one-authoritative-document, the at-least-one-authoritative-document having at-least-one-authoritative-document-sentence, and

the distribution of the query-word in the user-query,

evaluating the similarity of the at-least-one-authoritative-document to the user-query on the basis of a distribution of the query-word in the at-least-one-authoritative-document,

evaluating the similarity of a public document from a public database to the user-query on the basis of a distribution of the query-word in the public document, the public document having at-least-one-public-document-sentence,

evaluating the similarity of the at-least-one-authoritative-document-sentence to the user-query on the basis of the frequency with which the query-word occurs in the at-least-one-authoritative-document-sentence,

evaluating the similarity of the at-least-one-public-document-sentence to the user-query on the basis of the frequency with which the query-word occurs in the at-least-one-public-document-sentence,

ranking the at-least-one-public document relative to the at-least-one-authoritative document on the basis of the similarity of the at-least-one-authoritative-document to the user-query, the similarity of the public document to the user-query, the similarity of the at-least-one-authoritative-document-sentence to the user-query, and the similarity of the at-least-one-public-document-sentence to the user-query.

12. The computer-readable medium of claim 11 wherein the software further comprises instructions for executing the steps of

evaluating the similarity of the at-least-one-authoritative-document-sentence to the user-query on the basis of a location of the at-least-one-authoritative-document-sentence in the at-least-one-authoritative-document,

evaluating the similarity of the at-least-one-public-document-sentence to the user-query on the basis of a location of the at-least-one-public-document-sentence relative to the public document.

13. The computer-readable medium of claim 11 wherein the software further comprises instructions for executing the step of generating a derivative query-word, the derivative query-word being selected on the basis of an association with the query-word.

14. The computer-readable medium of claim 13 wherein the software further comprises instructions for executing the steps of

assessing an importance of the derivative query-word on the basis of

the frequency with which the derivative query-word occurs in an authoritative database having at-least-one-authoritative-document, the at-least-one-authoritative-document having at-least-one-authoritative-document-sentence, and

the distribution of the derivative query-word in the user-query.

15. The computer-readable medium of claim 14 wherein the software further comprises instructions for executing the step of diminishing the importance of the derivative query-word relative to the importance of the query-word.

16. The computer-readable medium of claim 11 wherein the instructions for executing the steps of assessing the importance of the query-word further comprise instructions for executing the steps of

determining the overall concentration of the query-word in the authoritative database,

determining the concentration of the query-word in the user-query,

determining a query-conductance indicative of the importance of the query-word, the query-conductance varying with the concentration of the query-word in the user-query and varying inversely with the overall concentration of the query-word in the authoritative database.

17. The computer-readable medium of claim 11 wherein the instructions for executing the step of evaluating the similarity of the at-least-one-authoritative document to the user-query comprises instructions for executing the steps of

determining the concentration of the query-word in the at-least-one-authoritative document,

assigning a document-conductance to the at-least-one-authoritative document, the document-conductance being selected to vary with the concentration of the query-word in the at-least-one-authoritative document.

18. The computer-readable medium of claim 11 wherein the instructions for executing the step of evaluating the similarity of the public document to the user-query comprise instructions for executing the steps of

determining the concentration of the query-word in the public document,

assigning a document-conductance to the public document, the document-conductance being selected to vary with the concentration of the query-word in the public document.

19. The computer-readable medium of claim 11 wherein the instructions for evaluating the similarity of the at-least-one-authoritative-document-sentence to the user-query comprise instructions for executing the steps of

determining the overall concentration of the query-word in the authoritative database,

determining the concentration of the query-word in the at-least-one-authoritative-document-sentence

assigning a position-independent sentence similarity to the at-least-one-authoritative-document-sentence, the position-independent sentence similarity being responsive to the ratio of

the overall concentration of the query-word in the authoritative database to the concentration of the query-word in the at-least-one-authoritative-document-sentence.

20. The computer-readable medium of claim 11 wherein the instructions for executing the step of evaluating the similarity of the at-least-one-public-document-sentence to the user-query comprise instructions for executing the steps of

determining the overall concentration of the query-word in the authoritative database,

determining the concentration of the query-word in the at-least-one-public-document-sentence assigning a position-independent sentence similarity to the at-least-one-public-document-sentence, the position-independent sentence similarity being responsive to the ratio of

the overall concentration of the query-word in the public database to the concentration of the query-word in the at-least-one-public-document-sentence.