INFORMATION-RETRIEVAL-PERFORMANCE EVALUATING METHOD, INFORMATION-RETRIEVAL-PERFORMANCE EVALUATING APPARATUS, AND STORAGE MEDIUM CONTAINING INFORMATION-RETRIEVAL-PERFORMANCE-RETRIEVAL-EVALUATION PROCESSING PROGRAM

• US 6,615,200 B1
• Filed: 03/22/2000
• Issued: 09/02/2003
• Est. Priority Date: 03/23/1999
• Status: Active Grant

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Kayahara, Naoki
• Primary Examiner(s): AMSBURY, WAYNE P
• Application Number:
  US09/533,100
• Time in Patent Office: 1,259 Days
• Field of Search: 707/2
• US Class Current: 1/1
• CPC Class Codes: G06F 16/30 of unstructured textual data document management systems G06F16/93
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing

Abstract

The invention concerns quantitatively evaluating retrieval performance of retrieving systems that allow information retrieval when natural sentences are input as retrieval requests. Text and a title (heading) thereof are recognized as a pair of documents, and multiple documents individually having the texts and the headings are prepared as retrieval-target documents. In this, for example, each of the headings is used as contents of a retrieval request, and each of the texts is used as retrieval-target information. One of the headings is input to a retrieving device, and retrieving processing is performed, results of retrieval performed ranked for the headings according to the above, and retrieval-process evaluation is performed according to the ranks. The same processing is repeated for the following headings, and total retrieval processing is performed for the retrieving system according to the individual results of retrieval.

Claims

1. An information-retrieval-performance evaluating method for performing an evaluation of an information retrieving system that retrieves documents, the method comprising:

providing a retrieval request comprising either a heading or a text of a document to the information retrieving system, and wherein the other corresponding one of the text and the heading for the document is a retrieval-target information such that when the retrieval request comprises a heading of the document, the retrieval-target information comprises the corresponding text of the document and when the retrieval request comprises a text of the document, the retrieval-target information comprises the corresponding heading of the document;

retrieving one or more texts from a storage device in response to the retrieval request comprised of a heading or retrieving one or more headings from a storage device in response to the retrieval request comprised of a text; and

performing a retrieval-performance evaluation to evaluate an accuracy of the information retrieving system in retrieving the retrieval-target information according to a result of the retrieval performed in response to the retrieval request.

2. The information-retrieval-performance evaluating method of claim 1, further comprising:

determining whether the retrieval-target information paired with the retrieval request exists, the determination being part of the result of retrieval performed and evaluated, such that if the retrieval-target information paired with the retrieval request is determined to exist, the retrieval-performance evaluation is performed for the retrieving system according to a condition where the retrieval-target information paired with the retrieval request exists in the retrieval result.

3. The information-retrieval-performance evaluating method of claim 2, the performing the retrieval-result evaluation step including retrieving multiple items of information according to the retrieval request, giving ranks to individual results of the retrieval according to a degree of conformity to the retrieval request, and performing the retrieval-performance evaluation for the information retrieving system according to the ranks, the retrieval-result evaluation being performed according to the condition where the retrieval-target information being paired with the retrieval request exists in the retrieval result.

4. The information-retrieval-performance evaluating method of claim 3, wherein when multiple documents having a text and a heading are stored in the storage device, a retrieval request is provided for each document one by one in the providing step, wherein the results of retrieval for all of the individual retrieval requests are totaled, and wherein the retrieval-performance evaluation is based on the total.

5. The information-retrieval-performance evaluating method of claim 2, wherein when multiple documents having a text and a heading are stored in the storage device, a retrieval request is provided for each document one by one in the providing step, wherein the results of retrieval for all of the individual retrieval requests are totaled, and wherein the retrieval-performance evaluation is based on the total.

6. The information-retrieval-performance evaluating method of claim 1, wherein when multiple documents having a text and a heading are stored in the storage device, a retrieval request is provided for each document one by one in the providing step, wherein the results of retrieval for all of the individual retrieval requests are totaled, and wherein the retrieval-performance evaluation is based on the total.

7. An information-retrieval-performance evaluating apparatus for an information retrieval system that retrieves documents, comprising:

a storage device;

a retrieval-request inputting device that receives a retrieval request comprised of either the text or the heading of a document, wherein the other corresponding one of the text and the heading of the document is a retrieval-target information;

a retrieving device that retrieves one or more texts from the storage device in response to the retrieval request comprised of a heading or that retrieves one or more headings from the storage device in response to the retrieval request comprised of a text, said retrieved items being stored in the storage device; and

a retrieval-result evaluating device that performs a retrieval-performance evaluation to evaluate an accuracy of the information retrieving system in accurately retrieving the retrieval-target information corresponding to the retrieval request.

8. The information-retrieval-performance evaluating apparatus of claim 7, the retrieval-result evaluating device determining whether the retrieval-target information paired with the retrieval request exists, the determination being part of the result of retrieval performed and evaluated, such that if the retrieval-target information paired with the retrieval request is determined to exist, the retrieval-performance evaluation is performed for the retrieving system according to a condition where the retrieval-target information paired with the retrieval request exists in the retrieval result.

9. The information-retrieval-performance evaluating apparatus of claim 8, the retrieval-result evaluating device retrieving multiple items of information according to the retrieval request, giving ranks to individual results of the retrieval according to a degree of conformity to the retrieval request, and performing the retrieval-performance evaluation for the information retrieving system according to the ranks, the retrieval-result evaluation being performed according to the condition where the retrieval-target information being paired with the retrieval request exists in the retrieval result.

10. The information-retrieval-performance evaluating apparatus of claim 9, wherein the retrieving device retrieving multiple documents when multiple documents have the text and the title thereof stored in the storage device, wherein a retrieval request is received by the retrieval-request inputting device one by one for each of the multiple documents, wherein the results of retrieval for the individual retrieval requests are totaled, and wherein the retrieval-performance evaluation is based on the total.

11. The information-retrieval-performance evaluating apparatus of claim 8, wherein the retrieving device retrieving multiple documents when multiple documents have the text and the title thereof stored in the storage device, wherein a retrieval request is received by the retrieval-request inputting device one by one for each of the multiple documents, wherein the results of retrieval for the individual retrieval requests are totaled, and wherein the retrieval-performance evaluation is based on the total.

12. The information-retrieval-performance evaluating apparatus of claim 7, wherein the retrieving device retrieving multiple documents when multiple documents have the text and the title thereof stored in the storage device, wherein a retrieval request is received by the retrieval-request inputting device one by one for each of the multiple documents, wherein the results of retrieval for the individual retrieval requests are totaled, and wherein the retrieval-performance evaluation is based on the total.

13. The information-retrieval-performance evaluating apparatus of claim 7, wherein the storage device comprises two databases, one of the databases storing the headings of the documents and the other of the databases storing the texts of the documents.

14. A storage medium that stores programs for performing an evaluation of an information retrieving system that retrieves documents, the storage medium comprising:

a program for receiving a retrieval request comprising either a text or a heading of a document, wherein the other corresponding one of the text and the heading of the document is a retrieval-target information;

a program for retrieving one or more texts or one or more headings from a storage device in response to the retrieval request; and

a program for performing a retrieval-performance evaluation to evaluate an accuracy of the information retrieving system in accurately retrieving the retrieval-target information corresponding to the retrieval request.

15. The storage medium of claim 14, further comprising:

a program for determining whether the retrieval-target information paired with the retrieval request exists, the determination being part of the result of retrieval performed and evaluated, such that if the retrieval-target information paired with the retrieval request is determined to exist, the retrieval-performance evaluation is performed for the retrieving system according to a condition where the retrieval-target information paired with the retrieval request exists in the retrieval result.

16. The storage medium of claim 15, the program for performing the retrieval-result evaluation including retrieving multiple items of information according to the retrieval request, giving ranks to individual results of the retrieval according to a degree of conformity to the retrieval request, and performing the retrieval-performance evaluation for the information retrieving system according to the ranks, the retrieval-result evaluation being performed according to the condition where the retrieval-target information being paired with the retrieval request exists in the retrieval result.

17. The storage medium of claim 16, the program for performing a retrieval-performance evaluation including that, when multiple documents having the text and the title thereof are stored in the storage device, a retrieval request is received one by one for each of the multiple documents, the results of retrieval for the individual retrieval requests are totaled, and the retrieval-performance evaluation is based on the total.

18. The storage medium of claim 15, the program for performing a retrieval-performance evaluation including that, when multiple documents having the text and the title thereof are stored in the storage device, a retrieval request is received one by one for each of the multiple documents, the results of retrieval for the individual retrieval requests are totaled, and the retrieval-performance evaluation is based on the total.

19. The storage medium of claim 14, the program for performing a retrieval-performance evaluation including that, when multiple documents having the text and the title thereof are stored in the storage device, a retrieval request is received one by one for each of the multiple documents, the results of retrieval for the individual retrieval requests are totaled, and the retrieval-performance evaluation is based on the total.

20. The storage medium of claim 14, wherein the storage device comprises two databases, one of the databases storing the headings of the documents and the other of the databases storing the texts of the documents.