Dynamic comparison of search systems in a controlled environment

• US 20040205051A1
• Filed: 04/11/2003
• Published: 10/14/2004
• Est. Priority Date: 04/11/2003
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kozakov, Lev, Kim, Moon Ju, Drissi, Youssef, Doganata, Yurdaer Nezihi
• Granted Patent: US 7,483,877 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/3347 using vector based model
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A reference document is stripped of relevant search terms and then repopulated with the search terms in accordance with at least one known relevancy algorithm. An example of such a relevancy algorithm involves the position of search terms in a document. When search terms appear early in the document, the document is more relevant than when the search terms appear later in the document. To obtain a relevancy vector chart, a random document is stripped of the relevant terms to generate a non-relevant document. The relevant terms are then formed into sentences by an insertion controller which arranges them in grammatically correct but not necessarily technically correct sentences. The created sentences then are placed back into the document in a selected portion of the document. For instance, correct sentences could be placed at the beginning of the document in one pass through the system. In the middle of a document the second pass through the system and the third time be placed at the end of the document. Each time a different relevancy vector is computed for the reference document by using the known relevancy algorithm to create a relevancy vector chart. Sample search terms are then placed through various search engines, the results obtained from each search engine are compared to the relevancy vector chart to determine the relevancy of the returned search terms based on the relevancy vector chart. The higher the score the more relevant the documents obtained by a particular search engine. Thus the search engines can be rated based on the selected relevancy algorithm or algorithms.

Claims

1. A method of determining the relative effectiveness a search engine comprising the steps of:

selecting a relevant document;

converting the relevant document to controlled relevant documents by manipulation of search terms based on a relevancy algorithm to create a relevancy vector chart; and

comparing the results obtained from the search system to the relevancy vector chart to provide a relevancy rating for the search system.

2. The method of claim 1 including using the relevancy vector chart in accordance with claim 1 to obtain the relative ranking of a plurality of search systems.

3. The method of claim 1, wherein the relevancy algorithm relates to the position of search terms in documents and the conversion of the relevant document to controlled relevant documents is by placing search terms into various positions in the document in accordance with the placement algorithm.

4. The method of claim 3, wherein at least 3 different controlled relevant documents are produced for a given relevancy algorithm with search terms placed in 3 different positions in different documents with each document compared with the document relevant to the search area with an absence of search terms in the document to generate the relevancy vector chart.

5. The method of claim 4, wherein the search terms in the controlled relevancy documents are used sentences that are not necessarily technically meaningful.

9. The software program of claim 3, wherein at least 3 different controlled relevant documents are produced for a given relevancy algorithm with search terms placed in different positions in different documents with each document compared with the document relevant to the search area with absence of search terms in that document to generate the relevancy vector chart.

6. A software program on a computer usable medium for performing a computer controlled method of determining the effectiveness of a search system in performing searches in a search area comprising:

computer code for converting documents relevant to the search area to controlled relevant documents by manipulation of search terms based on a relevancy algorithm to create a relevancy vector chart; and

comparing the results obtained from the search system to the relevancy vector chart to provide a relevancy rating for the search system.

7. The software program of claim 6 including computer code for using the relevancy vector chart in accordance with claim 6 to obtain the relative ranking of a plurality of search systems.

8. The software program of claim 6, wherein the relevancy algorithm relates to the position of search terms in documents and the conversion of the relevant document to controlled relevant documents operated by placing search terms into various positions in the document in accordance with the placement algorithm.

10. The software program of claim 6, wherein the search terms in the controlled relevancy documents are used sentences that are not necessarily technically meaningful.

11. The software program of claim 6, wherein the computer code for creating controlled relevant documents created a plurality of different relevancy vector charts each based on a different relevancy algorithm.

12. The software program of claim 11 including computer code for using the relevancy vector charts of claim 11 to obtain a relative ranking of the plurality of search systems.