Dynamic comparison of search systems in a controlled environment

• US 7,483,877 B2
• Filed: 04/11/2003
• Issued: 01/27/2009
• Est. Priority Date: 04/11/2003
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kozakov, Lev, Kim, Moon Ju, Drissi, Youssef, Doganata, Yurdaer Nezihi
• Primary Examiner(s): Rones; Charles
• Assistant Examiner(s): Mahmood; Rezwanul
• Application Number:
  US10/412,808
• Publication Number:
  US 20040205051A1
• Time in Patent Office: 2,118 Days
• Field of Search: 707 3- 5, 707/104.1, 707/102
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3347 using vector based model
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A random document is stripped of the relevant search terms to generate a non-relevant document. The relevant search terms are formed into grammatically correct but not necessarily technically correct sentences. The grammatically correct sentences are placed at the beginning of the random document in one pass through the system in the middle of a document in a second pass through the system and at the end of the document in a third pass through the system. A relevancy vector chart is computed using the references documents and a known relevancy algorithm relating to position of search terms. The results obtained from search engines are compared to the relevancy vector chart to determine the relative relevancy of the returned search results from the search engines.

Claims

1. A software program on a computer usable storage medium for performing a computer controlled method of determining the effectiveness of a search engine in performing searches in a search area comprising:

computer code generating controlled relevant documents by manipulation of relevant search terms of documents in the search area based on a relevancy algorithm;

wherein the relevancy algorithm relates to positions of relevant search terms in documents and the generation of the controlled relevant documents is done by placing search terms into various positions in the document in accordance with the relevancy algorithm and wherein at least 3 different controlled relevant documents are produced for a given relevancy algorithm with search terms placed in different positions in different documents with each document compared with the document relevant to the search area with absence of search terms in that document to generate the relevancy vector chart;

using the controlled relevant documents to create a relevancy vector chart; and

comparing results obtained from the search engine to the relevancy vector chart to provide a relevancy rating for the search engine with computer code using the relevancy vector chart to obtain a relative ranking of a plurality of search engines.

2. The software program of claim 1, wherein the computer code for creating controlled relevant documents creates a plurality of different relevancy vector charts each based on a different relevancy algorithm.

3. The software program of claim 2, including computer code for using the relevancy vector charts to obtain a relative ranking of a plurality of search systems.

4. A method of determining the relative effectiveness of search engines comprising the steps of:

selecting documents from a database;

generating from the selected documents controlled relevant documents by manipulation of relevant terms in the selected documents based on a relevancy algorithm;

wherein the relevancy algorithm relates to positions of the relevant terms in documents and the generation of the controlled relevant documents is done by placing search terms into various positions in the controlled relevant documents in accordance with the relevant algorithm;

using the generated controlled relevant documents to create relevancy reference vectors;

searching the database using each of the search engines and generating performance vectors with the search results for each engine; and

comparing the performance vectors for each engine to the relevancy reference vectors to provide a relevancy rating for the search engines based on the relevancy algorithm using a relevancy vector chart made from the relevancy reference vectors to obtain a relative ranking of each of the search engines wherein at least 3 different controlled relevant documents are produced for each selected document using a given relevancy algorithm with search terms placed in 3 different positions in different documents with each document compared with the document relevant to the search area with an absence of search terms in the document to generate the relevancy reference vectors of the relevancy vector chart.

5. The method of claim 4, including removing relevant terms from the selected documents prior to placing the search terms into the various positions of the controlled relevant documents.

6. A method of determining the relative effectiveness of search engines in performing searches in a search area comprising:

generating a controlled relevant document by the arrangement of relevant search terms of a selected document in the search area in accordance with a relevancy algorithm;

wherein the relevancy algorithm relates to positions of relevant search terms in documents and the generation of the controlled relevant documents is done by placing search terms into various positions in the document in accordance with the relevancy algorithm and wherein at least 3 different controlled relevant documents are produced for a given relevancy algorithm with search terms placed in different positions in different documents with each document compared with the document relevant to the search area with absence of search terms in that document to generate the relevancy vector chart;

interrogating a database containing the search area with the search engines;

comparing by vector analysis, references obtained from the search engines with the controlled relevant document to provide a relevancy rating for the search engine to determine the relative effectiveness of the search engines;

generating vectors V_r for the control relevant document and V_SEj for search results of each search engine SE_j (J=1, 2, . . . n);

computing the distance d(V_SEj V_r) between V_SEj and V_r for each search engine;

computing a score Sj=100/1+d(V_SEj, V_r) for each search engine;

assigning the score Sj to each search engine SE_j; and

comparing the search engines SE₁, SE2, . . . SE_n based on their scores Si, S2, . . . Sn.

7. The method of claim 6, including removing all relevant terms from the selected document prior to placing the search terms into various positions of the controlled relevant document.

8. The method of claim 6, wherein the relevancy algorithm relates to positions of the relevant terms in documents and the controlled relevant document is created by placing search terms into various positions in the selected document in accordance with the relevant algorithm.

9. The method of claim 8, wherein at least 3 different controlled relevant documents are produced from the selected document using a given relevancy algorithm with search terms placed in 3 different positions in the three different controlled relevant documents with each controlled relevant document compared with the selected document with an absence of search terms in the selected document to generate three controlled relevant vectors.

10. The method of claim 9 including using a relevancy vector chart made up of the three controlled relevant vectors to obtain a relative ranking of a plurality of search engines.

11. The method of claim 6 including creating a plurality of controlled relevant documents for different relevancy vectors each controlled relevant document in the plurality based on a different relevancy algorithm.