Systems and methods to evaluate search qualities

• US 9,519,712 B2
• Filed: 01/06/2009
• Issued: 12/13/2016
• Est. Priority Date: 01/06/2009
• Status: Active Grant

• Current Assignee: AT&T, Inc., AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Jacobson, Guy Joseph, Kumar, Ritu, Goodall, Colin R., Halasz, Sylvia, Hsu, Wen-Ling, Mohs, Bradley Keith, Skudlark, Ann E.
• Primary Examiner(s): Gofman, Alex
• Assistant Examiner(s): Black, Linh
• Application Number:
  US12/349,491
• Publication Number:
  US 20100174736A1
• Time in Patent Office: 2,898 Days
• Field of Search: 707/714, 707/765, 707/607, 707722-723, 707706-708, 707770-771, 705 738- 739, 718/101
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Systems and methods are provided to evaluate a search engine using search results retrieved from a plurality of different search engines. For example, a controller can randomly distribute a set of pre-formulated queries to a plurality of computers to query the search engines at random times to obtain search results and combine the search results to evaluate the completeness of a query result provided by a particular search engine. The controller may simulate the random queries from users using a set of computers and may reduce the burden on the search engines by spreading out the query activities over a period of time, such as a month.

Claims

1. A method comprising:

scheduling, by a computer controller, submission of a batch of query requests to a first search engine and a plurality of second search engines, wherein the batch of query requests is submitted to the first search engine and the plurality of second search engines by a set of computers selected by the computer controller;

receiving, at the computer controller, first search results from the first search engine and second search results from the plurality of second search engines in response to submission of the batch of query requests to the first search engine and the plurality of second search engines;

aggregating, by the computer controller, the first search results received from the first search engine and the second search results received from the plurality of second search engines to form a set of aggregated unique search results, wherein aggregating the first search results received from the first search engine and the second search results received from the plurality of the second search engines to form the set of aggregated unique search results comprises eliminating an invalid result from the first search result received from the first search engine; and

comparing, via the computer controller, the first search results received from the first search engine to the set of aggregated unique search results to determine a performance of the first search engine, wherein the performance of the first search engine is based on a completeness of the first search results received from the first search engine in comparison with the set of aggregated unique search results.

2. The method of claim 1, wherein the set of aggregated unique search results includes at least a telephone number, and wherein the set of aggregated unique search results is identified based at least in part on the telephone number.

3. The method of claim 2, wherein the set of aggregated unique search results further includes one of an address and a name.

4. The method of claim 3, wherein the set of aggregated unique search results is further identified based at least in part on the one of the name and the address.

5. The method of claim 1, wherein eliminating the invalid result from the first search results received from the first search engine comprises, for each of the first search results received from the first search engine:

extracting respective data from a respective telephone number field of a search result of the first search results received from the first search engine, wherein the respective data comprises a telephone number;

determining whether the telephone number of the respective data is a valid telephone number; and

eliminating the search result if the telephone number of the respective data is not a valid telephone number.

6. The method of claim 5, wherein determining whether the telephone number of the respective data is a valid telephone number comprises determining whether the telephone number of the respective data has a proper exchange number.

7. The method of claim 5, wherein determining whether the telephone number of the respective data is a valid telephone number comprises determining a most recent date of the telephone number of the respective data being used for a call.

8. The method of claim 5, wherein determining whether the telephone number of the respective data is a valid telephone number comprises initiating a call routing signal to call the telephone number of the respective data to determine whether the telephone number of the respective data is in service.

9. The method of claim 5, further comprising determining a rate of invalid results for the first search engine based on identifying invalid telephone numbers in the first search results received from the first search engine.

10. The method of claim 1, further comprising:

receiving queries from end users at one of the plurality of second search engines; and

determining the batch of query requests based on the queries received from the end users.

11. The method of claim 1, wherein the batch of query requests relates to a predetermined geographic search criterion.

12. The method of claim 1, further comprising:

selecting, by the computer controller, the set of computers; and

providing, by the computer controller, the batch of query requests to the set of computers.

13. The method of claim 12, further comprising instructing, by the computer controller, the set of computers to submit a random number of the batch of query requests to the first search engine and the plurality of second search engines and to pause a random period of time between submitting the random number of the batch of query requests to the first search engine and the plurality of second search engines.

14. A computer-readable memory device storing instructions, the instructions, when executed by a computer controller, cause the computer controller to perform operations comprising:

scheduling submission of a batch of query requests to a first search engine and a plurality of second search engines, wherein the batch of query requests is submitted to the first search engine and the plurality of second search engines by a set of computers selected by the computer controller;

receiving first search results from the first search engine and second search results from the plurality of second search engines in response to submission of the batch of query requests to the first search engine and the plurality of second search engines;

aggregating the first search results received from the search engine and the second search results received from the plurality of the second search engines to form a set of aggregated unique search results, wherein aggregating the first search result received from the first search engine and the second search result received from the plurality of the second search engines to form the set if aggregated unique search results comprises eliminating an invalid result from the first search results received from the first search engine; and

comparing the first search results received from the first search engine to the set of aggregated unique search results to determine a performance of the first search engine, wherein the performance of the first search engine is based on a completeness of the first search results received from the first search engine in comparison with the set of aggregated unique search results.

15. The computer-readable memory device of claim 14, wherein the operations further comprise:

selecting the set of computers; and

providing the batch of query requests to the set of computers.

16. The computer-readable memory device of claim 15, wherein the operations further comprise instructing the set of computers to submit a random number of the batch of query requests to the first search engine and the plurality of second search engines and to pause a random period of time between submitting the random number of the batch of query requests to the first search engine and the plurality of second search engines.

17. A computer controller comprising:

a processor; and

a memory storing instructions that, when executed by the processor, cause the processor to perform operations comprising

scheduling submission of a batch of query requests to a first search engine and a plurality of second search engines, wherein the batch of query requests is submitted to the first search engine and the plurality of second search engines by a set of computers selected by the computer controller,

receiving first search results from the first search engine and second search results from the plurality of second search engines in response to submission of the batch of query requests to the first search engine and the plurality of second search engines,

aggregating the first search results received from the first search engine and the second search results received from the plurality of second search engines to form a set of aggregated unique search results, wherein aggregating the first search results received from the first search engine and the second search results received from the plurality of second search engines to form the set of aggregated unique search results comprises eliminating an invalid result from the first search results received from the first search engine, and

comparing the first search results received from the first search engine to the set of aggregated unique search results to determine a performance of the first search engine, wherein the performance of the first search engine is based on a completeness of the first search results received from the first search engine in comparison with the set of aggregated unique search results.

18. The computer controller of claim 17, wherein the operations further comprise:

selecting the set of computers; and

providing the batch of query requests to the set of computers.

19. The computer controller of claim 18, wherein the operations further comprise instructing the set of computers to submit a random number of the batch of query requests to the first search engine and the plurality of second search engines and to pause a random period of time between submitting the random number of the batch of query requests to the first search engine and the plurality of second search engines.