Search engine and method with improved relevancy, scope, and timeliness

• US 7,917,483 B2
• Filed: 04/26/2004
• Issued: 03/29/2011
• Est. Priority Date: 04/24/2003
• Status: Expired due to Fees

• Current Assignee: Affini, Inc.
• Original Assignee: Affini, Inc.
• Inventors: Chang, William I.
• Primary Examiner(s): Kim; Charles
• Assistant Examiner(s): Le; Jessica N
• Application Number:
  US10/832,757
• Publication Number:
  US 20050004943A1
• Time in Patent Office: 2,528 Days
• Field of Search: 707/104.1, 707 3- 10, 707/200, 707/705, 707/709, 706/45, 345/440, 715/515
• US Class Current: 707/705
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

A search engine and a method achieve timeliness of documents returned in a search result by a relevancy feedback mechanism driven by the frequency in which a URL is returned in recent searches. The relevancy feedback mechanism includes one or more random processes which determine whether or not a cached or indexed web page associated with a URL in the search result should be refreshed. In addition, the random processes also determine whether or not hyperlinks in the cached or indexed web page should be followed to access related web pages. Accesses of web pages resulting from the operations of the random processes are used to update any document index maintained by the search engine. Relevancy scoring functions implemented in look-up tables are also disclosed. A more accurate relevancy scoring function is achieved using a lexicon based on anchortexts of extracted hyperlinks of web documents.

Claims

1. A computer implemented method for adaptive feedback ensuring timeliness of a collection of web pages, the method comprising:

extracting a uniform resource locator (URL) from a search result; and

determining whether or not a web page corresponding to the URL is present in whole or in part in the collection, wherein;

when the web page is determined to be present in the collection, downloading and replacing the web page in the collection in accordance with a first probability, wherein the first probability depends on an age of the web page in the collection; and

when the web page is determined not to be present in the collection, downloading and including the web page in the collection, further includes extracting hyperlinks from the web page corresponding to the URL and downloading the web pages corresponding to the hyperlinks each with a second probability, wherein the second probability depends on the number of hyperlinks in the web page.

2. The method as in claim 1, wherein the web pages are collected in a cache.

3. The method as in claim 1, wherein the web pages are indexed in a document index.

4. The method as in claim 2, wherein the cache incorporates a replacement policy that favors retaining most recently accessed the web pages.

5. The method as in claim 2, wherein the cache is indexed using a hash signature of the URL.

6. The method as in claim 1, wherein the first probability depends on one of more of:

source parameters, a type of the URL, an index size and a workload of a web crawler.

7. The method as in claim 1, wherein the second probability further depends on an age of the URL.

8. The method as in claim 3, further comprising updating the document index using information obtained from accessing the web page corresponding to the URL.

9. An adaptive feedback system for ensuring timeliness of a collection of web pages, the system comprising:

a query processor that extracts a uniform resource locator (URL) from a search result;

a cache including a process for determining whether or not a web page corresponding to the URL is present in whole or in part in the collection;

a web crawler coupled to the processor wherein;

when the web page is determined to be present in the collection, the web crawler downloads and replaces the web page in the collection in accordance with a first probability, wherein the first probability depends on an age of the web page in the collection; and

when the web page is determined not to be present in the collection, the web crawler downloads and including the web page in the collection,

wherein the processor extracts hyperlinks from the web page corresponding to the URL and directs the web crawler to download the web pages corresponding to the hyperlinks each with a second probability,

wherein the second probability depends on the number of hyperlinks in the web page.

10. The system as in claim 9, wherein the web pages are collected in the cache.

11. The system as in claim 9, wherein the web pages are indexed in a document index.

12. The system as in claim 10, wherein the cache incorporates a replacement policy that favors retaining most recently accessed the web pages.

13. The system as in claim 10, wherein the cache is indexed using a hash signature of the URL.

14. The system as in claim 9, wherein the first probability depends on one of more of:

source parameters, a type of the URL, an index size and a workload of a web crawler.

15. The system as in claim 9, wherein the second probability further depends on an age of the URL.

16. The system as in claim 11, wherein the processor updates the document index using information obtained from accessing the web page corresponding to the URL.