Using network traffic logs for search enhancement

• US 8,203,952 B2
• Filed: 07/07/2008
• Issued: 06/19/2012
• Est. Priority Date: 04/16/2001
• Status: Expired due to Term

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Borkovsky, Arkady, Cook, Douglas M., Langlois, Jean-Marc, Poutanen, Tomi, Zha, Hongyuan
• Primary Examiner(s): Yao, Kwang B
• Assistant Examiner(s): LIU, JUNG
• Application Number:
  US12/168,797
• Publication Number:
  US 20080270484A1
• Time in Patent Office: 1,443 Days
• Field of Search: 707/5, 707/200, 707/2, 707/513, 707/219, 707/3, 370/395.7, 370/395.71, 370/395.72, 370/203, 370229-240, 370241-253, 370254-258, 370298-306, 370351-356, 370357-395, 370/395.1, 370/395.2, 370/395.21, 370/395.3, 370/395.31, 370/395.32, 370/395.4, 370/395.41, 370/395.42, 370/395.43, 370/395.5, 370/395.52, 370/395.53, 370/395.54, 370/395.6, 370/395.61, 370396-411, 370412-429, 370503-520
• US Class Current: 370/230.1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99937 Sorting

Abstract

A method and apparatus for using network traffic logs for search enhancement is disclosed. According to one embodiment, network usage is tracked by generating log files. These log files among other things indicate the frequency web pages are referenced and modified. These log files or information from these log files can then be used to improve document ranking, improve web crawling, determine tiers in a multi-tiered index, determine where to insert a document in a multi-tiered index, determine link weights, and update a search engine index.

Claims

1. A computer-implemented method comprising the steps of:

receiving, at a search engine operating on one or more computing devices from a traffic server that is not the search engine and is not operating on the one or more computing devices, network usage information that indicates a frequency with which users of a plurality of users requested, during a particular period of time, a particular document through the traffic server, which monitors frequencies with which documents in a plurality of documents are provided, through the traffic server, from a plurality of origin servers upon which the plurality of documents reside, to the plurality of users in response to a plurality of document-identifying requests from the plurality of users;

the search engine positioning the particular document within a queue at a queue position that is selected based on said network usage information;

the search engine performing a crawling or indexing operation based at least in part on the position of the particular document within the queue.

2. The method of claim 1, further comprising, after crawling said particular document, updating an index used by the search engine based on new content of said particular document.

3. The method of claim 1, wherein said network usage information further indicates a frequency that said particular document is modified during the particular period of time.

4. The method of claim 3, further comprising monitoring the frequency that the particular document is modified by:

determining if said particular document has been modified during the particular period of time by passing a document name and a last modified date to a particular origin server of the plurality of origin servers; and

receiving an indication whether said particular document has been modified during the particular period of time from said particular origin server.

5. The method of claim 3, further comprising determining if said particular document is modified during the particular period of time at least in part based on a last modified date associated with the HTTP protocol.

6. The method of claim 1, further comprising monitoring said frequency that said particular document is requested by the users of the plurality of users.

7. The method of claim 1, wherein the plurality of requests from the plurality of users include requests that are not requests for search results provided by the search engine.

8. The method of claim 1, wherein the search engine comprises an indexer that indexes documents and a spider that crawls documents.

9. The method of claim 1, wherein the traffic server is associated with a web cache and the web cache contains frequency information for said particular document and wherein the queue position represents the number of documents which will receive a crawling or indexing operation before the particular document.

10. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform the steps of:

receiving, at a search engine operating on one or more computing devices from a traffic server that is not the search engine and is not operating on the one or more computing devices, network usage information that indicates a frequency with which users of a plurality of users requested, during a particular period of time, a particular document through the traffic server, which monitors frequencies with which documents in a plurality of documents are provided, through the traffic server, from a plurality of origin servers upon which the plurality of documents reside, to the plurality of users in response to a plurality of document-identifying requests from the plurality of users;

positioning the particular document within a queue at a queue position that is selected based on said network usage information;

performing a crawling or indexing operation based at least in part on the position of the particular document within the queue.

11. The computer-readable medium of claim 10, wherein the one or more sequences of instructions, when executed by the one or more processors, further cause the one or more processors to perform, after crawling said particular document, updating an index used by the search engine based on new content of said particular document.

12. The computer-readable medium of claim 10, wherein said network usage information further indicates a frequency that said document is modified during the particular period of time.

13. The computer-readable medium of claim 12, wherein the one or more sequences of instructions, when executed by the one or more processors, further cause the one or more processors to perform monitoring said frequency that the particular document is modified by:

determining if said particular document has been modified during the particular period of time by passing a document name and a last modified date to a particular origin server of the plurality of origin servers; and

receiving an indication whether said particular document has been modified during the particular period of time from said particular origin server.

14. The computer-readable medium of claim 12, wherein the one or more sequences of instructions, when executed by the one or more processors, further cause the one or more processors to perform determining if said particular document is modified during the particular period of time at least in part based on a last modified date associated with the HTTP protocol.

15. The computer-readable medium of claim 10, wherein the one or more sequences of instructions, when executed by the one or more processors, further cause the one or more processors to perform monitoring said frequency that said particular document is requested by the users of the plurality of users.

16. The computer-readable medium of claim 10, wherein the plurality of requests from the plurality of users include requests that are not requests for search results provided by the search engine.

17. The computer-readable medium of claim 10, wherein the search engine comprises an indexer that indexes documents and a spider that crawls documents.

18. The computer-readable medium of claim 10, wherein the traffic server is associated with a web cache and the web cache contains frequency information for said particular document and wherein the queue position represents the number of documents which will receive a crawling or indexing operation before the particular document.