Network traffic monitoring for search popularity analysis

• US 20050198268A1
• Filed: 02/10/2004
• Published: 09/08/2005
• Est. Priority Date: 02/10/2004
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: Narus, Inc. (Gen Digital Inc.)
• Inventors: Chandra, Rohit
• Granted Patent: US 7,594,011 B2
• Time in Patent Office: Days
• US Class Current: 709/224
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: H04L 43/00 Arrangements for monitoring or testing data switching networks

Abstract

The popularity of web pages is monitored and used to rank the web pages retrieved in response to an Internet search. The popularity of a web page is proportionate to the number of visits to that web page. Web pages with greater popularity are ranked higher in priority. Furthermore, the score of a first web page is propagated to a plurality of second web pages to which the first web page is linked substantially in relative proportion to the popularity of the links from the first web page to each of the second web pages. Monitoring devices monitor TCP packets traversing the Internet and extract information from the TCP packets, such as the requested URI or URL, the client IP address, a server IP address and a server host name, and a referrer URI, if any. The extracted information is forwarded to a processing module that keeps track of the popularity of the web pages.

Claims

1. A system for determining popularity of web pages on a network, the system comprising:

a plurality of monitoring devices placed in the network, the monitoring devices monitoring packets traversing the network and extracting information on the packets; and

a processing module coupled to the monitoring devices and receiving the extracted information from the monitoring devices, the processing module analyzing the extracted information and determining the popularity of the web pages based upon the extracted information, the popularity of the web pages being substantially proportionate to number of visits to the web pages as indicated by the extracted information.

2. The system of claim 1, wherein the monitoring devices are placed in locations where aggregate packet traffic may be monitored.

3. The system of claim 1, wherein the monitoring devices are placed at a traversal point for complete activity between a client device and a server on the network.

4. The system of claim 1, wherein the monitoring devices extract the information from packets in a TCP session, and the extracted information includes:

a requested URI or URL;

a client IP address; and

a server IP address and a server host name.

5. The system of claim 4, wherein the extracted information further includes a referrer URL.

6. The system of claim 4, wherein the monitoring devices analyzes the packets relating to GET Requests in the TCP session to extract the information.

7. The system of claim 1, wherein the monitoring devices discard packets relating to invalid URLs, invalid GET Requests, requests from a web crawler, or auto-refreshment of previous TCP sessions in extracting the information.

8. The system of claim 1, wherein the processing module maintains a counter corresponding to a URL and increments a count of the counter, if the extracted information indicates that the web page corresponding to the URL was visited, the count indicating the number of visits to the web page.

9. The system of claim 1, wherein the processing module maintains a plurality of counters corresponding to a URL and increments a count of one of the counters, if the extracted information indicates that the web page corresponding to the URL was visited by a client device located in a geographical location corresponding to the counter of which the count was incremented, the count indicating the number of visits to the web page by client devices in the corresponding geographical location.

10. The system of claim 9, wherein the processing module increments the count only if the extracted information indicates that the web page was visited by the client device having a distinct IP address.

11. The system of claim 9, wherein the processing module does not increment the count if the extracted information indicates that the packets were automatically and repeatedly generated by a computer.

12. A search system for ranking Internet search results based upon popularity of web pages on a network, the search system comprising:

a plurality of monitoring devices placed in the network, the monitoring devices monitoring packets traversing the network and extracting information on the packets;

a processing module coupled to the monitoring devices and receiving the extracted information from the monitoring devices, the processing module analyzing the extracted information and determining the popularity of the web pages based upon the extracted information, the popularity of the web pages being substantially proportionate to number of visits to the web pages as indicated by the extracted information; and

a search engine for receiving search terms and retrieving web pages containing the search terms, the search engine ranking the web pages at least in part based upon the popularity of the retrieved web pages.

13. The search system of claim 12, wherein the search engine ranks the retrieved web pages based upon the content of the web pages and the hyperlink structure linking the web pages as well as the popularity of the retrieved web pages.

14. The search system of claim 12, wherein the search engine propagates a score of a first web page to a plurality of second web pages to which the first web page is linked substantially in relative proportion to the popularity of links from the first web page to each of the second web pages.

15. The search system of claim 12, wherein the search engine ranks a first retrieved web page in higher priority than a second retrieved web page if the popularity of the first web page is greater than the popularity of the second web page.

16. The search system of claim 12, wherein the monitoring devices are placed in locations where aggregate packet traffic may be monitored.

17. The search system of claim 12, wherein the monitoring devices are placed at a traversal point for complete bidirectional activity between a client device and a server on the network.

18. The search system of claim 12, wherein the monitoring devices extract the information from packets in a TCP session, and the extracted information includes:

a requested URI or URL;

a client IP address; and

a server IP address and a server host name.

19. The search system of claim 18, wherein the extracted information further includes a referrer URL.

20. The search system of claim 18, wherein the monitoring devices analyzes the packets relating to GET Requests in the TCP session to extract the information.

21. The search system of claim 12, wherein the monitoring devices discard packets relating to invalid URLs, invalid GET Requests, requests from a web crawler, or auto-refreshment of previous TCP sessions in extracting the information.

22. The search system of claim 12, wherein the processing module maintains a counter corresponding to a URL and increments a count of the counter if the extracted information indicates that the web page corresponding to the URL was visited, the count indicating the number of visits to the web page.

23. The search system of claim 12, wherein the processing module maintains a plurality of counters corresponding to a URL and increments a count of one of the counters if the extracted information indicates that the web page corresponding to the URL was visited by a client device located in a geographical location corresponding to the counter of which the count was incremented, the count indicating the number of visits to the web page from client devices in the corresponding geographical location.

24. The search system of claim 23, wherein the processing module increments the count only if the extracted information indicates that the web page was visited by the client device having a distinct IP address.

25. The search system of claim 23, wherein the processing module does not increment the count if the extracted information indicates that the packets were automatically and repeatedly generated by a computer.

26. The search system of claim 12, wherein the monitoring devices detect requests to stale web pages.

27. The search system of claim 12, wherein the monitoring devices detect pages unknown to the search engine.

28. A method for determining popularity of web pages on a network, the method comprising:

receiving a TCP packet traversing the network;

determining from the received TCP packet whether a TCP session has started; and

responsive to determining that a TCP session has started, extracting information on the TCP session from subsequent TCP packets traversing the network, the extracted information indicating the popularity of the web pages, and the popularity of the web pages being substantially proportionate to number of visits to the web pages as indicated by the extracted information.

29. The method of claim 28, wherein receiving a TCP packet comprises retrieving packets having a protocol field value of 6.

30. The method of claim 28, wherein determining whether a TCP session has started comprises determining whether the received TCP packet is a SYN packet and has a destination port number of 80.

31. The method of claim 28, wherein the extracted information includes:

a requested URI or URL;

a client IP address; and

a server IP address and a server host name.

32. The method of claim 31, wherein the extracted information further includes a referrer URL.

33. The method of claim 31, further comprising increasing a popularity count corresponding to a web page if the extracted information indicates that the web page was visited as determined based upon the requested URI or URL.

34. The method of claim 33, wherein the popularity count indicates the number of visits to the web page from client devices in a geographical location corresponding to the popularity count.

35. The method of claim 33, wherein the popularity count is increased only if the extracted information indicates that the web page was visited by a client device having a distinct IP address.

36. The method of claim 33, wherein the popularity count is not increased if the extracted information indicates that the packets were automatically and repeatedly generated by a computer.

37. The method of claim 28, wherein extracting information on the TCP sessions comprises discarding packets relating to invalid URLs, invalid GET Requests, requests from a web crawler, or auto-refreshment of previous TCP sessions.

38. A method for ranking Internet search results based upon popularity of web pages, the method comprising:

receiving a search term;

performing search of web pages on the Internet based upon the received search term;

retrieving a plurality of web pages containing the search term; and

ranking the web pages at least in part based upon the popularity of the retrieved web pages, the popularity of the retrieved web pages being substantially proportionate to number of visits to the web pages.

39. The method of claim 38, wherein ranking the web pages comprises ranking the web pages based upon the content of the web pages and the hyperlink structure linking the web pages as well as the popularity of the retrieved web pages.

40. The method of claim 38, further comprising propagating a score of a first web page to a plurality of second web pages to which the first web page is linked substantially in relative proportion to the popularity of links from the first web page to each of the second web pages.

41. The method of claim 38, wherein ranking the web pages comprises ranking a first retrieved web page in higher priority than a second retrieved web page if the popularity of the first retrieved web page is greater than the popularity of the second retrieved web page.

42. A system for determining popularity of web pages, the system comprising:

a plurality of monitoring means placed in a network for monitoring packets traversing the network and extracting information on the packets; and

a processing means coupled to the monitoring means for receiving the extracted information from the monitoring device, analyzing the extracted information, and determining the popularity of the web pages based upon the extracted information, the popularity of the web pages being substantially proportionate to number of visits to the web pages.

43. A search system for ranking Internet search results based upon popularity of web pages, the search system comprising:

a plurality of monitoring means placed in a network for monitoring packets traversing the network and extracting information on the packets;

processing means coupled to the monitoring means for receiving the extracted information from the monitoring devices, analyzing the extracted information, and determining the popularity of the web pages based upon the extracted information, the popularity of the web pages being substantially proportionate to number of visits to the web pages; and

search engine means for receiving search terms and retrieving web pages containing the search terms, the search engine means ranking the web pages at least in part based upon the popularity of the retrieved web pages.

44. A method for determining popularity of links from a first web page to a plurality of second web pages on a network, the method comprising:

receiving TCP packets traversing the links;

extracting information on a TCP session comprised of the received TCP packets; and

determining popularity of the links based upon the extracted information, the popularity of each of the links being substantially proportionate to number of times each of the links is traversed.

45. The method of claim 44, further comprising propagating a score of a first web page to a plurality of second web pages to which the first web page is linked substantially in relative proportion to the popularity of links from the first web page to each of the second web pages.

46. The method of claim 44, wherein the extracted information includes at least a referrer URI and a requested URI of the TCP packets.

47. A system for determining popularity of links from a first web page to a plurality of second web pages on a network, the system comprising:

a plurality of monitoring devices placed in the network, the monitoring devices monitoring packets traversing the network and extracting information on the packets; and

a processing module coupled to the monitoring devices and receiving the extracted information from the monitoring devices, the processing module analyzing the extracted information and determining the popularity of the links based upon the extracted information, the popularity of each of the links being substantially proportionate to number of times each of the links is traversed.

48. A search system comprising:

a plurality of monitoring devices placed in the network, the monitoring devices monitoring packets traversing the network and extracting information on the packets;

a processing module coupled to the monitoring devices and receiving the extracted information from the monitoring devices, the processing module analyzing the extracted information and determining the popularity of the links from a first web page to a plurality of second web pages based upon the extracted information, the popularity of each of the links being substantially proportionate to number of times each of the links is traversed; and

a search engine for receiving search terms and retrieving web pages containing the search terms, the search engine propagating a score of the first web page to the second web pages to which the first web page is linked substantially in relative proportion to the popularity of links from the first web page to each of the second web pages.