Generating an index for a network search engine

• US 8,065,292 B2
• Filed: 04/17/2006
• Issued: 11/22/2011
• Est. Priority Date: 04/17/2006
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Yu, Gilbert
• Primary Examiner(s): Vital; Pierre
• Assistant Examiner(s): Rahman; Sabana
• Application Number:
  US11/379,044
• Publication Number:
  US 20070244857A1
• Time in Patent Office: 2,045 Days
• Field of Search: 707/3, 707/753
• US Class Current: 707/711
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method of generating a TrafficRank index for a network search engine is described. The method includes monitoring network traffic through a network device. The method further uses the monitored traffic information to generate and maintain a TrafficRank index. The invention extends to a method of generating search results, including generating search results from a search engine index and ranking the search results based on the TrafficRank index. The invention also extends to a method of generating and updating a search engine index either in real-time or in batch mode. If the method is adopted, any website or URLs accessed through website monitor/notifier/updater module by a web user, either by clicking on a hypertext link, entering the URL address on a web browser, or selecting a bookmark on a web browser, would be included and updated in the search engine index.

Claims

1. A computer-implemented method of generating an index for a search engine, the method comprising:

receiving monitored network traffic information from a plurality of network devices including a first network device and a second network traffic device, the plurality of network devices respectively receiving and forwarding at least a part of the monitored network traffic information, the monitored network traffic information including at least one Uniform Resource Locator (URL) and traffic statistics, the traffic statistics including at least one traffic ranking respectively assigned to the at least one URL;

filtering to identify and remove at least one duplicate access to the at least one URL from the monitored network traffic information to generate filtered monitored network traffic information, the filtering being based on a single source/destination in a single session, the filtering comprising;

identifying a first access to a first URL by identifying a first monitored network traffic information that is received from the first network device as indicating at least one packet as originated from a first network traffic source and destined for the first URL,

identifying a second access to the first URL by identifying a second monitored network traffic information that is received from the second network device indicating at least one packet as originated from the first network traffic source and destined for the first URL, and

removing the second access to the first URL responsive to identifying the second access to the first URL as being performed within a predetermined time period after the first access to the first URL;

ranking the filtered monitored network traffic information according to the traffic statistics; and

updating a search engine index based on the filtered monitored network traffic information as ranked according to the traffic statistics.

2. The method as claimed in claim 1, wherein the monitored network traffic information includes Uniform Resource Locators (URLs) information, the URL information associated with the websites.

3. The method as claimed in claim 2, further comprising:

keeping track of a number of times a website is accessed, wherein a first web site is ranked higher than a second website when a number of times the first website is accessed is higher than a number of times the second website is accessed.

4. The method as claimed in claim 2, wherein the monitored network traffic information is presented further based on algorithmic weightings.

5. The method as in claim 1, wherein the filtering at least one duplicate access to the at least one URL further comprises packet communications in a single session.

6. The method as in claim 1, wherein the receiving the monitored network traffic information from a plurality of network devices includes receiving network traffic information from a first network device and receiving monitored network traffic information from a second network device.

7. A computer-implemented method of generating an index for a network search engine, the method comprising:

receiving monitored network traffic information from a plurality of network devices including a first network device and a second network device, the plurality of network devices respectively receiving and forwarding at least a part of the monitored network traffic information, the monitored network traffic information including at least one Uniform Resource Locator (URL) and traffic statistics;

filtering to identify and remove at least one duplicate access to the at least one URL from the monitored network traffic information to generate filtered monitored network traffic information, the filtering being based on a single source/destination in a single session, the filtering comprising;

identifying a first access to a first URL by identifying a first monitored network traffic information that is received from the first network device as indicating at least one packet as originated from a first network traffic source and destined for the first URL,

identifying a second access to the first URL by identifying a second monitored network traffic information that is received from the second network device indicating at least one packet as originated from the first network traffic source and destined for the first URL,

removing the second access to the first URL responsive to identifying the second access to the first URL as being performed within a predetermined time period after the first access to the first URL;

ranking the filtered monitored network traffic information according to the traffic statistics; and

updating a search engine index based on the filtered monitored network traffic information as ranked according to the traffic statistics.

8. The method as claimed in claim 7, wherein the traffic statistics include a number of times a website associated with the URL has been accessed, wherein search results associated with the search engine index includes information about multiple websites, and wherein the search results are ranked at least according to the number of times each of the multiple websites is accessed.

9. The method as claimed in claim 8, wherein the search results are further ranked according to algorithmic weightings.

10. The method as claimed in claim 7, wherein the search engine index is updated when the URL is associated with more updated content.

11. A computer-implemented method of providing search results, the method comprising:

responsive to a search request,

generating search results using information from a search engine index, the search results including information associated with a plurality of websites;

ranking the plurality of websites based on traffic statistics received from a plurality of network devices including a first network device and a second network device, the plurality of network devices respectively receiving and forwarding at least a part of monitored network traffic information that includes the traffic statistics, the traffic statistics including accesses to the plurality of websites that have been filtered to identify and remove at least one duplicate access based on a single source/destination in a single session, the filter comprises;

an identification of a first access to a first URL by an identification of a first traffic statistic that is received from the first network device that indicates at least one packet as originated from a first network traffic source and destined for the first URL,

an identification of a second access to the first URL by an identification of a second traffic statistic that is received from the second network device that indicates at least one packet as originated from the first network traffic source and destined for the first URL,

a removal of the second access to the first URL responsive to an identification of the second access to the first URL as being performed within a predetermined time period after the first access to the first UR, and

presenting the search results according to said ranking.

12. The method as claimed in claim 11, wherein each of the plurality of websites is associated with an access frequency, and wherein a website accessed more frequently is ranked higher than a website accessed less frequently.

13. A non-transitory computer readable medium embodying instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving monitored network traffic data for each of a plurality of websites, the monitored network traffic data being monitored by a plurality of network devices including a first network device and a second network device that respectively receive and forward at least part of the monitored network traffic data;

storing the monitored network traffic data for each of the plurality of websites;

filtering to identify and remove at least one duplicate access to the plurality of websites from the monitored network traffic data, the filtering being based on a single source/destination in a single session and comprising;

identifying a first access to a first URL by identifying a first monitored network traffic data that is received from the first network device as indicating at least one packet as originated from a first network traffic source and destined for the first URL,

identifying a second access to the first URL by identifying a second monitored network traffic data that is received from the second network device indicating at least one packet as originated from the first network traffic source and destined for the first URL, and

removing the second access to the first URL responsive to identifying the second access to the first URL as being performed within a predetermined time period after the first access to the first URL; and

ranking the plurality of websites in search results at least according to their respective access frequency as determined from the filtered monitored network traffic data.

14. A system for generating an index for a network search engine, the system comprising:

a processor for executing;

means for receiving monitored network traffic data monitored by a plurality of network devices including a first network device and a second network device, the plurality of network devices respectively receive and forward at least a part of the monitored network traffic data;

means for determining data to be added to a search engine index using the monitored network traffic data;

means for keeping track of access frequency of websites associated with Uniform Resource Locators (URLS) included in the monitored network traffic data;

means for filtering to identify and remove at least one duplicate access to at least one of the websites from the monitored network traffic data to generate filtered monitored network traffic data, the filtering being based on a single source/destination in a single session and comprising;

identifying a first access to a first URL, by identifying a first monitored network traffic data that is received from the first network device that indicates at least one packet as originated from a first network traffic source and destined for the first URL,

identifying a second access to the first URL, by identifying a second monitored network traffic data that is received from the second network device that indicates at least one packet as originated from the first network traffic source and destined for the first URL, and

removing the second access to the first URL responsive to an identification of the second access to the first URL as being performed within a predetermined time period after the first access to the first URL;

means for including the filtered monitored network traffic data in the search engine index in search results responsive to a search request; and

means for ranking websites included in the search results at least according to the access frequency of the websites.

15. The system of claim 14, wherein said means for determining the data to be added to the search engine index includes means for determining whether the data is already existing in the search engine index.

16. The system of claim 14, wherein said means for determining the data to be added to the search engine index includes means for determining whether an older version of the data is already existing in the search engine index.

17. The system of claim 14, further including means for ranking the websites included in the search results according to algorithmic weightings in addition said access frequency.

18. A network device comprising:

a processor;

a search engine index coupled to the processor and configured to store information related to a plurality of websites;

a monitoring module coupled to the search engine index and configured to monitor network data received and forwarded by a network device and network data received and forwarded by a plurality of other network devices including a first network device and a second network device, the monitoring module to filter the network data to identify and remove at least one duplicate access to a first website that is included in the plurality of websites based on a single source/destination in a single session, the filter comprises the monitoring module to;

identify a first access to a first URL by an identification of a first network data that is received from the first network device that indicates at least one packet as originated from a first network traffic source and destined for the first URL,

identify a second access to the first URL by an identification of a second network data that is received from the second network device that indicates at least one packet as originated from a first network traffic source and destined for the first URL, and

remove the second access to the first URL responsive to an identification of the second access to the first URL as being performed within a predetermined time period after the first access to the first URL,

the monitoring module to determine whether the network data includes information to be added to the search engine index;

a search module coupled to the search engine index to perform search against the search engine index responsive to a search request; and

a ranking module to rank search results generated by the search module according to a filtered number of accesses of each of the websites included in the search results.

19. The device of claim 18, wherein the number of access of a website is incremented when the network data includes information that indicates a Uniform Resource Locator (URL) associated with the website.

20. A search server system to provide search results to at least one computer connected to the server via a network, the system comprising:

a processor;

a search engine index coupled to the processor and configured to store information used to generate search results, the search results includes information associated with a plurality of websites;

a traffic rank index coupled to the search engine index and configured to store traffic rank information associated with at least some of the plurality of websites, the traffic rank information generated from network traffic information received and forwarded by a plurality of network devices including a first network device and a second network device, the search results are presented at least according to the traffic rank information, and the traffic rank information associated with a website is determined based on number of times the website is accessed and is filtered to identify and remove at least one duplicate access to the website based on a single source/destination in a single session, the filter comprises;

an identification of a first access to a first URL by an identification of a first network traffic information that is received from the first network device that indicates at least one packet as originated from a first network traffic source and destined for the first URL;

an identification of a second access to the first URL by an identification of a second network traffic information that is received from the second network device that indicates at least one packet as originated from a first network traffic source and destined for the first URL and

a removal of the second access to the first URL responsive to an identification of the second access to the first URL as being performed within a predetermined time period after the first access to the first URL.