Sampling Internet user traffic to improve search results

• US 20070112730A1
• Filed: 11/07/2005
• Published: 05/17/2007
• Est. Priority Date: 11/07/2005
• Status: Active Grant

• Current Assignee: IAC Search & Media Incorporated (Match Group, Inc.)
• Original Assignee: IAC Search & Media Incorporated (Match Group, Inc.)
• Inventors: Mori, Monica, Gulli, Antonino, Savona, Antonio
• Granted Patent: US 8,046,357 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods, systems and apparatus for improving Internet search results include monitoring network activity on a communications network, generating a near real-time map of the network activity, and integrating the near real-time map with a search engine.

Claims

1. A method, comprising:

monitoring network activity on a communications network;

generating a near real-time map of the network activity; and

integrating the near real-time map of the network activity with a search engine.

2. The method of claim 1, wherein the communications network comprises the World Wide Web, and wherein monitoring network activity on the communications network comprises monitoring web traffic on at least one of a plurality of remote web servers, a plurality of proxy servers and a plurality of routers.

3. The method of claim 2, wherein generating the near real-time map of the network activity comprises generating a web graph from the web traffic, the web graph comprising a plurality of web nodes linked by a plurality of web edges, wherein each of the plurality of web nodes represents a corresponding web object and each of the plurality of web edges represents a weighted link from a first web object to a second web object.

4. The method of claim 3, wherein integrating the near real-time map of the network activity with a search engine comprises:

updating a keyword index based on the near real-time map;

adding and deleting web objects based on the near real-time map; and

clustering web objects and search engine queries based on the near real-time map

5. The method of claim 3, further comprising:

creating a web edge when a user navigates between the first web object and the second web object without a pre-existing web edge; and

extinguishing the web edge after a specified period of disuse.

6. The method of claim 3, further comprising:

incrementing a weight of the web edge each time a hyperlink in the first web object is used to navigate from the first web object to the second web object; and

decrementing the weight of the web edge after the specified period of disuse.

7. The method of claim 3, further comprising:

incrementing a weight of the web edge each time the user navigates from the first web object to the second web object without using a hyperlink; and

decrementing the weight of the web edge after the specified period of disuse.

8. The method of claim 6, further comprising anchoring the web edge with an anchor comprising a portion of text surrounding the hyperlink in the first web object.

9. The method of claim 8, wherein a search request invokes at least some portion of the anchor, further comprising ranking a search result in a search engine based on the weight of the web edge linking the first web object to the second web object.

10. The method of claim 2, wherein monitoring web traffic comprises extracting HTTP_REFER tags from access logs of the plurality of remote web servers.

11. The method of claim 3, wherein a web object in the web graph is connected to one or more web edges, wherein a weight of the web object corresponds to the weights of the one or more web edges, further comprising adding the web object to a search engine index when the weight of the web object exceeds a predetermined threshold, and deleting the web object from the search engine index when the weight of the web object falls below the predetermined threshold.

12. The method of claim 11, further comprising generating a relevance score for the web object based on the weight of the web object and ranking the web object in a search result.

13. The method of claim 11, further comprising deleting the web object from the search engine index when the web traffic includes missing object errors for the web object.

14. The method of claim 2, wherein the web traffic includes a query and a pick in a first search engine, wherein the pick corresponds to a web object, the method further comprising associating the web object with the query in a second search engine.

15. The method of claim 2, wherein the web traffic includes a first query and a first pick in a first search engine corresponding to a web object and a second query and a second pick in a second search engine corresponding to the web object, the method further comprising clustering the first query and the second query in at least the first search engine.

16. The method of claim 2, wherein the web traffic comprises a burst of queries directed toward a set of web objects, the method further comprising clustering the set of web objects and the burst of queries in the search engine while the burst of queries is sustained above a predetermined threshold.

17. The method of claim 3, wherein the web graph is associated with the web traffic of a definable group of users having one or more web traffic metrics in common during a monitoring period, the method further comprising biasing the search results of a new user with the web graph when the new user exhibits web activity that is similar to the web traffic metrics of the definable group of users.

18. The method of claim 17, wherein the biasing of the search results has a defined lifetime related to the monitoring period.

19. A system, comprising:

a memory;

a communications port; and

a processing device coupled with the memory and the communications port, the processing device configured to;

monitor network activity on a communications network;

generate a near real-time map of the network activity; and

integrate the near real-time map of the network activity with a search engine.

20. The system of claim 19, wherein the communications network comprises the World Wide Web, and wherein to monitor network activity on the communications network, the processing device is configured to monitor web traffic on at least one of a plurality of remote web servers, a plurality of proxy servers and a plurality of routers coupled with the communications port.

21. The system of claim 20, wherein to generate a near real-time map of the network activity the processing device is configured to generate a web graph from the web traffic, the web graph comprising a plurality of web nodes linked by a plurality of web edges, wherein each of the plurality of web nodes represents a corresponding web object and each of the plurality of web edges represents a weighted link from a first web object to a second web object.

22. An article of manufacture, including machine-accessible instructions that when accessed by a data processing system, cause the data processing system to perform a method, comprising:

monitoring network activity on a communications network;

generating a near real-time map of the network activity; and

integrating the near real-time map of the network activity with a search engine.

23. The article of manufacture of claim 22, wherein the communications network comprises the World Wide Web, and wherein monitoring network activity on the communications network comprises monitoring web traffic on at least one of a plurality of remote web servers, a plurality of proxy servers and a plurality of routers.

24. The article of manufacture of claim 23, wherein generating the near real-time map of the network activity comprises generating a web graph from the web traffic, the web graph comprising a plurality of web nodes linked by a plurality of web edges, wherein each of the plurality of web nodes represents a corresponding web objects and each of the plurality of web edges represents a weighted link from a few web object to a second web object.