Web crawling

• US 20050125412A1
• Filed: 03/24/2004
• Published: 06/09/2005
• Est. Priority Date: 12/09/2003
• Status: Abandoned Application

• Current Assignee: NEC Laboratories America Inc (NEC Corporation)
• Original Assignee: NEC Laboratories America Inc (NEC Corporation)
• Inventors: Glover, Eric J.
• Application Number:
  US10/807,698
• Publication Number:
  US 20050125412A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

The present invention is directed to mechanisms for improving the “crawling” of resources on a network, which takes into account the notion of browser state. An improved indexing scheme for the crawled results and improved search mechanisms are also disclosed.

Claims

1. A method for crawling for resources in a network, the method comprising:

receiving a list of resources on the network and for at least one of the resources on the list of resources,

sending a first request to a server in the network for the resource using a first browser state, and

sending a second request for the same resource using a second browser state.

2. The method of claim 1 wherein the resources are identified by uniform resource locators and wherein the first and second request specify a same uniform resource locator.

3. The method of claim 1 wherein the browser state comprises a language preference.

4. The method of claim 1 wherein the browser state comprises a locale preference.

5. The method of claim 1 wherein the browser state comprises location information.

6. The method of claim 1 wherein the browser state comprises a browser identification.

7. The method of claim 1 wherein the browser state comprises a network address.

8. The method of claim 1 wherein the first request and the second request are issued by a first and second crawler applications that respectively have a first and second browser state.

9. The method of claim 1 wherein the first and second requests are issued by a crawler application that varies its browser state between the first and second requests.

10. A method for processing crawled resources in a network, the method comprising:

receiving a resource in response to a request for the resource using one of a plurality of browser states;

storing the resource; and

indexing the resource, the indexing step further comprising the step of associating the resource with a first browser state where the first browser state is the one of the plurality of browser states used to request the resource.

11. The method of claim 10 wherein resources are identified by uniform resource locators and wherein at least a first resource and a second resource identified by a same uniform resource locator are associated with different browser states.

12. The method of claim 11 wherein the first and second resources are both stored only if the second resource is different from the first resource.

13. The method of claim 12 wherein if the second resource is a duplicate of the first resource, a reference is stored that associates the stored first resource with the second browser state.

14. The method of claim 10 wherein the browser state comprises any one of a group consisting of language preference, locale preference, location information, browser identification, and network address.

15. A method for searching a database of crawled resources, the method comprising the steps of:

receiving a search query from a browser client;

detecting a browser state for the browser client; and

searching for results from the database of resource using both the search query and the browser state of the browser client.

16. The method of claim 15 wherein the database includes at least one record which associates a first resource and a second resource in the database with a same uniform resource locator but with different browser states.

17. The method of claim 15 wherein results that match the search query are filtered using the browser state of the browser client.

18. The method of claim 15 wherein a relevance function is utilized to rank results from search of the database and wherein the relevance function considers the browser state of the browser client in ranking the results.

19. The method of claim 15 wherein if the browser state of the browser client does not match any of the browser states in the database, then a default browser state is used in the search.

20. The method of claim 15 wherein the browser state comprises any one of a group consisting of language preference, locale preference, location information, browser identification, and network address.

21. A computer-readable medium comprising one or more instructions which when executed perform the following:

receiving a list of resources on the network and for at least one of the resources on the list of resources,

sending a first request to a server in the network for a resource using a first browser state, and

sending a second request for the same resource using a second browser state.

22. The computer-readable medium of claim 21 wherein the resources are identified by a uniform resource locator and wherein the first and second request specify a same uniform resource locator.

23. The computer-readable medium of claim 21 wherein the browser state comprises any one of a group consisting of language preference, locale preference, location information, browser identification, and network address.

24. A computer-readable medium comprising one or more instructions which when executed perform the following:

receiving a resource in response to a request for the resource using one of a plurality of browser states;

storing the resource; and

indexing the resource, the indexing step further comprising the step of associating the resource with a first browser state where the first browser state is the one of the plurality of browser states used to request the resource.

25. The computer-readable medium of claim 24 wherein resources are identified by uniform resource locators and wherein at least a first resource and a second resource identified by a same uniform resource locator are associated with different browser states.

26. The computer-readable medium of claim 25 wherein the first and second resources are both stored only if the second resource is different from the first resource.

27. The computer-readable medium of claim 26 wherein if the second resource is a duplicate of the first resource, a reference is stored that associates the stored first resource with the second browser state.

28. The computer-readable medium of claim 24 wherein the browser state comprises any one of a group consisting of language preference, locale preference, location information, browser identification, and network address.

29. A computer-readable medium comprising one or more instructions which when executed perform the following:

receiving a search query from a browser client;

detecting a browser state for the browser client; and

searching for results from the database of resource using both the search query and the browser state of the browser client.

30. The computer-readable medium of claim 29 wherein the database includes at least one record which associates a first resource and a second resource in the database with a same uniform resource locator but with different browser states.

31. The computer-readable medium of claim 29 wherein results that match the search query are filtered using the browser state of the browser client.

32. The computer-readable medium of claim 29 wherein a relevance function is utilized to rank results from search of the database and wherein the relevance function considers the browser state of the browser client in ranking the results.

33. The computer-readable medium of claim 29 wherein if the browser state of the browser client does not match any of the browser states in the database, then a default browser state is used in the search.

34. The computer-readable medium of claim 29 wherein the browser state comprises any one of a group consisting of language preference, locale preference, location information, browser identification, and network address.