Methods and systems for prioritizing a crawl

• US 8,386,728 B1
• Filed: 09/14/2004
• Issued: 02/26/2013
• Est. Priority Date: 03/31/2004
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ionescu, Mihai Florin, Marmaros, David
• Primary Examiner(s): TSAI, SHENG JEN
• Application Number:
  US10/940,314
• Time in Patent Office: 3,087 Days
• Field of Search: 711/158
• US Class Current: 711/158
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

Methods and systems for prioritizing a crawl are described. One aspect of the invention includes a method for identifying a plurality of storage locations each comprising a plurality of articles, ranking the plurality of storage locations based at least in part on events associated with the plurality of articles, and crawling the storage locations based at least in part on the ranking. Another aspect of the invention includes identifying a plurality of storage locations each comprising a plurality of articles, identifying a plurality of types of the plurality of articles, ranking the plurality of storage locations based at least in part on the plurality of types of the plurality of articles; and crawling the storage locations based at least in part on the ranking.

Claims

1. A method for crawling articles on a storage device local to a client device, comprising:

(a) identifying, by an application executing on the client device, a plurality of storage locations located on the client device, each storage location storing a plurality of articles;

(b) identifying, by the application executing on the client device, events performed by a user of the client device, wherein the events are associated with the plurality of articles;

(c) ranking, by the application executing on the client device, the plurality of storage locations based at least in part on the events; and

(d) crawling, by the application executing on the client device, the plurality of storage locations based at least in part on the ranking, the crawling comprising;

identifying, for one of the storage locations, a duplicate set of storage locations;

crawling the one of the storage locations and repressing crawls of the duplicate set of storage locations;

indexing the plurality of articles of at least the one of the storage locations, and

determining a time to re-crawl based on the plurality of articles of at least the one of the storage locations.

2. The method of claim 1, wherein the plurality of storage locations is located on a single storage device local to the client device.

3. The method of claim 1, wherein the plurality of storage locations is associated with a plurality of storage devices local to the client device.

4. The method of claim 1, wherein the plurality of storage locations comprises at least one of the following:

a client side device, a local hard drive, a shared drive, a data storage device.

5. The method of claim 1, wherein events associated with the plurality of articles comprise at least one of the following:

editing an article, saving an article to a directory, saving a file to a directory, saving an article to a folder, saving a file to a folder, saving an article, saving a file, receiving an e-mail from a particular sender, and sending an e-mail to a particular recipient.

6. The method of claim 1, wherein the events comprise event data that is obtained from at least one of the following:

an operating system, a browser application program, an enabling/monitoring application program, a previous version of a search application program, and a previous version of an application program associated with a search engine application.

7. The method of claim 1, wherein the plurality of articles comprises at least one of the following:

a directory, a folder, a file, a webpage, e-mail, a message, and a file associated with an application program.

8. The method of claim 1, wherein identifying a plurality of storage locations comprises at least one of the following:

identifying a recently accessed file list, identifying a my favorites list, identifying a my documents folder, identifying a desktop folder, identifying a list of recently accessed files associated with an application program, identifying a folder of recent e-mails, identifying a folder of recent messages, and identifying a cached list of webpages.

9. The method of claim 1, wherein identifying a plurality of storage locations comprises identifying events associated with the plurality of articles.

10. The method of claim 9, wherein the events associated with the plurality of articles comprise at least one of the following:

user access of an article, user access of a file, accessing an article, accessing a file, saving an article, saving a file, opening an article, and opening a file.

11. The method of claim 1, wherein identifying a plurality of storage locations comprises identifying characteristics associated with the plurality of articles.

12. The method of claim 11, wherein characteristics associated with the plurality of articles comprises at least one of the following:

file size, file type, file recency, a date/time of last access, a date/time of recent access, a date/time of last modification, a date/time of creation, a date, and a time.

13. The method of claim 1, further comprising:

(e) identifying a plurality of types of the plurality of articles, wherein ranking the plurality of storage locations comprises ranking the storage locations based at least in part on the plurality of types of the plurality of articles.

14. The method of claim 13, wherein identifying a plurality of types of the plurality of articles comprises prioritizing each of the plurality of types, wherein ranking the storage locations based at least in part on the plurality of types comprises ranking the storage locations based at least in part on the prioritizing each of the plurality of types.

15. The method of claim 1, wherein (c) ranking the plurality of storage locations based at least in part on events associated with the plurality of articles comprises prioritizing the plurality of storage locations based on a frequency of the events.

16. The method of claim 1, wherein (c) ranking the plurality of storage locations based at least in part on events associated with the plurality of articles comprises prioritizing the plurality of storage locations based on a recency of the events.

17. The method of claim 15, wherein (d) crawling the storage locations based at least in part on the ranking comprises crawling a predefined number of the plurality of storage locations, wherein the ranking is prioritized from a highest frequency to a lower frequency.

18. The method of claim 1, wherein (d) crawling the storage locations based at least in part on the ranking comprises at least one of the following:

crawling a predefined number of storage locations, crawling a predefined number of directories, crawling a top directory in a file hierarchy, crawling a predefined number of folders, crawling a predefined number of articles, crawling a predefined number of files, crawling a predefined number of indexable files, crawling for a predefined amount of time, and crawling to a predefined depth in a folder hierarchy starting from either a root or a predefined starting folder.

19. The method of claim 1, wherein (d) crawling the storage locations based at least in part on the ranking comprises crawling at least some of the storage locations.

20. The method of claim 1, wherein (d) crawling the storage locations based at least in part on the ranking comprises prioritizing the crawling of at least some of the storage locations based at least in part on the ranking.

21. A method for crawling articles on a storage device local to a client device, comprising:

(a) identifying, by an application executing on the client device, a plurality of storage locations located on the client device, each storage location storing a plurality of articles;

(b) identifying, by the application executing on the client device, a plurality of types of the plurality of articles;

(c) ranking, by the application executing on the client device, the plurality of storage locations based at least in part on the plurality of types of the plurality of articles stored by each storage location; and

(d) crawling, by the application executing on the client device, the plurality of storage locations based at least in part on the ranking, the crawling comprising;

identifying, for one of the storage locations, a duplicate set of storage locations;

crawling the one of the storage locations and repressing crawls of the duplicate set of storage locations;

indexing the plurality of articles of at least the one of the storage locations, and

determining a time to re-crawl based on the plurality of articles of at least the one of the storage locations.

22. The method of claim 21, wherein the plurality of storage locations is located on a single storage device local to the client device.

23. The method of claim 21, wherein the plurality of storage locations is associated with a plurality of storage devices local to the client device.

24. The method of claim 21, wherein the plurality of storage locations comprises at least one of the following:

a client side device, a local hard drive, a shared drive, a data storage device.

25. The method of claim 21, wherein types of the plurality of articles comprise at least one of the following:

a file type, a document type, an e-mail type, a message type, and a website domain.

26. The method of claim 21, wherein the plurality of articles comprises at least one of the following:

a directory, a folder, a file, a webpage, e-mail, a message, and a file associated with an application program.

27. The method of claim 21, wherein identifying a plurality of storage locations comprises at least one of the following:

identifying a recently accessed file list, identifying a my favorites list, identifying a my documents folder, identifying a desktop folder, identifying a list of recently accessed files associated with an application program, identifying a folder of recent e-mails, identifying a folder of recent messages, and identifying a cached list of webpages.

28. The method of claim 21, wherein identifying a plurality of storage locations comprises identifying events associated with the plurality of articles.

29. The method of claim 28, wherein the events comprise event data that is obtained from at least one of the following:

an operating system, a browser application program, an enabling/monitoring application program, a previous version of a search application program, and a previous version of an application program associated with a search engine application.

30. The method of claim 21, wherein identifying a plurality of storage locations comprises identifying characteristics associated with the plurality of articles.

31. The method of claim 30, wherein characteristics associated with the plurality of articles comprises at least one of the following:

file size, file type, file recency, a date/time of last access, a date/time of recent access, a date/time of last modification, a date/time of creation, a date, and a time.

32. The method of claim 21, wherein (c) ranking the plurality of storage locations based at least in part on the plurality of types of the plurality of articles comprises ranking the plurality of storage locations based at least in part on events associated with the plurality of articles.

33. The method of claim 21, wherein (c) ranking the plurality of storage locations based at least in part on the plurality of types of the plurality of articles comprises prioritizing the plurality of storage locations based at least in part on a frequency of events associated with the plurality of articles.

34. The method of claim 21, wherein (c) ranking the plurality of storage locations based at least in part on the plurality of types of the plurality of articles comprises prioritizing the plurality of storage locations based on a recency of events associated with the plurality of articles.

35. The method of claim 33, wherein (d) crawling the storage locations based at least in part on the ranking comprises crawling a predefined number of the plurality of storage locations, wherein the ranking is prioritized from a highest frequency of events to a lower frequency of events.

36. The method of claim 21, wherein (d) crawling the storage locations based at least in part on the ranking comprises at least one of the following:

crawling a predefined number of storage locations, crawling a predefined number of directories, crawling a top directory in a file hierarchy, crawling a predefined number of folders, crawling a predefined number of articles, crawling a predefined number of files, crawling a predefined number of indexable files, crawling for a predefined amount of time, and crawling to a predefined depth in a folder hierarchy starting from either a root or a predefined starting folder.

37. The method of claim 21, wherein (d) crawling the storage locations based at least in part on the ranking comprises crawling at least some of the storage locations.

38. The method of claim 21, wherein (d) crawling the storage locations based at least in part on the ranking comprises prioritizing the crawling of at least some of the storage locations based at least in part on the ranking.

39. A non-transitory computer-readable storage medium containing executable program code, comprising:

(a) program code for execution on a client device for identifying a plurality of storage locations located on the client device, each storage location storing a plurality of articles;

(b) program code for execution on the client device for identifying events performed by a user of the client device, wherein the events are associated with the plurality of articles;

(c) program code for execution on the client device for ranking the plurality of storage locations based at least in part on events; and

(d) program code for execution on the client device for crawling the plurality of storage locations based at least in part on the ranking, the crawling comprising;

identifying, for one of the storage locations, a duplicate set of storage locations;

crawling the one of the storage locations and repressing crawls of the duplicate set of storage locations;

indexing the plurality of articles of at least the one of the storage locations, and

determining a time to re-crawl based on the plurality of articles of at least the one of the storage locations.

40. The non-transitory computer-readable storage medium of claim 39, wherein the plurality of storage locations is located on a single storage device local to the client device.

41. The non-transitory computer-readable storage medium of claim 39, wherein the plurality of storage locations is associated with a plurality of storage devices local to the client device.

42. The non-transitory computer-readable storage medium of claim 39, wherein the plurality of storage locations comprises at least one of the following:

a client side device, a local hard drive, a shared drive, a data storage device.

43. The non-transitory computer-readable storage medium of claim 39, wherein events associated with the plurality of articles comprise at least one of the following:

editing an article, saving an article to a directory, saving a file to a directory, saving an article to a folder, saving a file to a folder, saving an article, saving a file, receiving an e-mail from a particular sender, and sending an e-mail to a particular recipient.

44. The non-transitory computer-readable storage medium of claim 39, wherein the events comprise event data that is obtained from at least one of the following:

an operating system, a browser application program, an enabling/monitoring application program, a previous version of a search application program, and a previous version of an application program associated with a search engine application.

45. The non-transitory computer-readable storage medium of claim 39, wherein the plurality of articles comprises at least one of the following:

a directory, a folder, a file, a webpage, e-mail, a message, and a file associated with an application program.

46. The non-transitory computer-readable storage medium of claim 39, wherein program code for identifying a plurality of storage locations comprises at least one of the following:

program code for identifying a recently accessed file list, program code for identifying a my favorites list, program code for identifying a my documents folder, program code for identifying a desktop folder, program code for identifying a list of recently accessed files associated with an application program, program code for identifying a folder of recent e-mails, program code for identifying a folder of recent messages, and program code for identifying a cached list of webpages.

47. The non-transitory computer-readable storage medium of claim 39, wherein program code for identifying a plurality of storage locations comprises program code for identifying events associated with the plurality of articles.

48. The non-transitory computer-readable storage medium of claim 47, wherein the events associated with the plurality of articles comprise at least one of the following:

user access of an article, user access of a file, accessing an article, accessing a file, saving an article, saving a file, opening an article, and opening a file.

49. The non-transitory computer-readable storage medium of claim 39, wherein program code for identifying a plurality of storage locations comprises program code for identifying characteristics associated with the plurality of articles.

50. The non-transitory computer-readable storage medium of claim 49, wherein characteristics associated with the plurality of articles comprises at least one of the following:

file size, file type, file recency, a date, and a time.

51. The non-transitory computer-readable storage medium of claim 39, further comprising:

(e) program code for identifying a plurality of types of the plurality of articles, wherein program code for ranking the plurality of storage locations comprises program code for ranking the storage locations based at least in part on the plurality of types of the plurality of articles.

52. The non-transitory computer-readable storage medium of claim 51, wherein the program code for identifying a plurality of types of the plurality of articles comprises program code for prioritizing each of the plurality of types, wherein program code for ranking the storage locations based at least in part on the plurality of types comprises program code for ranking the storage locations based at least in part on the prioritizing each of the plurality of types.

53. The non-transitory computer-readable storage medium of claim 39, wherein (c) the program code for ranking the plurality of storage locations based at least in part on events associated with the plurality of articles comprises program code for prioritizing the plurality of storage locations based on a frequency of the events.

54. The non-transitory computer-readable storage medium of claim 39, wherein (c) the program code for ranking the plurality of storage locations based at least in part on events associated with the plurality of articles comprises program code for prioritizing the plurality of storage locations based on a recency of the events.

55. The non-transitory computer-readable storage medium of claim 53, wherein (d) the program code for crawling the storage locations based at least in part on the ranking comprises program code for crawling a predefined number of the plurality of storage locations, wherein the ranking is prioritized from a highest frequency to a lower frequency.

56. The non-transitory computer-readable storage medium of claim 39, wherein (d) the program code for crawling the storage locations based at least in part on the ranking comprises at least one of the following:

program code for crawling a predefined number of storage locations, program code for crawling a predefined number of directories, program code for crawling a top directory in a file hierarchy, program code for crawling a predefined number of folders, program code for crawling a predefined number of articles, program code for crawling a predefined number of files, program code for crawling a predefined number of indexable files, program code for crawling for a predefined amount of time, and program code for crawling to a predefined depth in a folder hierarchy starting from either a root or a predefined starting folder.

57. The non-transitory computer-readable storage medium of claim 39, wherein (d) the program code for crawling the storage locations based at least in part on the ranking comprises program code for crawling at least some of the storage locations.

58. The non-transitory computer-readable storage medium of claim 39, (d) the program code for crawling the storage locations based at least in part on the ranking comprises program code for prioritizing the crawling of at least some of the storage locations based at least in part on the ranking.

59. A non-transitory computer-readable storage medium containing executable program code, comprising:

(a) program code for execution on a client device for identifying a plurality of storage locations located on the client device, each storage location storing a plurality of articles;

(b) program code for execution on the client device for identifying a plurality of types of the plurality of articles;

(c) program code for execution on the client device for ranking the plurality of storage locations based at least in part on the plurality of types of the plurality of articles; and

(d) program code for execution on the client device for crawling the plurality of storage locations based at least in part on the ranking, the crawling comprising;

identifying, for one of the storage locations, a duplicate set of storage locations;

crawling the one of the storage locations and repressing crawls of the duplicate set of storage locations;

indexing the plurality of articles of at least the one of the storage locations, and

determining a time to re-crawl based on the plurality of articles of at least the one of the storage locations.

60. The non-transitory computer-readable storage medium of claim 59, wherein the plurality of storage locations is located on a single storage device local to the client device.

61. The non-transitory computer-readable storage medium of claim 59, wherein the plurality of storage locations is associated with a plurality of storage devices local to the client device.

62. The non-transitory computer-readable storage medium of claim 59, wherein the plurality of storage locations comprises at least one of the following:

a client side device, a local hard drive, a shared drive, a data storage device.

63. The non-transitory computer-readable storage medium of claim 59, wherein types of the plurality of articles comprise at least one of the following:

a file type, a document type, an e-mail type, a message type, and a website domain.

64. The non-transitory computer-readable storage medium of claim 59, wherein the plurality of articles comprises at least one of the following:

a directory, a folder, a file, a webpage, e-mail, a message, and a file associated with an application program.

65. The non-transitory computer-readable storage medium of claim 59, wherein the program code for identifying a plurality of storage locations comprises at least one of the following:

program code for identifying a recently accessed file list, program code for identifying a my favorites list, program code for identifying a my documents folder, program code for identifying a desktop folder, program code for identifying a list of recently accessed files associated with an application program, program code for identifying a folder of recent e-mails, program code for identifying a folder of recent messages, and program code for identifying a cached list of webpages.

66. The non-transitory computer-readable storage medium of claim 59, wherein program code for identifying a plurality of storage locations comprises program code for identifying events associated with the plurality of articles.

67. The non-transitory computer-readable storage medium of claim 66, wherein the events comprise event data that is obtained from at least one of the following:

an operating system, a browser application program, an enabling/monitoring application program, a previous version of a search application program, and a previous version of an application program associated with a search engine application.

68. The non-transitory computer-readable storage medium of claim 59, wherein the program code for identifying a plurality of storage locations comprises program code for identifying characteristics associated with the plurality of articles.

69. The non-transitory computer-readable storage medium of claim 68, wherein the characteristics associated with the plurality of articles comprises at least one of the following:

file size, file type, file recency, a date/time of last access, a list of dates/times of recent accesses, a date/time of last modification, a date/time of creation, a date, and a time.

70. The non-transitory computer-readable storage medium of claim 59, wherein (c) the program code for ranking the plurality of storage locations based at least in part on the plurality of types of the plurality of articles comprises program code for ranking the plurality of storage locations based at least in part on events associated with the plurality of articles.

71. The non-transitory computer-readable storage medium of claim 59, wherein (c) the program code for ranking the plurality of storage locations based at least in part on the plurality of types of the plurality of articles comprises program code for prioritizing the plurality of storage locations based at least in part on a frequency of events associated with the plurality of articles.

72. The non-transitory computer-readable storage medium of claim 59, wherein (c) the program code for ranking the plurality of storage locations based at least in part on the plurality of types of the plurality of articles comprises program code for prioritizing the plurality of storage locations based on a recency of events associated with the plurality of articles.

73. The non-transitory computer-readable storage medium of claim 59, wherein (d) the program code for crawling the storage locations based at least in part on the ranking comprises program code for crawling a predefined number of the plurality of storage locations, wherein the ranking is prioritized from a highest frequency of events to a lower frequency of events.

74. The non-transitory computer-readable storage medium of claim 59, wherein (d) the program code for crawling the storage locations based at least in part on the ranking comprises at least one of the following:

program code for crawling a predefined number of storage locations, program code for crawling a predefined number of directories, program code for crawling a top directory in a file hierarchy, program code for crawling a predefined number of folders, program code for crawling a predefined number of articles, program code for crawling a predefined number of files, program code for crawling a predefined number of indexable files, program code for crawling for a predefined amount of time, and program code for crawling to a predefined depth in a folder hierarchy starting from either a root or a predefined starting folder.

75. The non-transitory computer-readable storage medium of claim 59, wherein (d) the program code for crawling the storage locations based at least in part on the ranking comprises program code for crawling at least some of the storage locations.

76. The non-transitory computer-readable storage medium of claim 59, wherein (d) the program code for crawling the storage locations based at least in part on the ranking comprises program code for prioritizing the crawling of at least some of the storage locations based at least in part on the ranking.

77. A system for crawling local articles, comprising:

a computer processor for executing computer program instructions;

a computer-readable storage medium having executable computer program instructions tangibly embodied thereon, the executable computer program instructions comprising instructions for;

(a) identifying a plurality of storage locations located on the system, each storage location storing a plurality of articles;

(b) identifying a plurality of types of the plurality of articles;

(c) ranking the plurality of storage locations based at least in part on the plurality of types of the plurality of articles stored by each storage location and based at least in part on events performed by a user of the system wherein the events are associated with the plurality of articles; and

(d) crawling the plurality of storage locations based at least in part on the ranking, the crawling comprising;

identifying, for one of the storage locations, a duplicate set of storage locations;

crawling the one of the storage locations and repressing crawls of the duplicate set of storage locations;

indexing the plurality of articles of at least the one of the storage locations, and

determining a time to re-crawl based on the plurality of articles of at least the one of the storage locations.