Methods and systems for prioritizing a crawl

• US 9,189,553 B2
• Filed: 01/28/2013
• Issued: 11/17/2015
• Est. Priority Date: 03/31/2004
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ionescu, Mihai Florin, Marmaros, David
• Primary Examiner(s): TSAI, SHENG JEN
• Application Number:
  US13/752,292
• Publication Number:
  US 20150186513A1
• Time in Patent Office: 1,023 Days
• Field of Search: 711/158
• US Class Current: 1/1
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

Methods and systems for prioritizing a crawl are described. One aspect of the invention includes a method for identifying a plurality of storage locations each comprising a plurality of articles, ranking the plurality of storage locations based at least in part on events associated with the plurality of articles, and crawling the storage locations based at least in part on the ranking. Another aspect of the invention includes identifying a plurality of storage locations each comprising a plurality of articles, identifying a plurality of types of the plurality of articles, ranking the plurality of storage locations based at least in part on the plurality of types of the plurality of articles; and crawling the storage locations based at least in part on the ranking.

Claims

1. A computer-implemented method comprising the following operations performed by one or more processors:

identifying a plurality of directories located on a client device, each of the directories storing a plurality of articles of different types;

determining, for each of the article types, a number of the articles that have the same article type in each identified directory;

ranking the directories in an order, with respect to each article type, based at least in part on the determined number of articles in each directory having the same article type;

crawling the directories based at least in part on the ranking; and

indexing the plurality of articles based on the crawling.

2. The computer-implemented method of claim 1, wherein the plurality of directories is located on one or more storage devices local to the client device.

3. The computer-implemented method of claim 1, further comprising ranking the directories based at least in part on events associated with the articles, wherein the events comprise at least one of the following:

editing an article, saving an article to a directory, saving a file to a directory, saving an article to a folder, saving a file to a folder, saving an article, saving a file, receiving an e-mail from a particular sender, and sending an e-mail to a particular recipient.

4. The computer-implemented method of claim 1, further comprising ranking the directories based at least in part on events associated with the articles, wherein the events comprise event data that is obtained from at least one of the following:

an operating system, a browser application program, an enabling/monitoring application program, a previous version of a search application program, and a previous version of an application program associated with a search engine application.

5. The computer-implemented method of claim 1, wherein identifying the plurality of directories comprises at least one of the following:

identifying a recently accessed file list, identifying a my favorites list, identifying a my documents list, identifying a desktop folder, identifying a list of recently accessed files associated with an application program, identifying a folder of recent e-mails, identifying a folder of recent messages, and identifying a cached list of webpages.

6. The computer-implemented method of claim 1, wherein identifying the plurality of directories comprises identifying events associated with the articles.

7. The computer-implemented method of claim 1, further comprising ranking the directories based at least in part on events associated with the articles, wherein ranking the directories based at least in part on the events comprises prioritizing the directories based on at least one of a frequency of the events and a recency of the events.

21. The computer-implemented method of claim 1, further comprising:

detecting that articles having a first article type of the article types occur more frequently in the directories than articles having a second article type of the article types;

ranking a first directory storing an article having the first article type higher than a second directory storing an article having the second article type based on the detection; and

crawling the first directory before the second directory based on the ranking.

8. A computer-implemented method comprising the following operations performed by one or more processors:

identifying a plurality of storage locations located on a client device, each of the storage locations storing a plurality of articles of different types;

determining, for each of the article types, a number of the articles having the same article type in each identified storage location;

ranking the storage locations in an order, with respect to each article type, based at least in part on the determined number of articles in each storage location having the same article type;

crawling the storage locations based at least in part on the ranking; and

indexing the plurality of articles based on the crawling.

9. The computer-implemented method of claim 8, wherein the plurality of storage locations is located on one or more storage devices local to the client device.

10. The computer-implemented method of claim 8, wherein identifying the plurality of storage locations comprises at least one of the following:

identifying a recently accessed file list, identifying a my favorites list, identifying a my documents folder, identifying a desktop folder, identifying a list of recently accessed files associated with an application program, identifying a folder of recent e-mails, identifying a folder of recent messages, and identifying a cached list of webpages.

11. The computer-implemented method of claim 8, wherein identifying the plurality of storage locations comprises identifying events associated with the articles.

12. The computer-implemented method of claim 8, wherein the types of the articles comprise at least one of the following:

a file type, a document type, an e-mail type, a message type, and a website domain.

13. The computer-implemented method of claim 8, wherein the articles comprise at least one of the following:

a directory, a folder, a file, a webpage, e-mail, a message, and a file associated with an application program.

20. The computer-implemented method of claim 8, further comprising identifying one of the article types by:

identifying a first storage location configured to store recently-accessed articles;

identifying different types of articles stored within the first storage location; and

identifying the one article type from among the different article types based on a number of the articles stored within the first storage location that have the one article type.

22. The computer-implemented method of claim 8, further comprising:

detecting that articles having a first article type of the article types occur more frequently in the directories than articles having a second article type of the article types;

ranking a first directory storing an article having the first article type higher than a second directory storing an article having the second article type based on the detection; and

crawling the first directory before the second directory based on the ranking.

14. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, configure the at least one processor to perform operations comprising:

identifying a plurality of storage locations located on a client device, each of the storage locations storing a plurality of articles of different types;

determining, for each of the article types, a number of the articles having the same article type in each identified storage location;

ranking the storage locations in an order, with respect to each article type, based at least in part on the determined number of articles in each storage location having the same article type;

crawling the storage locations based at least in part on the ranking; and

indexing the plurality of articles based on the crawling.

15. The non-transitory computer-readable storage medium of claim 14, wherein the plurality of storage locations is located on one or more storage devices local to the client device.

16. The non-transitory computer-readable storage medium of claim 14, wherein identifying the plurality of storage locations comprises identifying events associated with the articles.

17. The non-transitory computer-readable storage medium of claim 14, wherein the types of the plurality of articles comprise at least one of the following:

a file type, a document type, an e-mail type, a message type, and a website domain.

18. The non-transitory computer-readable storage medium of claim 14, wherein the articles comprise at least one of the following:

a directory, a folder, a file, a webpage, e-mail, a message, and a file associated with an application program.

19. The non-transitory computer-readable storage medium of claim 14, further comprising ranking the directories based at least in part on events associated with the articles, wherein ranking the storage locations based at least in part on events comprises prioritizing the storage locations based on at least one of a frequency of the events and a recency of the events.

23. The non-transitory computer-readable storage medium of claim 14, further comprising instructions that configure the at least one processor to perform the following operations:

detecting that articles having a first article type of the article types occur more frequently in the directories than articles having a second article type of the article types;

ranking a first directory storing an article having the first article type higher than a second directory storing an article having the second article type based on the detection; and

crawling the first directory before the second directory based on the ranking.