Methods and systems for prioritizing a crawl

• US 9,836,544 B2
• Filed: 11/13/2015
• Issued: 12/05/2017
• Est. Priority Date: 03/31/2004
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ionescu, Mihai Florin, Marmaros, David
• Primary Examiner(s): TSAI, SHENG JEN
• Application Number:
  US14/940,862
• Publication Number:
  US 20160070797A1
• Time in Patent Office: 753 Days
• Field of Search: 711158
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

Methods and systems for prioritizing a crawl are described. One aspect of the invention includes a method for identifying a plurality of storage locations each comprising a plurality of articles, ranking the plurality of storage locations based at least in part on events associated with the plurality of articles, and crawling the storage locations based at least in part on the ranking. Another aspect of the invention includes identifying a plurality of storage locations each comprising a plurality of articles, identifying a plurality of types of the plurality of articles, ranking the plurality of storage locations based at least in part on the plurality of types of the plurality of articles; and crawling the storage locations based at least in part on the ranking.

Claims

1. A computer-implemented method for indexing articles, the method comprising the following operations performed by at least one processor:

identifying a plurality of directories located on a client device, each directory of the plurality of directories storing a plurality of articles, wherein at least one or more of the directories store articles of different types;

determining, for each article of the plurality of articles stored within each directory of the plurality of directories, at least one associated event performed with the client device for the article;

ranking the plurality of directories identified on the client device, the ranking of each directory based at least in part on the associated events for each article of the plurality of articles stored in the directory, wherein the ranking of each directory of the one or more directories that store articles of different types is based on the events associated with the each of the articles of different types stored in the directory; and

in response to ranking the plurality of directories identified on the client device;

crawling the plurality of directories based at least in part on the ranking of the plurality of directories, wherein crawling the plurality of directories comprises retrieving multiple different articles from each directory of the plurality of directories; and

indexing the plurality of articles stored in the plurality of directories on the client device based on the crawling of the plurality of directories identified on the client device.

2. The computer-implemented method of claim 1, wherein the events comprise at least one of the following:

editing an article, saving an article to a directory, saving a file to a directory, saving an article to a folder, saving a file to a folder, saving an article, saving a file, receiving an e-mail from a particular sender, and sending an e-mail to a particular recipient.

3. The computer-implemented method of claim 1, wherein the events comprise event data that is obtained from at least one of the following:

an operating system, a browser application program, a monitoring application program, a previous version of a search application program, and a previous version of an application program associated with a search engine application.

4. The computer-implemented method of claim 1, wherein ranking each directory of the plurality of directories based at least in part on the associated events for each article of the plurality of articles stored in the directory comprises prioritizing the directories based on at least one of a frequency of the events, a recency of the events, and a type of the events.

5. The computer-implemented method of claim 1, wherein ranking each directory of the plurality of directories comprises identifying one or more types of the plurality of articles and ranking the plurality of directories based in part on the identified types of articles.

6. The computer-implemented method of claim 1, wherein determining, for each of the articles, at least one associated event comprises:

receiving a notification indicating that an event has been captured and stored in a queue;

retrieving the event from the queue; and

associating the event with at least one article.

7. The computer-implemented method of claim 1, wherein the events comprise:

historical events resulting from a periodic crawl of memory of the client device; and

real-time events describing a current activity on the client device.

8. A computer-implemented system comprising:

a non-transitory computer-readable medium that stores program code; and

at least one processor that executes the program code to perform operations comprising;

identifying a plurality of directories located on a client device, each directory of the plurality of directories storing a plurality of articles, wherein at least one or more of the directories store articles of different types;

determining, for each article of the plurality of articles stored within each directory of the plurality of directories, at least one associated event performed with the client device for the article;

ranking the plurality of directories identified on the client device, the ranking of each directory based at least in part on the associated events for each article of the plurality of articles stored in the directory, wherein the ranking of each directory of the one or more directories that store articles of different types is based on the events associated with each of the articles of different types stored in the directory; and

in response to ranking the plurality of directories identified on the client device;

crawling the plurality of directories based at least in part on the ranking of the plurality of directories, wherein crawling the plurality of directories comprises retrieving multiple different articles from each directory of the plurality of directories; and

indexing the plurality of articles stored in the plurality of directories on the client device based on the crawling of the plurality of directories identified on the client device.

9. The computer-implemented system of claim 8, wherein the events comprise at least one of the following:

editing an article, saving an article to a directory, saving a file to a directory, saving an article to a folder, saving a file to a folder, saving an article, saving a file, receiving an e-mail from a particular sender, and sending an e-mail to a particular recipient.

10. The computer-implemented system of claim 8, wherein the events comprise event data that is obtained from at least one of the following:

an operating system, a browser application program, a monitoring application program, a previous version of a search application program, and a previous version of an application program associated with a search engine application.

11. The computer-implemented system of claim 8, wherein ranking each directory of the plurality of directories based at least in part on the associated events for each article of the plurality of articles stored in the directory comprises prioritizing the directories based on at least one of a frequency of the events, a recency of the events, and a type of the events.

12. The computer-implemented system of claim 8, wherein ranking each directory of the plurality of directories comprises identifying one or more types of the plurality of articles and ranking the plurality of directories based in part on the identified types of articles.

13. The computer-implemented system of claim 8, wherein determining, for each of the articles, at least one associated event comprises:

receiving a notification indicating that an event has been captured and stored in a queue;

retrieving the event from the queue; and

associating the event with at least one article.

14. The computer-implemented system of claim 8, wherein the events comprise:

historical events resulting from a periodic crawl of memory of the client device; and

real-time events describing a current activity on the client device.

15. A non-transitory computer-readable medium storing instructions, that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

identifying a plurality of directories located on a client device, each directory of the plurality of directories storing a plurality of articles, wherein at least one or more of the directories store articles of different types;

determining, for each of the article of the plurality of articles stored within each directory of the plurality of directories, at least one associated event performed with the client device for the article;

ranking the plurality of directories identified on the client device, the ranking of each directory based at least in part on the associated events for each article of the plurality of articles in each stored in the directory, wherein the ranking of each directory of the one or more directories that store articles of different types is based on the events associated with each of the articles of different types stored in the directory; and

in response to ranking the plurality of directories identified on the client device;

crawling the plurality of directories based at least in part on the ranking of the plurality of directories, wherein crawling the plurality of directories comprises retrieving multiple different articles from each directory of the plurality of directories; and

indexing the plurality of articles stored in the plurality of directories on the client device based on the crawling of the plurality of directories identified on the client device.

16. The non-transitory computer-readable medium of claim 15, wherein the events comprise at least one of the following:

editing an article, saving an article to a directory, saving a file to a directory, saving an article to a folder, saving a file to a folder, saving an article, saving a file, receiving an e-mail from a particular sender, and sending an e-mail to a particular recipient.

17. The non-transitory computer-readable medium of claim 15, wherein the events comprise event data that is obtained from at least one of the following:

an operating system, a browser application program, a monitoring application program, a previous version of a search application program, and a previous version of an application program associated with a search engine application.

18. The non-transitory computer-readable medium of claim 15, wherein ranking each directory of the plurality of directories based at least in part on the associated events for each article of the plurality of articles stored in the directory comprises prioritizing the directories based on at least one of a frequency of the events, a recency of the events, and a type of the events.

19. The non-transitory computer-readable medium of claim 15, wherein ranking each directory of the plurality of directories comprises identifying one or more types of the plurality of articles and ranking the plurality of directories based in part on the identified types of articles.

20. The non-transitory computer-readable medium of claim 15, wherein determining, for each of the articles, at least one associated event comprises:

receiving a notification indicating that an event has been captured and stored in a queue;

retrieving the event from the queue; and

associating the event with at least one article.