Managing items in crawl schedule

• US 8,666,964 B1
• Filed: 04/25/2005
• Issued: 03/04/2014
• Est. Priority Date: 04/25/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Alpert, Jesse L.
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): Khakhar, Nirav K
• Application Number:
  US11/113,820
• Time in Patent Office: 3,235 Days
• Field of Search: None
• US Class Current: 707/709
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Determining a schedule for recrawling pages is disclosed. A crawling schedule that specifies a due date at which each page is to be crawled is determined according to a first scheme. A set of pages that includes one or more pages each of which has a due date that has passed is determined. The set of pages is ordered according to a second scheme.

Claims

1. A method of determining a schedule for recrawling items including:

estimating, using a processor of one or more devices, a change period for each of a set of items to be crawled;

generating, using a processor of the one or more devices, a crawl list of items where each item on the crawl list is overdue to be crawled in accordance with the change period for the item and with when the item was last crawled;

selecting, using a processor of the one or more devices, a sorting method from multiple different sorting methods based on one or more different factors; and

sorting, using a processor of the one or more devices, the crawl list prior to crawling the items on the crawl list using the selected sorting method.

2. The method of claim 1 where the sorting method is based, at least in part, on an importance rank assigned to each item.

3. The method of claim 1 where the sorting method is based, at least in part, on the change period estimated for each item.

4. The method of claim 1 where the sorting method is based, at least in part, on how overdue to be crawled is each item.

5. The method of claim 1 where the change period is estimated based on an importance rank assigned to the item.

6. The method of claim 1 where the item is a page or document stored on a network.

7. The method of claim 1 where the item is a page or document stored on a corporate intranet.

8. A system for crawling items in a network comprising:

one or more processors configured to;

estimate a change period for each of a set of items to be crawled;

generate a crawl list of items where each item on the crawl list is overdue to be crawled in accordance with the change period for the item and with when the item was last crawled;

select a sorting method from multiple different sorting methods based on one or more different factors; and

sort the crawl list prior to crawling the items on the crawl list using the selected sorting method.

9. The system of claim 8 where the sorting method is based, at least in part, on an importance rank assigned to each item.

10. The system of claim 8 where the sorting method is based, at least in part, on the change period estimated for each item.

11. The system of claim 8 where the sorting method is based, at least in part, on how overdue to be crawled is each item.

12. The system of claim 8 where the change period is estimated based on an importance rank assigned to the item.

13. The system of claim 8 where the item is a page or document stored on a network.

14. The system of claim 8 where the item is a page or document stored on a corporate intranet.

15. A search appliance comprising:

a crawl scheduler for estimating a change period for each of a set of items to be crawled; and

a crawl manager for generating a crawl list of items where each item on the crawl list is overdue to be crawled in accordance with the change period for the item and with when the item was last crawled, for selecting a sorting method from multiple different sorting methods based on one or more different factors and for sorting the crawl list prior to crawling the items on the crawl list using the selected sorting method.

16. The search appliance of claim 15 where the sorting method is based, at least in part, on an importance rank assigned to each item.

17. The search appliance of claim 15 where the sorting method is based, at least in part, on the change period estimated for each item.

18. The search appliance of claim 15 where the sorting method is based, at least in part, on how overdue to be crawled is each item.

19. The search appliance of claim 15 where the change period is estimated based on an importance rank assigned to the item.

20. The search appliance of claim 15 where the item is a page or document stored on a network.