Scheduling a recrawl

• US 8,386,459 B1
• Filed: 02/22/2011
• Issued: 02/26/2013
• Est. Priority Date: 04/25/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Alpert, Jesse L., Auerbach, David B.
• Primary Examiner(s): Mahmoudi, Tony
• Assistant Examiner(s): Pham, Tuan A
• Application Number:
  US13/032,578
• Time in Patent Office: 735 Days
• Field of Search: None
• US Class Current: 707/709
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method for determining a recrawl schedule of items on an intranet using a search appliance includes estimating, using the search appliance installed on the intranet, a change period for items on the intranet using a history of crawl information for the items. The change period for an item may be a frequency of change in content of the item. The method includes calculating a crawl period for each of the items, where the crawl period for an item is set relative to the change period for the item. The method includes determining whether to adjust the calculated crawl period for each of the items using adjustment criteria, adjusting the calculated crawl period for selected items determined for adjustment using the adjustment criteria and creating a recrawl schedule of the items on the intranet using a recrawl time for each of the items.

Claims

1. A method for determining a recrawl schedule of items on an intranet using a search appliance, the method comprising:

estimating, using a search appliance installed on an intranet, a change period for items on the intranet using a history of crawl information for the items, the change period for an item being a frequency of change in content of the item;

calculating, using the search appliance, a crawl period for each of the items, the crawl period for an item being set relative to the change period for the item;

determining, using the search appliance, whether to adjust the calculated crawl period for each of the items using adjustment criteria, wherein the adjustment criteria includes a combination of an importance rank for an item and a list of hints, wherein the list of hints provides crawl periods for items that include information that match a hint from the list of hints;

adjusting the calculated crawl period for selected items determined for adjustment using the adjustment criteria including when the selected items include information that matches a hint from the list of hints and the calculated crawl period for the selected items exceeds a threshold for the crawl period associated with the hint; and

creating, using the search appliance, a recrawl schedule of the items on the intranet using a recrawl time for each of the items, the recrawl time for each of the items being based on a last crawl time and either the calculated crawl period for the item or the adjusted crawl period for the item.

2. The method as in claim 1 wherein an item includes a document.

3. The method as in claim 1 further comprising optionally sorting the recrawl schedule.

4. The method as in claim 3 wherein sorting the recrawl schedule includes sorting the recrawl schedule such that more overdue items are recrawled before less overdue items.

5. A system for determining a recrawl schedule of items on an intranet, comprising:

one or more processors installed in a search appliance on an intranet, the one or more processors being configured to;

estimate a change period for items on the intranet using a history of crawl information for the items, the change period for an item being a frequency of change in content of the item;

calculate a crawl period for each of the items, the crawl period for an item being set relative to the change period for the item;

determine whether to adjust the calculated crawl period for each of the items using adjustment criteria, wherein the adjustment criteria includes a combination of an importance rank for an item and a list of hints, wherein the list of hints provides crawl periods for items that include information that match a hint from the list of hints;

adjust the calculated crawl period for selected items determined for adjustment using the adjustment criteria, including when the selected items include information that matches a hint from the list of hints and the calculated crawl period for the selected items exceeds a threshold for the crawl period associated with the hint; and

create a recrawl schedule of the items on the intranet using a recrawl time for each of the items, the recrawl time for each of the items being based on a last crawl time and either the calculated crawl period for the item or the adjusted crawl period for the item.

6. The system of claim 5 wherein an item includes a document.

7. The system of claim 5 wherein the one or more processors are configured to optionally sort the recrawl schedule.

8. The system of claim 7 wherein the one or more processors are configured to sort the recrawl schedule such that more overdue items are recrawled before less overdue items.

9. A non-transitory computer-readable storage device comprising instructions for determining a recrawl schedule of items on an intranet that, when executed, cause one or more processors installed on a search appliance to perform the actions of:

estimating, using a search appliance installed on an intranet, a change period for items on the intranet using a history of crawl information for the items, the change period for an item being a frequency of change in content of the item;

calculating, using the search appliance, a crawl period for each of the items, the crawl period for an item being set relative to the change period for the item;

determining, using the search appliance, whether to adjust the calculated crawl period for each of the items using adjustment criteria, wherein the adjustment criteria includes a combination of an importance rank for an item and a list of hints, wherein the list of hints provides crawl periods for items that include information that match a hint from the list of hints;

adjusting the calculated crawl period for selected items determined for adjustment using the adjustment criteria, including when the selected items include information that matches a hint from the list of hints and the calculated crawl period for the selected items exceeds a threshold for the crawl period associated with the hint; and

creating, using the search appliance, a recrawl schedule of the items on the intranet using a recrawl time for each of the items, the recrawl time for each of the items being based on a last crawl time and either the calculated crawl period for the item or the adjusted crawl period for the item.

10. The non-transitory computer-readable storage device of claim 9 wherein an item includes a document.

11. The non-transitory computer-readable storage device of claim 9 further comprising instructions that, when executed, cause the one or more processors to perform the action of optionally sorting the recrawl schedule.

12. The non-transitory computer-readable storage device of claim 11 wherein sorting the recrawl schedule includes sorting the recrawl schedule such that more overdue items are recrawled before less overdue items.