System and method for providing a change profile of a web page

• US 20080104502A1
• Filed: 10/26/2006
• Published: 05/01/2008
• Est. Priority Date: 10/26/2006
• Status: Abandoned Application

• Current Assignee: Oath Inc. (Verizon Communications Inc.)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Olston, Christopher
• Application Number:
  US11/588,019
• Publication Number:
  US 20080104502A1
• Time in Patent Office: Days
• US Class Current: 715/229
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9574 of access to content, e.g. by caching

Abstract

An improved system and method is provided for adaptively refreshing a web page. A base version of the web page may be partitioned into a collection of fragments. Then the collection of fragments may be compared with the corresponding fragments of a recent version of the web page to determine a divergence measurement of the difference between the base version and the recent version of the web page. The divergence measurement may be recorded in a change profile representing a change history of the web page that includes a sequence of numeric pairs indicating a time offset and a divergence measurement of the difference between a version of the web page at the time offset and a base version of the web page. The refresh period for the web page may be adjusted by applying an adaptive refresh policy using the divergence measurements recorded in the change profile.

Claims

1. A computer system for providing a change history of a web page, comprising:

a change profile of a web page representing a change history of the web page, the change profile including one or more divergence measurements, each divergence measurement indicating a numeric difference between versions of the web page; and

a change profile manager operably coupled to the change profile for maintaining the change profile of the web page.

2. The system of claim 1 further comprising a web page fragmentor operably coupled to the change profile manager for partitioning the web page into a collection of fragments.

3. The system of claim 2 further comprising a fragment comparator operably coupled to the web page fragmentor for determining the one or more divergence measurements for the web page by comparing two collections of fragments, each of the collections representing different versions of the web page.

4. A computer-readable medium having computer-executable components comprising the system of claim 1.

5. A computer-implemented method for providing a change history of a web page, comprising:

receiving a base version of a web page;

partitioning the base version of the web page into a collection of fragments; and

creating a change profile including a summary of the collection of fragments of the web page.

6. The method of claim 5 wherein partitioning the base version of the web page into a collection of fragments comprises partitioning a sequence of characters from the web page into a set of fixed length sequences of characters appearing in the web page to produce the collection of fragments.

7. The method of claim 5 wherein partitioning the base version of the web page into a collection of fragments comprises partitioning a sequence of words from the web page into a set of shingles to produce the collection of fragments.

8. The method of claim 5 wherein partitioning the base version of the web page into a collection of fragments comprises defining fragments based on fixed-size subtrees of a tree representing the web page to produce the collection of fragments.

9. The method of claim 5 further comprising updating the change profile by recording an entry for one or more subsequent versions of the web page, each entry including a a pair indicating a time and a divergence measurement of the difference between versions of the web page.

10. The method of claim 5 wherein creating a change profile including a summary of the collection of fragments of the web page comprises creating a change profile including a pair indicating a time and a divergence measurement representing a base measurement of the base version of the web page.

11. The method of claim 9 wherein recording an entry for one or more subsequent versions of the web page, each entry including a pair indicating a time and a divergence measurement of the difference between versions of the web page comprises determining the time based on a refresh time for the web page.

12. A computer-readable medium having computer-executable instructions for performing the method of claim 5.

13. A computer system for providing a change history of a web page, comprising:

means for receiving a subsequent version of a web page;

means for partitioning the subsequent version of the web page into a collection of fragments; and

means for updating a change profile including a divergence measurement indicating a difference between versions of the web page.

14. The computer system of claim 13 wherein means for partitioning the subsequent version of the web page into the collection of fragments comprises means for partitioning a sequence of words from the subsequent version of the web page to produce the collection of fragments.

15. The computer system of claim 13 wherein means for updating the change profile including the divergence measurement indicating the difference between versions of the web page comprises means for extending a sequence of pairs indicating a time and the divergence measurement indicating the difference between versions of the web page.

16. The computer system of claim 13 further comprising means for determining the divergence measurement between the subsequent version of the web page and a base version of the web page.

17. The computer system of claim 16 further comprising means for including the divergence measurement between the subsequent version of the web page and the base version of the web page in the change profile.

18. The computer system of claim 16 wherein means for determining the divergence measurement between the subsequent version of the web page and a base version of the web page comprises means for comparing the difference between the collection of fragments of the subsequent version of the web page and another collection of fragments of the base version of the web page.

19. The computer system of claim 13 further comprising means for creating a change profile including a summary of another collection of fragments of a base version of the web page.

20. The computer system of claim 19 wherein means for creating the change profile including the summary of the another collection of fragments of the base version of the web page comprises means for creating the change profile including a pair indicating a time and a base measurement of the base version of the web page.