Unique-change detection of dynamic web pages using history tables of signatures

• US 6,012,087 A
• Filed: 05/20/1998
• Issued: 01/04/2000
• Est. Priority Date: 01/14/1997
• Status: Expired due to Term

• Current Assignee: NetMind Technologies, Inc. (Nokia Corporation)
• Original Assignee: NetMind Technologies, Inc. (Nokia Corporation)
• Inventors: Noble, Alan C., Freivald, Matthew P.
• Primary Examiner(s): Maung, Zarni
• Assistant Examiner(s): DINH, KHANH Q
• Application Number:
  US09/081,991
• Time in Patent Office: 594 Days
• Field of Search: 709/218, 709/229, 709/224, 709/226, 709/201, 395/705, 380/21, 380/25, 380/29, 380/4, 705/8
• US Class Current: 709/218
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking

Abstract

An improved change-detection tool detects only relevant changes within Internet web pages on the world-wide-web. Changes back to an earlier version of a web page are not relevant and do not cause the user to be notified. Only changes to a new, unique version of the web page generate a user notification. After the user finishes registering the web page by specifying the URL and the user'"'"'s e-mail address, the change-detection tool periodically retrieves the web-page at the specified URL and generates a checksum or signature to determine when to send a notification to the user. Signatures from several older versions of the web page are stored in a history table. When a new signature for a re-fetched page matches the most-recent signature at the top of the stack in the history table, no change has occurred. When the new signature matched any of the older signatures in the history table, the detected change is not unique and notification is not made even though a change has occurred. When the new signature matches one of the older, not-most-recent signatures in the history table, the signature is moved into a permanent history table. Signatures in the permanent history table are for recurring versions of the web page and are likely to appear again. Error pages displayed when a web server is down for routine maintenance can be screened out using the history table. The frequency of notifications is tracked. When too many notifications are being sent for a web page, the last-modified header is used rather than signature-matching to reduce the frequency of notifications.

Claims

1. A change-detection web server for detecting unique changes in web pages comprising:

a network connection for transmitting and receiving packets from a remote client and a remote web-page server;

a responder, coupled to the network connection, for communicating with the remote client, the responder registering a web page for change detection by receiving from the remote client a uniform-resource-locator (URL) identifying the web page, the responder fetching the web page from the remote web-page server;

a database, coupled to the responder, for receiving the URL from the responder when the web page is registered by the remote client, the database for storing a plurality of records each containing a URL;

a history table in each of the records in the database, the history table for storing a most-recent signature and a plurality of older-version signatures for a registered web page identified by the URL, the older-version signatures being condensed checksums for earlier versions of the registered web page previously fetched by the change-detection web server, whereas the most-recent signature is a condensed checksum for a most-recently-fetched copy of the registered web page; and

a periodic minder, coupled to the database and the network connection, for periodically re-fetching the web page from the remote web-page server by transmitting the URL from the database to the network connection, the periodic minder receiving a fresh copy of the web page from the remote web-page server, the periodic minder generating a new signature from the fresh copy of the web page, the periodic minder notifying the remote client of a unique change when the new signature does not match the most-recent signature and does not match any of the older-version signatures in the record,

wherein the unique change in the web page is detected by comparing the new signature to the most-recent signature and to older-version signatures for the web page, wherein changes in the web page which are not unique but match an earlier version of the web page do not notify the remote client.

2. The change-detection web server of claim 1 wherein the database does not store the web page, the database storing the most-recent signature and earlier-version signatures for the web page,

whereby storage requirements for the database are reduced by archiving the most-recent signature and not entire web pages.

3. The change-detection web server of claim 2 further comprising:

a permanent history table, the permanent history table for storing new signatures that match one of the older-version signatures,

wherein older-version signatures that are matched are copied to the permanent history table.

4. The change-detection web server of claim 3 wherein the history table is a temporary history table organized as a first-in-first-out stack, wherein a least-recent signature in the history table is replaced by a new signature when notification is made,

wherein signatures in the permanent history table are not deleted by new signatures written to the temporary history table.

5. The change-detection web server of claim 4 wherein the older-version signatures are stored in both the permanent history table and the history table, the periodic minder comparing the new signature to older-version signatures from both the history table and from the permanent history table.

6. The change-detection web server of claim 2 wherein each record in the database further comprises:

a last-modified field for storing a last-modified date and time contained within the most-recently-fetched copy of the registered web page;

wherein the periodic minder further comprises;

last-modified parsing means, receiving the fresh copy of the web page, for parsing the web page for an HTTP response from the Web server for a last-modified header for indicating a date and time that the web page was last modified;

date compare means, receiving the last-modified header and the last-modified field, for generating a date-match signal before the periodic minder notifies the remote client when the last-modified header from the fresh copy of the web page matches the last-modified field from the database;

wherein the last-modified header is stored in the database for each record.

7. The change-detection web server of claim 6 further comprising:

re-fetch means, responsive to the date-match signal from the date compare means, for fetching another fresh copy of the web page when the last-modified header from the fresh copy of the web page matches the last-modified field from the database but the new signature does not match;

the periodic minder generating a second new signature for the another fresh copy;

the periodic minder waiting for the second new signature before notifying the remote client of a unique change when the second new signature does not match the most-recent signature and does not match any of the older-version signatures in the record,

wherein the web page is re-fetched before notification when the date-match signal is generated to prevent false notifications due to a network error when fetching the web page.

8. The change-detection web server of claim 7 wherein each record in the database further comprises:

a detect-tracker field for storing a number of notifications sent to the remote client,

an ignore-signature flag for indicating when too many notifications were sent to the remote client.

9. The change-detection web server of claim 8 further comprising:

increment means for incrementing the detect-tracker field in the database when notification is sent to the remote client.

10. The change-detection web server of claim 9 further comprising:

frequency check means, responsive to a threshold value, for comparing the detect-tracker field to the threshold value and setting the ignore-signature flag when the detect-tracker field exceeds the threshold value,

wherein the ignore-signature flag is set when the number of notifications exceeds the threshold value.

11. The change-detection web server of claim 10 wherein the frequency check means only sets the ignore-signature flag when the web server returns the last-modified header, whereas web pages without a last-modified header cannot have their ignore-signature flags set.

12. The change-detection web server of claim 11 further comprising:

disable means, responsive to the ignore-signature flag, for disabling the periodic minder from comparing signatures, whereas notification is only made when the date-match signal is not generated by the date compare means,

wherein the ignore-signature flag disables signature comparison for web pages with a number of notifications that exceeds the threshold value.

13. The change-detection web server of claim 2 wherein the web page is a hyper-text markup-language (HTML) web page containing HTML headers, the HTML headers for indicating formatting, layout, and hyper-links specifying URLs of other servers.

14. The change-detection web server of claim 2 further comprising:

mailer means, coupled to the network connection, for sending a change notification message to the remote client when the unique change is signaled,

wherein the responder receives an electronic-mail address from the remote client, the responder storing the electronic-mail address of the remote client in the database, and the mailer means reading the electronic-mail address from the database, the change notification message being sent to the remote client as an electronic-mail message addressed to the electronic-mail address,

wherein the remote client is notified of the unique change by electronic mail.

15. The change-detection web server of claim 14 wherein packets transmitted to the network connection are TCP/IP packets and wherein the remote client and the remote web-page server are on the Internet.

16. A computer-implemented method for detecting a unique change in a web page and notifying a user of the unique change, the method comprising the steps of:

registering the web page by;

receiving an address of the user and a locator for the web page;

fetching the web page from a remote server by transmitting the locator to a network server;

generating a signature for the web page;

storing the signature in a history table for storing signatures for older versions of the web page, and storing the locator, the address of the user in a database;

after a period of time, periodically checking to determine if a unique change has occurred by;

reading the locator from the database and transmitting the locator to remote server to fetch a recent copy of the web page;

generating a new signature for the recent copy of the web page;

signaling that a unique change is detected when the new signature does not match any of the signatures in the history table including signatures for the older versions of the web page;

wherein the web page is not stored in the database which stores the history table of signatures for the web page.

17. The computer-implemented method of claim 16 further comprising the step of:

writing the new signature to the history table and deleting a least-recent signature in the history table while retaining other signatures in the history table,

wherein the history table is a stack of signatures of recent versions of the web page.

18. A computer-program product comprising:

a computer-usable medium having computer-readable program code means embodied therein for detecting unique changes in a web page, the computer-readable program code means in the computer-program product comprising;

network connection means for transmitting and receiving packets from a remote client and a remote web page server;

responder means, coupled to the network connection means, for communicating with the remote client, the responder means registering web pages for change detection by receiving from the remote client a uniform-resource-locator (URL) identifying the web page;

history-table means for storing a most-recent signature and a plurality of older-version signatures for a registered web page identified by the URL, the older-version signatures being condensed checksums for earlier versions of the registered web page previously fetched for change detection, whereas the most-recent signature is a condensed checksum for a most-recently-fetched copy of the registered web page;

database means, coupled to the responder means, for receiving the URL from the responder means when the web page is registered by the remote client, the database means coupled to the history-table means, the database means for storing a plurality of records each containing a URL and a history table for a registered web page, the database means not storing the web page or the registered web pages, the database means storing condensed signatures for the web page; and

periodic minder means, coupled to the database means and the network connection means, for periodically re-fetching the web page from the remote web page server by transmitting the URL from the database means to the network connection means, the periodic minder means receiving a fresh copy of the web page from the remote web page server, the periodic minder means generating a new signature from the fresh copy of the web page, the periodic minder means notifying the remote client of a unique change when the new signature does not match the most-recent signature and does not match any of the older-version signatures in the record,

wherein the unique change in the web page is automatically detected by periodically re-fetching the web page, generating the new signature, and comparing to signatures in the history table.

19. The computer-program product of claim 18 wherein the history-table means includes permanent signatures and temporary signatures, wherein when the new signature matches one of the older-version signatures the new signature becomes a permanent signature that is not replaced by other new signatures.

20. The computer-program product of claim 18 wherein the database means stores for each record an actual content-length of the registered web page and a content-length header within the registered web page.