Change-detection tool indicating degree and location of change of internet documents by comparison of cyclic-redundancy-check(CRC) signatures

• US 5,898,836 A
• Filed: 01/14/1997
• Issued: 04/27/1999
• Est. Priority Date: 01/14/1997
• Status: Expired due to Term

• Current Assignee: NetMind Technologies, Inc. (Nokia Corporation)
• Original Assignee: Netmind Services Inc.
• Inventors: Noble, Alan C., Freivald, Matthew P., Richards, Mark S.
• Primary Examiner(s): Barry, Lance Leonard
• Application Number:
  US08/783,625
• Time in Patent Office: 833 Days
• Field of Search: 395/200.48, 395/400.49, 707/10, 707/203, 707/511, 707/513
• US Class Current: 709/218
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users

Abstract

A change-detection web server automatically checks web-page documents for recent changes. The server retrieves and compares documents one or more times a week. The user is notified by electronic mail when a change is detected. The user registers a web-page document by submitting his e-mail address and the uniform-resource locator (URL) of the desired document. The document is fetched and the user can select text on the page of interest. Non-selected text is ignored; only changes in the selected text are reported back to the user. Thus changes to less relevant parts of the document are ignored. The document is divided into sections bounded by hyper-text markup-language (HTML) tags. A checksum is generated and stored for each HTML-bound section. Storage requirements are reduced since only checksums are stored rather than the original documents. During periodic comparisons a fresh copy of the document is retrieved, divided into HTML-bound sections and checksums generated for each section. The freshly-generated checksums are compared to the archived checksums. Sections with non-matching checksums are highlighted as changed, and the percentage of changed sections is reported. The user-defined selection is also stored as a checksum and compared to a freshly-generated checksum. Changed checksums outside the user-defined selection do not generate a change notification. Re-ordering of sections does not generate a change notification when the checksums otherwise match. Thus format and layout changes do not generate change notifications, and the frequency of notices to user is reduced.

Claims

1. A change-detection web server comprising:

a network connection for transmitting and receiving packets from a remote client and a remote document server;

a responder, coupled to the network connection, for communicating with the remote client, the responder registering a document for change detection by receiving from the remote client a uniform-resource-locator (URL) identifying the document, the responder fetching the document from the remote document server and generating an original checksum for a checked portion of the document, the checked portion being less than the entire document;

a database, coupled to the responder, for receiving the URL and the original checksum from the responder when the document is registered by the remote client, the database for storing a plurality of records each containing a URL and a checksum for a registered document;

a periodic minder, coupled to the database and the network connection, for periodically re-fetching the document from the remote document server by transmitting the URL from the database to the network connection, the periodic minder receiving a fresh copy of the document from the remote document server, the periodic minder generating a fresh checksum of a portion of the fresh copy of the document and comparing the fresh checksum to the original checksum, the periodic minder signaling a detected change to the remote client when the fresh checksum does not match the original checksum;

whereby a change in the document is detected by comparing a checksum for the checked portion of the document, wherein changes in portions of the document outside the checked portion are not signaled to the remote client.

2. The change-detection web server of claim 1 wherein the database does not store the document, the database storing a checksum for the document,

whereby storage requirements for the database are reduced by archiving checksums and not entire documents.

3. The change-detection web server of claim 2 further comprising:

selection means, coupled to the responder, for receiving a selection from the remote client, the selection identifying boundaries of the checked portion of the document;

parsing means, coupled to the periodic minder, for parsing the fresh copy and generating checksums for a plurality of portions of the fresh copy;

compare means, coupled to the parsing means, for signaling a match when any of the checksums generated by the parsing means matches the original checksum from the database;

whereby a change in the document is detected when the match is not signaled by the compare means, the parsing means generating a plurality of checksums for the plurality of portions of the fresh copy.

4. The change-detection web server of claim 3 wherein the database further comprises:

a length field for indicating a size of the checked portion, the length field written by the selection means,

the parsing means generating each checksum for portions having the size of the checked portion,

whereby the size of the checked portion is stored and used by the parsing means.

5. The change-detection web server of claim 1 wherein the document is a hyper-text markup-language (HTML) document containing HTML tags, the HTML tags for indicating formatting, layout, and hyper-links specifying URLs of other servers, the change-detection web server further comprising:

divider means, coupled to the responder, for dividing the document into portions bound by the HTML tags;

checksum means for generating original checksums, an original checksum generated for each portion bound by HTML tags;

the database storing the original checksums for the portions bound by the HTML tags;

the periodic minder further comprising;

second divider means for dividing the fresh copy of the document into portions bound by the HTML tags;

second checksum means for generating fresh checksums for portions of the fresh copy bound by HTML tags in the fresh copy of the document;

compare means, receiving the fresh checksums of the fresh copy from the second checksum means, for comparing the fresh checksums to the original checksums from the database;

report means for signaling a change in the document when an original checksum for the document has no matching fresh checksum;

whereby checksums are generated and stored for portions of the document bound by the HTML tags.

6. The change-detection web server of claim 5 wherein the report means further comprises:

mailer means, coupled to the network connection, for sending a change notification message to the remote client when the change is signaled,

wherein the responder receives an electronic-mail address from the remote client, the responder storing the electronic-mail address of the remote client in the database, and the mailer means reading the electronic-mail address from the database, the change notification message being sent to the remote client as an electronic-mail message addressed to the electronic-mail address,

whereby the remote client is notified of the change by electronic mail.

7. The change-detection web server of claim 6 further comprising:

change statistics generator, coupled to the compare means, for counting a total number of portions in the document and for determining a number of original checksums without matching fresh checksums, the change statistics generator coupled to the mailer means to include in the electronic-mail message an indication of a degree of changes in the document,

wherein the degree of changes is determined for the document and included in the electronic-mail message to the remote client when a change is detected.

8. The change-detection web server of claim 7 wherein the degree of changes in the document is the number of original checksums without matching fresh checksums divided by the total number of portions in the document,

whereby the degree of change reported to the remote client indicates a fraction of portions of the document which have changed.

9. The change-detection web server of claim 7 further comprising:

highlighting means, coupled to the mailer means, for attaching the fresh copy of the document to the electronic-mail message, the fresh copy having highlighting marks inserted to indicate which portions of the document have mismatching checksums,

whereby the fresh copy of the document is highlighted to indicate changes to the remote client.

10. The change-detection web server of claim 9 wherein the packets transmitted to the network connection are TCP/IP packets and wherein the remote client and the remote document server are on the Internet.

11. A computer-implemented method for detecting recent changes in a document and notifying a user of the recent changes, the method comprising the steps of:

registering the document by receiving an address of the user and a locator for the document;

fetching the document from a remote server by transmitting the locator to a network server;

determining when the document is a hyper-text markup-language (HTML) document;

when the document is an HTML document;

dividing the document into sections, each section beginning and ending with an HTML tag, the HTML tag not directly visible to a user viewing the document on a browser;

generating a cyclical-redundancy-checksum (CRC) for each section of the document;

storing the CRC generated for each section of the document in a database together with the locator of the document and the address of the user;

after a period of time;

reading the locator from the database and transmitting the locator to remote server to fetch a recent copy of the document;

when the document is an HTML document;

dividing the recent copy of the document into sections, each section beginning and ending with an HTML tag;

generating a recent cyclical-redundancy-checksum (CRC) for each section of the recent copy of the document;

reading the CRC'"'"'s from the database and comparing the CRC'"'"'s to the recent CRC'"'"'s to determine which CRC'"'"'s from the database do not have a matching recent CRC;

signaling that a change is detected when a CRC'"'"'s from the database does not have a matching recent CRC;

whereby the document is not stored in the database which stored CRC'"'"'s for HTMMLbound sections of HTML documents.

12. The computer-implemented method of claim 11 wherein the step of signaling that a change is detected comprises:

reading the address of the user from the database and sending a message to the address of the user stating that a change has occurred,

whereby the user is notified by a message when a change is detected.

13. The computer-implemented method of claim 12 wherein the step of signaling that a change is detected further comprises:

including an indication of a degree of change in the message to the user, the degree of change for the document being a function of a number of CRC'"'"'s from the database that do not have a matching recent CRC,

whereby the message to the user indicates the degree of change to the document.

14. The computer-implemented method of claim 13 wherein the degree of change is expressed as the number of CRC'"'"'s from the database that do not have a matching recent CRC, as a percentage of a total number of CRC'"'"'s for the document,

whereby the percentage of change of the document is sent to the user in the message.

15. The computer-implemented method of claim 12 wherein the document is a web-page document on the world-wide web and the locator is a uniform-resource locator (URL).

16. The computer-implemented method of claim 12 wherein the period of time is about a week.

17. The computer-implemented method of claim 16 wherein the HTML tags are not included when generating the CRC'"'"'s,

whereby formatting changes embedded in the HTML tags do not signal a change, reducing occurrences of change notifications when only minor formatting changes occur to the document.

18. A computer-program product comprising:

a computer-usable medium having computer-readable program code means embodied therein for detecting changes in a document, the computer-readable program code means in the computer-program product comprising;

network connection means for transmitting and receiving packets from a remote client and a remote document server;

responder means, coupled to the network connection means, for communicating with the remote client, the responder means registering documents for change detection by receiving from the remote client a uniform-resource-locator (URL) identifying the document, the responder means fetching the document from the remote document server and generating an original checksum for a checked portion of the document, the checked portion being less than an entire document;

database means, coupled to the responder means, for receiving the URL and the original checksum from the responder means when the document is registered by the remote client, the database means for storing a plurality of records each containing a URL and a checksum for a registered document, the database means not storing the document or the registered documents, the database means storing a checksum for the document;

periodic minder means, coupled to the database means and the network connection means, for periodically re-fetching the document from the remote document server by transmitting the URL from the database means to the network connection means, the periodic minder means receiving a fresh copy of the document from the remote document server, the periodic minder means generating a fresh checksum of a portion of the fresh copy of the document and comparing the fresh checksum to the original checksum, the periodic minder means signaling a detected change to the remote client when the fresh checksum does not match the original checksum;

whereby a change in the document is detected by comparing a checksum for the checked portion of the document, wherein changes in portions of the document outside the checked portion are not signaled to the remote client and whereby storage requirements for the database means are reduced by archiving checksums and not entire documents.

19. The computer-program product of claim 18 wherein the computer-readable program code means further comprises:

selection means, coupled to the responder means, for receiving a selection from the remote client, the selection identifying boundaries of the checked portion of the document;

parsing means, coupled to the periodic minder means, for parsing the fresh copy and generating checksums for a plurality of portions of the fresh copy;

compare means, coupled to the parsing means, for signaling a match when any of the checksums generated by the parsing means matches the original checksum from the database means;

whereby a change in the document is detected when the match is not signaled by the compare means, the parsing means generating a plurality of checksums for the plurality of portions of the fresh copy.