Network quality control system for automatic validation of web pages and notification of author

• US 6,895,551 B1
• Filed: 09/23/1999
• Issued: 05/17/2005
• Est. Priority Date: 09/23/1999
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Sundaresan, Neelakantan, Huang, Anita Wai-Ling
• Primary Examiner(s): Shah, Sanjiv
• Assistant Examiner(s): DESAI, RACHNA SINGH
• Application Number:
  US09/404,421
• Time in Patent Office: 2,063 Days
• Field of Search: 715/513, 715/530, 709/223, 717/124, 707/3
• US Class Current: 715/205
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A computer program product is provided as a network quality control system to perform an automatic validation of all the World Wide Web pages. The quality control system aims at improving the quality of the World Wide Web by promoting widely accepted uniform standards such as HTML, and further by generating web documents that are valid and compliant with the standards according to which they are published. The quality control system actively seeks malformed web pages and corrects them in conformity with their publication standards. The quality control system also automatically determines the most probable addresses of the persons in charge of maintaining the non-conformant web documents using a probabilistic approach, notifies these persons of the non-conformity of their documents, and forwards the corrected web documents to their attention.

Claims

1. A network quality control system that performs an automatic validation of hypertext documents, comprising:

a crawling module for visiting and downloading the hypertext documents to a validation module;

the validation module identifying a hypertext standard used by the hypertext documents;

the validation module further automatically testing the validity of the hypertext documents for compliance with the identified standard, by parsing the hypertext documents and by identifying problems it encounters, wherein if a hypertext document fails the parse, the hypertext document is classified as non-conformant, otherwise, the hypertext document is classified as conformant;

a repair module for automatically correcting non-conformant hypertext documents by passing them through an editor corresponding to the hypertext standard identified by the validation module, by revising non-conformant elements in the hypertext documents;

an author search module for automatically determining a most probable reply link for the hypertext documents using a probabilistic prediction approach; and

a notification module for sending a notice to the most probable reply link determined by the author search module.

2. The network quality control system according to claim 1, further including a database;

wherein the hypertext documents include links to other documents, and the crawling module stores the links in the database; and

wherein the validation module extracts and stores metadata from the hypertext documents in the database.

3. The network quality control system according to claim 2, wherein the editor includes a HyperText Markup Language (HTML) editor.

4. The network quality control system according to claim 3, wherein the validation module includes a HTML parser that parses HTML documents.

5. The network quality control system according to claim 2, wherein the validation module includes a XML (Extensible Markup Language) parser that parses XML documents.

6. The network quality control system according to claim 2, wherein the metadata includes any one or more of:

annotation information for “

mailto”

links found within the visited document, as entered by the validation module;

a listing of errors found in non-valid pages, as entered by the validation module;

visit information for addresses of the pages visited, as entered by the crawling module;

addresses of the pages to be visited, as entered by the validation module;

hypertext link relationships among the addresses of the pages visited, as entered by the validation module;

repair information entered by the repair module;

author information entered by the author search module;

notification information sent to the authors of non-valid web pages, as entered by the notification module.

7. The network quality control system according to claim 2, wherein the author search module automatically determines the most probable reply link by searching the hypertext documents for metadata that explicitly specifies author information;

if unsuccessful, the author search module performs a probabilistic analysis of predetermined out-links contained within the hypertext documents; and

the author search module further computes and assigns a probability score to one or more links, by performing lexical and structure analyses of the links to other documents.

8. The network quality control system according to claim 7, wherein the author search module performs the probabilistic analysis by using one or more of the following probability factors:

frequency of occurrence of words and phrases in the links;

frequency of occurrence of words and phrases in the title, description or keyword metadata of the hypertext documents; and

distribution of depth from a principal hypertext document.

9. The network quality control system according to claim 8, wherein the hypertext documents include World Wide Web pages.

10. A computer program product that performs an automatic validation of hypertext documents, comprising:

a crawling module for visiting and downloading the hypertext documents to a validation module;

the validation module identifying a hypertext standard used by the hypertext documents;

the validation module further automatically testing the validity of the hypertext documents for compliance with the identified standard, by parsing the hypertext documents and by identifying problems it encounters, wherein if a hypertext document fails the parse, the hypertext document is classified as non-conformant, otherwise, the hypertext document is classified as conformant;

a repair module for automatically correcting non-conformant hypertext documents by passing them through an editor corresponding to the hypertext standard identified by the validation module, by revising non-conformant elements in the hypertext documents;

an author search module for automatically determining a most probable reply link for the hypertext documents using a probabilistic prediction approach; and

a notification module for sending a notice to the most probable reply link determined by the author search module.

11. The computer program product according to claim 10, further including a database;

wherein the hypertext documents include links to other documents, and the crawling module stores the links in the database;

wherein the validation module extracts and stores metadata from the hypertext documents in the database.

12. The computer program product according to claim 11, wherein the editor includes a HyperText Markup Language (HTML) editor;

and

wherein the validation module includes a HTML parser that parses HTML documents.

13. The computer program product according to claim 11, wherein the metadata includes any one or more of:

annotation information for “

mailto”

links found within the visited document, as entered by the validation module;

a listing of errors found in non-valid pages, as entered by the validation module;

visit information for addresses of the pages visited, as entered by the crawling module;

addresses of the pages to be visited, as entered by the validation module;

hypertext link relationships among the addresses of the pages visited, as entered by the validation module;

repair information entered by the repair module;

author information entered by the author search module;

notification information sent to the authors of non-valid web pages, as entered by the notification module.

14. The computer program product according to claim 11, wherein the author search module automatically determines the most probable reply link by searching the hypertext documents for metadata that explicitly specifies author information;

if unsuccessful, the author search module performs a probabilistic analysis of predetermined out-links contained within the hypertext documents; and

the author search module further computes and assigns a probability score to one or more links, by performing lexical and structure analyses of the links to other documents.

15. The computer program product according to claim 14, wherein the author search module performs the probabilistic analysis by using one or more of the following probability factors:

frequency of occurrence of words and phrases in the links;

frequency of occurrence of words and phrases in the title, description or keyword metadata of the hypertext documents; and

distribution of depth from a principal hypertext document.

16. A method of performing an automatic validation of hypertext documents, comprising:

a crawling module visiting and downloading the hypertext documents to a validation module;

the validation module identifying a hypertext standard used by the hypertext documents, and automatically testing the validity of the hypertext documents for compliance with the identified standard, by parsing the hypertext documents and by identifying problems it encounters, wherein if a hypertext document fails the parse, the hypertext document is classified as non-conformant, otherwise, the hypertext document is classified as conformant;

a repair module automatically correcting non-conformant hypertext documents by passing them through an editor corresponding to the hypertext standard identified by the validation module, by revising non-conformant elements in the hypertext documents;

an author search module automatically determining a most probable reply link for the hypertext documents using a probabilistic prediction approach; and

a notification module sending a notice to the most probable reply link determined by the author search module.

17. The method according to claim 16, wherein the hypertext documents include links to other documents, and further including:

storing the links in the database; and

extracting and storing metadata from the non-conformant hypertext documents in the database.

18. The method according to claim 17, wherein parsing the hypertext documents includes using a HyperText Markup Language (HTML) editor for parsing HTML documents.

19. The method according to claim 18, wherein storing metadata includes storing any one or more of the following factors:

annotation information for “

mailto”

links found within the visited document, as entered by the validation module;

a listing of errors found in non-valid pages, as entered by the validation module;

visit information for addresses of the pages visited, as entered by the crawling module;

addresses of the pages to be visited, as entered by the validation module;

hypertext link relationships among the addresses of the pages visited, as entered by the validation module;

repair information entered by the repair module;

author information entered by the author search module;

notification information sent to the authors of non-valid web pages, as entered by the notification module.

20. The method according to claim 18, wherein the author search module automatically determines the most probable reply link by searching the hypertext documents for metadata that explicitly specifies author information;

if unsuccessful, performing a probabilistic analysis of predetermined out-links contained within the hypertext documents; and

further computing and assigning a probability score to one or more links, by performing lexical and structure analyses of the links to other documents.

21. The method according to claim 20, wherein performing the probabilistic analysis includes using one or more of the following probability factors:

frequency of occurrence of words and phrases in the links;

frequency of occurrence of words and phrases in the title, description or keyword metadata of the hypertext documents; and

distribution of depth from a principal hypertext document.

22. The method according to claim 16, wherein the hypertext documents comprise XML documents;

and

further comprising parsing the XML documents.

23. A computer program product having instruction codes that performs an automatic validation of hypertext documents, comprising:

a first set of instruction codes for visiting and downloading the hypertext documents;

a second set of instruction codes identifying a hypertext standard used by the hypertext documents;

the second set of instruction codes further automatically testing the validity of the hypertext documents for compliance with the identified standard, by parsing the hypertext documents and by identifying problems it encounters, wherein if a hypertext document fails the parse, the hypertext document is classified as non-conformant, otherwise, the hypertext document is classified as conformant;

a third set of instruction codes for automatically correcting non-conformant hypertext documents by passing them through an editor corresponding to the hypertext standard identified by the second set of instruction codes, by revising non-conformant elements in the hypertext documents;

a fourth set of instruction codes for automatically determining a most probable reply link for the hypertext documents using a probabilistic prediction approach; and

a fifth set of instruction codes for sending a notice to the most probable reply link determined by the fourth set of instruction codes.

24. The computer program product according to claim 23, further comprising a database;

wherein the hypertext documents comprise links to other documents, and the first set of instruction codes stores the links in the database; and

wherein the second set of instruction codes extracts and stores metadata from the hypertext documents in the database.

25. The computer program product according to claim 24, wherein the hypertext documents comprise HTML documents;

and

wherein the second set of instruction codes parses the HTML documents.

26. The computer program product according to claim 24, wherein the metadata includes any one or more of:

annotation information for “

mailto”

links found within the visited document;

a listing of errors found in non-valid pages;

visit information for addresses of the pages visited;

addresses of the pages to be visited;

hypertext link relationships among the addresses of the pages visited;

repair information;

author information; and

notification information.

27. The computer program product according to claim 24, wherein the fourth set of instruction codes automatically determines the most probable reply link by searching the hypertext documents for metadata that explicitly specifies author information;

if unsuccessful, the fourth set of instruction codes performs a probabilistic analysis of predetermined out-links contained within the hypertext documents; and

the fourth set of instruction codes further computes and assigns a probability score to one or more links, by performing lexical and structure analyses of the links to other documents.

28. The computer program product according to claim 27, wherein the fourth set of instruction codes performs the probabilistic analysis by using one or more of the following probability factors:

frequency of occurrence of words and phrases in the links;

frequency of occurrence of words and phrases in the title, description or keyword metadata of the hypertext documents; and

distribution of depth from a principal hypertext document.

29. The method according to claim 23, wherein the hypertext documents comprise XML documents;

and

further comprising parsing the XML documents.

30. A computer program product that performs an automatic validation of hypertext documents, comprising:

means for visiting and downloading the hypertext documents;

means for identifying a hypertext standard used by the hypertext documents;

means for automatically testing the validity of the hypertext documents for compliance with the identified standard, by parsing the hypertext documents and by identifying problems it encounters, wherein if a hypertext document fails the parse, the hypertext document is classified as non-conformant, otherwise, the hypertext document is classified as conformant;

means for automatically correcting non-conformant hypertext documents by passing them through an editor corresponding to the identified hypertext standard, by revising non-conformant elements in the hypertext documents;

means for automatically determining a most probable reply link for the hypertext documents using a probabilistic prediction approach; and

means for sending a notice to the most probable reply link.