Method and system for classifying display pages using summaries

• US 7,392,474 B2
• Filed: 04/30/2004
• Issued: 06/24/2008
• Est. Priority Date: 04/30/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ma, Wei-Ying, Chen, Zheng, Zhang, Benyu, Shen, Dou, Zeng, Hua-Jun
• Primary Examiner(s): Hutton; Doug
• Assistant Examiner(s): Blackwell; James H
• Application Number:
  US10/836,319
• Publication Number:
  US 20050246410A1
• Time in Patent Office: 1,516 Days
• Field of Search: 715/531, 715/205, 715/209, 715/234, 715/254, 715/276, 715/277, 707 1- 5
• US Class Current: 715/254
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method and system for classifying display pages based on automatically generated summaries of display pages. A web page classification system uses a web page summarization system to generate summaries of web pages. The summary of a web page may include the sentences of the web page that are most closely related to the primary topic of the web page. The summarization system may combine the benefits of multiple summarization techniques to identify the sentences of a web page that represent the primary topic of the web page. Once the summary is generated, the classification system may apply conventional classification techniques to the summary to classify the web page. The classification system may use conventional classification techniques such as a Naïve Bayesian classifier or a support vector machine to identify the classifications of a web page based on the summary generated by the summarization system.

Claims

1. A method in a computer system for classifying web pages, the method comprising:

retrieving a web page;

automatically generating a summary of the retrieved web page by

identifying objects of the web page, the objects having sentences;

building a term frequency by inverted document frequency index for each object;

calculating similarity between pairs of objects based on the term frequency by inverted document frequency indexes of the objects;

when the calculated similarity between a pair of objects satisfies a similarity threshold, linking the pair objects to indicate that the objects satisfy the threshold;

selecting as a core object of the web page the object that has the most links;

assigning high scores to sentences of the core object and to objects with links to the core object and low scores to all other sentences;

selecting sentences to form the summary of the web page based on the assigned scores; and

determining a classification for the retrieved web page based on the automatically generated summary.

2. The method of claim 1 wherein the automatically generating of the summary includes calculating a score for each sentence of the web page using multiple summarization techniques.

3. The method of claim 2 wherein the score for each sentence is a linear combination of the scores of the multiple summarization techniques.

4. The method of claim 1 wherein the sentences with the top scores are selected to form the summary.

5. The method of claim 2 wherein the summarization techniques include a Luhn summarization technique, a latent semantic analysis summarization technique, a content body summarization technique, and a supervised summarization technique.

6. The method of claim 2 wherein the summarization techniques include any two or more of the set of summarization techniques consisting of a Luhn summarization technique, a latent semantic analysis summarization technique, a content body summarization technique, and a supervised summarization technique.

7. The method of claim 1 wherein the determining of the classification uses a Naï

ve Bayesian classifier.

8. The method of claim 1 wherein the determining of the classification uses a support vector machine.

9. The method of claim 1 wherein the automatically generating of a summary uses a Luhn summarization technique.

10. The method of claim 1 wherein the automatically generating of a summary uses a latent semantic analysis summarization technique.

11. The method of claim 1 wherein the automatically generating of a summary uses a content body summarization technique.

12. The method of claim 1 wherein the automatically generating of a summary uses a supervised summarization technique.

13. A method in a computer system for summarizing a web page, the method comprising:

retrieving the web page;

for each sentence of the retrieved web page,

assigning a score to the sentence based on multiple summarization techniques wherein one of the summarization techniques is

identifying objects of the web page, the objects having sentences;

building a term frequency by inverted document frequency index for each object;

calculating similarity between pairs of objects based on the term frequency by inverted document frequency indexes of the objects;

when the calculated similarity between a pair of objects satisfies a similarity threshold, linking the pair of objects to indicate that the objects satisfy the threshold;

selecting as a core object of the web page the object that has the most links; and

assigning a high score to sentences of the core object and to objects with links to the core object and a low score to all other sentences; and

combining the scores assigned to the sentence to generate a combined score for the sentence; and

selecting the sentences with the highest combined scores to form a summary of the retrieved web page.

14. The method of claim 13 wherein the combined score for each sentence is a linear combination of the assigned scores.

15. The method of claim 14 wherein the assigned scores of the multiple summarization techniques are weighted differently when combining.

16. The method of claim 13 wherein the summarization techniques include a Luhn summarization technique, a latent semantic analysis summarization technique, a content body summarization technique, and a supervised summarization technique.

17. The method of claim 13 wherein the summarization techniques include any two or more of the set of summarization techniques consisting of a Luhn summarization technique, a latent semantic analysis summarization technique, a content body summarization technique, and a supervised summarization technique.

18. The method of claim 13 wherein a summarization technique is a Luhn summarization technique in which a classification has a collection of significant words.

19. The method of claim 18 wherein noisy words are discarded from a collection.

20. The method of claim 13 wherein a summarization technique is a supervised summarization technique in which a sentence is represented by a set of features that includes a feature based on similarity between the sentence and metadata of the web page.

21. The method of claim 13 wherein a summarization technique is a supervised summarization technique in which a sentence is represented by a set of features that includes a feature based on words of the sentence that are highlighted in the web page.

22. The method of claim 13 wherein a summarization technique is a supervised summarization technique in which a sentence is represented by a set of features that includes a feature based on font size of words in the sentence.

23. The method of claim 13 including identifying a classification for the retrieved web page based on the summary of the retrieved web page.

24. The method of claim 23 wherein the identifying of the classification uses a Naï

ve Bayesian classifier.

25. The method of claim 23 wherein the identifying of the classification uses a support vector machine.

26. A computer-readable storage medium containing instructions for causing a computer system to generate a summary for a display page by a method comprising:

for each sentence of the display page, generating a score that is based on multiple summarization techniques wherein one of the summarization techniques is

calculating similarity between pairs of objects of the display page, the objects having sentences;

when the calculated similarity between a pair of objects satisfies a similarity threshold, linking the pair of objects to indicate that the objects satisfy the threshold;

selecting as a core object of the display page the object that has the most links; and

assigning high score to sentences of the core object and to objects with links to the core object and low score to all other sentences; and

selecting the sentences with the highest generated scores to form a summary of the display page.

27. The computer-readable medium of claim 26 wherein the generated score for each sentence is a combination of a score for each of the multiple summarization techniques.

28. The computer-readable medium of claim 27 wherein the scores of the multiple summarization techniques are weighted differently.

29. The computer-readable medium of claim 26 wherein the summarization techniques include a Luhn summarization technique, a latent semantic analysis summarization technique, a content body summarization technique, and a supervised summarization technique.

30. The computer-readable medium of claim 26 wherein the summarization techniques include any two or more of the set of summarization techniques consisting of a Luhn summarization technique, a latent semantic analysis summarization technique, a content body summarization technique, and a supervised summarization technique.

31. The computer-readable medium of claim 26 wherein a summarization technique is a Luhn summarization technique in which a classification has a collection of significant words.

32. The computer-readable medium of claim 31 wherein noisy words are discarded from the collection.

33. The computer-readable medium of claim 26 wherein a summarization technique is a supervised summarization technique in which a sentence is represented by a set of features that includes a feature based on similarity between the sentence and metadata of the display page.

34. The computer-readable medium of claim 26 wherein a summarization technique is a supervised summarization technique in which a sentence is represented by a set of features that includes a feature based on words of the sentence that are highlighted in the display page.

35. The computer-readable medium of claim 26 wherein a summarization technique is a supervised summarization technique in which a sentence is represented by a set of features that includes a feature based on font size of words in the sentence.

36. The computer-readable medium of claim 26 including identifying a classification for the display page based on the summary of the display page.

37. A computer system embodied on a computer-readable storage medium for classifying display pages, comprising:

means for automatically generating a summary of the display page by

calculating similarity between pairs of objects of the display page, the objects having sentences;

when the calculated similarity between a pair of objects satisfies a similarity threshold, linking the pair of objects to indicate that the objects satisfy the threshold;

selecting as a core object of the display page the object that has the most links; and

selecting sentences of the core object and objects with links to the core object to form the summary of the display page; and

means for identifying a classification for the display page based on the automatically generated summary.

38. The computer system of claim 37 wherein the means for automatically generating the summary calculates a score for each sentence of the display page using multiple summarization techniques.

39. The computer system of claim 38 wherein the score for each sentence is a linear combination of the scores of the multiple summarization techniques.

40. The computer system of claim 37 wherein the summarization techniques include a Luhn summarization technique, a latent semantic analysis summarization technique, a content body summarization technique, and a supervised summarization technique.

41. The computer system of claim 37 wherein the summarization techniques include any two or more of the set of summarization techniques consisting of a Luhn summarization technique, a latent semantic analysis summarization technique, a content body summarization technique, and a supervised summarization technique.

42. The computer system of claim 41 wherein each sentence of the display page is assigned a score that is a combination of the scores of the multiple summarization techniques.