System and method for detecting a web page

• US 20080275901A1
• Filed: 05/04/2007
• Published: 11/06/2008
• Est. Priority Date: 05/04/2007
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Ravikumar, Shanmugasundaram, Chakrabarti, Deepayan, Punera, Kunal
• Granted Patent: US 7,987,417 B2
• Time in Patent Office: Days
• US Class Current: 707/102
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

An improved system and method is provided for detecting a web page template. A web page template detector may be provided for performing page-level template detection on a web page. In general, the web page template classifier may be trained using automatically generated training data, and then the web page template classifier may be applied to web pages to identify web page templates. A web page template may be detected by classifying segments of a web page as template structures, by assigning classification scores to the segments of the web page classified as template structures, and then by smoothing the classification scores assigned to the segments of the web page. Generalized isotonic regression may be applied for smoothing scores associated with the nodes of a hierarchy by minimizing an optimization function using dynamic programming.

Claims

1. A computer system for detecting a template, comprising:

a web page template detector for performing page-level template detection on a web page;

a web page template classifier operably coupled to the web page template detector for identifying a web page template of the web page; and

a storage operably coupled to the web page template detector for storing a plurality of templateness scores assigned by the web page template classifier for nodes of the web page.

2. The system of claim 1 wherein the storage further comprises a plurality of DOM nodes representing the web page.

3. The system of claim 1 further comprising an isotonic smoothing engine operably coupled to the web page template detector for smoothing the plurality of templateness scores using generalized isotonic regression on a tree.

4. A computer-readable medium having computer-executable components comprising the system of claim 1.

5. A computer-implemented method for detecting a template, comprising:

receiving a plurality of web pages;

automatically generating training data; and

training a page-level classifier to identify a web page template.

6. The method of claim 5 wherein automatically generating training data comprises identifying features of nodes of the web pages that may be indicative that a node may be a template.

7. The method of claim 5 further comprising applying the web page template classifier to web pages to identify web page templates.

8. The method of claim 5 wherein automatically generating training data comprises using site-level template detection.

9. The method of claim 5 wherein applying the web page template classifier comprises extracting features of nodes of the web pages that may be indicative that a node may be a template.

10. The method of claim 5 wherein applying the web page template classifier comprises assigning templateness scores to the nodes of the web page.

11. The method of claim 10 wherein assigning templateness scores to the nodes of the web page comprises applying classifiers trained over a set of features indicative that a node may be a template.

12. The method of claim 5 wherein applying the web page template classifier comprises smoothing templateness scores assigned to nodes of the web page.

13. The method of claim 11 wherein smoothing templateness scores assigned to nodes of the web page comprises smoothing templateness scores using generalized isotonic regression on a tree.

14. A computer-readable medium having computer-executable instructions for performing the method of claim 5.

15. A computer system for detecting a template, comprising:

means for receiving a web page;

means for classifying segments of the web page as template structures;

means for assigning classification scores to the segments of the web page classified as the template structures; and

means for smoothing classification scores assigned to the segments of the web page.

16. The method of claim 15 further comprising means for training a page-level classifier to identify a template structure.

17. The method of claim 15 further comprising means for automatically generating training data for training a page-level classifier to identify a template structure.

18. The computer system of claim 15 wherein means for classifying segments of the web page as template structures comprises means for preprocessing the web page in order to extract features for identifying the template structures.

19. The computer system of claim 15 wherein means for assigning classification scores to the segments of the web page classified as the template structures comprises means for training classifiers over a set of features for identifying the template structures.

20. The computer system of claim 15 wherein means for smoothing classification scores comprises means for applying generalized isotonic regression on a tree.