SELECTIVE CONTENT EXTRACTION

• US 20120089903A1
• Filed: 06/30/2009
• Published: 04/12/2012
• Est. Priority Date: 06/30/2009
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Liu, Sam, Atkins, Clayton, Liu, Jerry, Joshi, Parag, Xiong, Yuhong
• Granted Patent: US 9,032,285 B2
• Time in Patent Office: Days
• US Class Current: 715/234
• CPC Class Codes: G06F 16/9577 Optimising the visualization of content, e.g. distillation of HTML documents

Abstract

A method for extracting web content includes detecting, within a web page, a hierarchical structure that includes a plurality of nodes. Potential article nodes from the plurality of nodes are identified. The identified potential article node with a highest rank in the hierarchical structure is identified as an article node. Content is extracted from the article node.

Claims

1. A method for extracting web content, comprising:

detecting, within a web page, a hierarchical structure that includes a plurality of nodes;

identifying potential article nodes from the plurality of nodes;

selecting as an article node one of the identified potential article nodes with a highest rank in the hierarchical structure; and

producing content extracted from the article node.

2. The method of claim 1:

further comprising merging the content of the article node with content from other nodes at the same hierarchical level as the article node; and

wherein producing comprises producing the merged content.

3. The method of claim 2, further comprising:

identifying a node at the same hierarchical level as the article node that contains one or more of advertisement content and text content of a quantity below a threshold value, and

excluding content from the identified node from the merged content.

4. The method of claim 1, wherein identifying potential article nodes comprises identifying one or more of the plurality of nodes that contain text content of a quantity that exceeds a threshold.

5. The method of claim 1, wherein the web page is a first web page and the article node is a first article node, the method further comprising:

determining if the first web page includes a node containing content indicative of the existence of a second web page containing a second article node with content that is a continuation of the first article node content;

upon a positive determination;

detecting, within the second web page, a hierarchical structure that includes a plurality of nodes;

identifying potential article nodes from the plurality of nodes;

selecting as the second article node the identified potential article node that appears first on the hierarchical structure of the second web page; and

extracting content from the second article node; and

wherein producing comprises producing content extracted and from the first and second article nodes.

6. A computer readable medium having computer executable instructions recorded thereon, the instructions wherein executed cause a processing system to implement a method that includes:

detecting, within a web page, a hierarchical structure that includes a plurality of nodes;

identifying potential article nodes from the plurality of nodes;

selecting as an article node the identified potential article node with a highest rank in the hierarchical structure; and

producing content extracted from the article node.

7. The medium of claim 6, wherein:

the method includes merging the content of the article node with content from other nodes at the same hierarchical level as the article node; and

producing comprises producing the merged content.

8. The medium of claim 7, wherein the method includes:

identifying a node at the same hierarchical level as the article node that contains one or more of advertisement content and text content of a quantity below a threshold value, and

excluding content from the identified node from the merged content.

9. The medium of claim 6, wherein identifying potential article nodes comprises identifying one or more of the plurality of nodes that contain text content of a quantity that exceeds a threshold.

10. The medium of claim 6, wherein the web page is a first web page, the article node is a first article node, and the method includes:

determining if the first web page includes a node containing content indicative of the existence of a second web page containing a second article node with content that is a continuation of the first article node content;

upon a positive determination;

detecting, within the second web page, a hierarchical structure that includes a plurality of nodes;

identifying potential article nodes from the plurality of nodes;

selecting as the second article node the identified potential article node that appears first on the hierarchical structure of the second web page; and

extracting content from the second article node; and

wherein producing comprises producing content extracted and from the first and second article nodes.

11. A system for extracting web content, comprising:

a structure engine operable to detect, within a web page, a hierarchical structure that includes a plurality of nodes;

an article engine operable to identify potential article nodes from the plurality of nodes and to select as an article node the identified potential article node with a highest rank in the hierarchical structure; and

a production engine operable to produce content extracted from the article node.

12. The system of claim 11 wherein:

the article engine is operable to merge the content of the article node with content from other nodes at the same hierarchical level as the article node and to extract the merged content; and

the production engine is operable to produce the merged content.

13. The system of claim 12, wherein the article engine is operable to:

identify a node at the same hierarchical level as the article node that contains one or more of advertisement content and text content of a quantity below a threshold value, and

exclude content from the identified node from the merged content.

14. The system of claim 11, wherein the article engine is operable to identify potential article nodes by identifying one or more of the plurality of nodes that contain text content of a quantity that exceeds a threshold,

15. The system of claim 11, wherein the web page is first web page and the article node is a first article node and the article engine is operable to determine if the first web page includes a node containing content indicative of the existence of a second web page containing a second article node with content that is a continuation of the first article node content and upon a positive determination:

cause the structure engine to detect, within the second web page, a hierarchical structure that includes a plurality of nodes;

identify potential article nodes from the plurality of nodes of the second web page;

select as the second article node the identified potential article node that appears first on the hierarchical structure of the second web page;

cause the production engine to produce content extracted from the first and second article nodes.