Information block extraction apparatus and method for Web pages

• US 20050066269A1
• Filed: 09/17/2004
• Published: 03/24/2005
• Est. Priority Date: 09/18/2003
• Status: Abandoned Application

• Current Assignee: Fujitsu Limited, Nanjing University
• Original Assignee: Fujitsu Limited, Nanjing University
• Inventors: Wang, Jicheng, Wu, Gangshan, Wang, Jun, Tsuda, Hiroshi
• Application Number:
  US10/943,157
• Publication Number:
  US 20050066269A1
• Time in Patent Office: Days
• US Class Current: 715/234
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 40/131 Fragmentation of text files, e.g. creating reusable text-blocks; Linking to fragments, e.g. using XInclude; Namespaces
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A method and apparatus for identifying coherent areas within a Web page. First, a Web page is parsed into an HTML DOM tree and an HTML tag token stream. Next, repeated-patterns are induced from the Web page. After filtering out improper repeated-patterns and generating corresponding instances of the repeated-patterns, the repeated-patterns are mapped back to corresponding regions in the Web page. Based on the mappings, a hierarchical RST tree containing information blocks is generated. Information items within the information blocks are detected then used to generate a hierarchical structural information block tree. Information blocks from the structural information block tree are then classified into text information blocks and link information blocks. Based on the classification and block semantic similarity, the bocks are clustered then grouped into semantic information blocks. The semantic information blocks contain main text information blocks and related link blocks which, if necessary, can be labeled.

Claims

1. A method for segmenting a Web page into information blocks with coherent contents comprising:

generating a structural information block tree of the Web page;

clustering and merging the structural information blocks; and

labeling the semantic of the resulting blocks.

2. The method of claim 1, wherein generating a structural information block tree comprises:

inducing repeated-patterns within the Web page;

matching the repeated-pattern and the corresponding region in the Web page;

constructing an RST tree (Root of the Smallest Subtree) according to the regions;

identifying information items within each information block; and

constructing the structural information block tree based on the RST tree and the information items.

3. The method of claim 2, wherein generating a structural information block tree comprises:

representing the Web page with both an HTML DOM tree and an HTML tag token stream.

4. The method of claim 3, wherein generating a structural information block tree comprises:

filtering out improper repeated-patterns; and

generating sets of candidate patterns and corresponding instances.

5. The method of claim 2, wherein generating a structural information block tree comprises:

filtering out improper repeated-patterns.

6. The method of claim 2, wherein generating a structural information block tree comprises:

generating sets of candidate patterns and corresponding instances.

7. The method of claim 1, wherein clustering and merging the structural information blocks comprises:

acquiring basic information blocks with appropriate granularity from the structural information block tree; and

clustering and merging the basic information blocks to generate semantic information blocks.

8. The method of claim 7, wherein labeling the semantic of the resulting blocks comprises:

labeling a main text information block and related link block in the semantic information blocks of the Web page.

9. An apparatus for segmenting a Web page into information blocks with coherent contents comprising:

a structural information block extracting unit generating a structural information block tree of the Web page; and

a semantic information block extracting unit clustering and merging the structural information blocks and labeling the semantic of the resulting blocks.

10. The apparatus of claim 9, wherein the structural information block extracting unit comprises:

a repeated-pattern discovery unit inducing repeated-patterns within the Web page;

a region detection unit matching the repeated-pattern and the corresponding region in the Web page;

a RST tree generation unit constructing an RST tree according to the regions;

an information item detecting unit identifying information items within each information block; and

a structural information block tree generation unit constructing the structural information block tree based on the RST tree and the information items.

11. The apparatus of claim 10, wherein the structural information block extracting unit comprises a page representation unit representing the Web page with both an HTML DOM tree and an HTML tag token stream.

12. The apparatus of claim 11, wherein the repeated-pattern discovery unit filters out improper repeated-patterns and generates sets of candidate patterns and corresponding instances.

13. The apparatus of claim 10, wherein the repeated-pattern discovery unit filters out improper repeated-patterns.

14. The apparatus of claim 10, wherein the repeated-pattern discovery unit generates sets of candidate patterns and corresponding instances.

15. The apparatus of claim 9, wherein the semantic information block extracting unit comprises:

a basic information block acquisition unit acquiring basic information blocks with appropriate granularity from the structural information block tree; and

a semantic information block generation unit clustering and merging the basic information blocks to generate semantic information blocks.

16. The apparatus of claim 15, wherein the semantic information block extracting unit comprises:

a main text block and related link block detection unit labeling a main text information block and related link block in the semantic information blocks of the Web page.

17. A method for segmenting a Web page into information blocks with coherent contents comprising the steps of:

extracting structural information blocks from the Web page; and

generating semantic information blocks based on the structural information blocks.