APPROACHES FOR THE UNSUPERVISED CREATION OF STRUCTURAL TEMPLATES FOR ELECTRONIC DOCUMENTS

• US 20100169311A1
• Filed: 12/30/2008
• Published: 07/01/2010
• Est. Priority Date: 12/30/2008
• Status: Abandoned Application

• Current Assignee: Oath Inc. (Verizon Communications Inc.)
• Original Assignee: Oath Inc. (Verizon Communications Inc.)
• Inventors: TENGLI, ASHWIN, Chitrapura, Krishna Prasad, Raghuveer, Aravindan
• Application Number:
  US12/346,483
• Publication Number:
  US 20100169311A1
• Time in Patent Office: Days
• US Class Current: 707/736
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method and apparatus for creating templates for electronic documents is provided. One or more attributes are extracted, using a seed template, from a first document, such as a web page. A second document that contains a particular attribute, extracted from the first document, is identified. The second document may be in a different cluster than the first document. The second document is annotated, using an extracted attribute, to create an annotated document. The second document is annotated without human intervention. A new template for the annotated document is generated. The new template facilitates extraction of information from the annotated document. The new template may be used to extract additional attributes from all documents in the cluster of documents of which the second document is a member. The process may continue over numerous iterations to generate a large number of templates in an automated fashion.

Claims

1. A method for creating templates for electronic documents, comprising:

extracting, using a first template, one or more attributes from a first document;

identifying a second document that contains a particular attribute of said one or more attributes;

annotating said second document, using said first template, to create an annotated document;

generating a new template for said annotated document, wherein said new template facilitates extraction of information from said annotated document; and

storing said new template on a volatile or non-volatile computer-readable medium.

2. The method of claim 1, wherein said second document is in a different cluster of documents than said first document.

3. The method of claim 1, wherein said annotated document is created by an automated process without human intervention.

4. The method of claim 1, wherein said step of annotating said second document comprises:

comparing content of at least one attribute of the one or more attributes extracted from the first document with content of said second document to identify locations within said second document which are desired to be extracted; and

storing data that identifies the locations within said second document which are desired to be extracted.

5. The method of claim 1, wherein said second document is a web page of a web site, wherein said first document is not part of said web site, and wherein the method further comprises:

grouping a plurality of web pages, of said web site, which have similar structural characteristics into a cluster of web pages, wherein said second document is included in said cluster of web pages.

6. The method of claim 5, further comprising:

extracting, using said new template, a same set of attributes from each web page of said cluster of web pages.

7. The method of claim 1, further comprising:

identifying one or more clusters of documents, wherein at least one member from each of said one or more cluster of documents contains at least one attribute of said one or more attributes; and

identifying for which of said one or more clusters of documents a corresponding template should be generated.

8. The method of claim 7, wherein said step of identifying for which of said one or more clusters of documents a corresponding template should be generated comprises:

upon determining that a specified level of correlation exists between portions of a tree representation for documents in a particular cluster of documents of said one or more clusters of documents, determining that a particular corresponding template should be generated for said particular cluster of documents.

9. The method of claim 1, wherein the step of identifying the second document comprises using an inverted index to determine that said second document contains said particular attribute.

10. The method of claim 1, wherein the step of identifying the second document comprises using fuzzy similarity metrics to determine that said second document contains said particular attribute.

11. The method of claim 1, wherein the step of identifying the second document comprises analyzing a representation of key features of said second document to determine that said second document contains said particular attribute.

12. The method of claim 1, wherein the step of identifying the second document comprises:

loading content from said second document into a trie data structure, wherein leaf nodes of said trie data structure represent DOM nodes that contain the same content; and

determining if any leaf nodes of said trie data structure correspond to said particular attribute.

13. A machine-readable storage medium storing one or more sets of instructions, which when executed, cause:

extracting, using a first template, one or more attributes from a first document;

identifying a second document that contains a particular attribute of said one or more attributes;

annotating said second document, using said first template, to create an annotated document; and

generating a new template for said annotated document, wherein said new template facilitates extraction of information from said annotated document.

14. The machine-readable storage medium of claim 13, wherein said second document is in a different cluster of documents than said first document.

15. The machine-readable storage medium of claim 13, wherein said annotated document is created by an automated process without human intervention.

16. The machine-readable storage medium of claim 13, wherein said step of annotating said second document comprises:

comparing content of at least one attribute of the one or more attributes extracted from the first document with content of said second document to identify locations within said second document which are desired to be extracted; and

storing data that identifies the locations within said second document which are desired to be extracted.

17. The machine-readable storage medium of claim 13, wherein said second document is a web page of a web site, wherein said first document is not part of said web site, and wherein execution of said one or more sets of instructions further causes:

grouping a plurality of web pages, of said web site, which have similar structural characteristics into a cluster of web pages, wherein said second document is included in said cluster of web pages.

18. The machine-readable storage medium of claim 17, execution of said one or more sets of instructions further causes:

extracting, using said new template, a same set of attributes from each web page of said cluster of web pages.

19. The machine-readable storage medium of claim 13, execution of said one or more sets of instructions further causes:

identifying one or more clusters of documents, wherein at least one member from each of said one or more cluster of documents contains at least one attribute of said one or more attributes; and

identifying for which of said one or more clusters of documents a corresponding template should be generated.

20. The machine-readable storage medium of claim 19, wherein said step of identifying for which of said one or more clusters of documents a corresponding template should be generated comprises:

upon determining that a specified level of correlation exists between portions of a tree representation for documents in a particular cluster of documents of said one or more clusters of documents, determining that a particular corresponding template should be generated for said particular cluster of documents.

21. The machine-readable storage medium of claim 13, wherein the step of identifying the second document comprises using an inverted index to determine that said second document contains said particular attribute.

22. The machine-readable storage medium of claim 13, wherein the step of identifying the second document comprises using fuzzy similarity metrics to determine that said second document contains said particular attribute.

23. The machine-readable storage medium of claim 13, wherein the step of identifying the second document comprises analyzing a representation of key features of said second document to determine that said second document contains said particular attribute.

24. The machine-readable storage medium of claim 13, wherein the step of identifying the second document comprises:

loading content from said second document into a trie data structure, wherein leaf nodes of said trie data structure represent DOM nodes that contain the same content; and

determining if any leaf nodes of said trie data structure correspond to said particular attribute.