Extraction and analysis of user-generated content

• US 8,458,584 B1
• Filed: 11/18/2010
• Issued: 06/04/2013
• Est. Priority Date: 06/28/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Nad, Tomislav, Wilson, Jonathan D., Zheng, Xinjie
• Primary Examiner(s): Paula, Cesar
• Assistant Examiner(s): ZUBERI, MOHAMMED H
• Application Number:
  US12/949,485
• Time in Patent Office: 929 Days
• Field of Search: 715/230, 715/234, 707/737, 707/740, 707/726, 706/12
• US Class Current: 715/234
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A page splitter may be configured to split a first page of a site into first boilerplate and first posts, and a second page of the site into second boilerplate and second posts. An aggregator may be configured to associate the first page with the second page, based on a similarity of the first boilerplate and the second boilerplate, and configured to associate at least one of the first posts and at least one of the second posts with a first post-type, and at least one of the second posts with a second post-type. A merger may be configured to merge the first boilerplate and the second boilerplate into a boilerplate template, posts of the first post-type from the first page and from the second page into a first post-type template, and posts of the second post-type from the second page into a second post-type template, and further configured to merge the boilerplate template, the first post-type template, and the second post-type template into a site template associated with the site.

Claims

1. A computer system including instructions stored on a computer-readable medium, the computer system comprising:

at least one processor;

a page splitter configured to cause the at least one processor of the computer system to split a first page of a site into first boilerplate and first posts, and a second page of the site into second boilerplate and second posts;

an aggregator configured to cause the at least one processor of the computer system to associate the first page with the second page, based on a similarity of the first boilerplate and the second boilerplate, and configured to associate at least one of the first posts and at least one of the second posts with a first post-type, and at least one of the second posts with a second post-type; and

a merger configured to cause the at least one processor of the computer system to merge the first boilerplate and the second boilerplate into a boilerplate template, posts of the first post-type from the first page and from the second page into a first post-type template, and posts of the second post-type from the second page into a second post-type template, and further configured to merge the boilerplate template, the first post-type template, and the second post-type template into a site template associated with the site.

2. The system of claim 1, comprising:

a post analyzer configured to analyze merged posts of the first post-type and to extract elements therefrom and to annotate the elements based on annotation criteria for inclusion within the first post-type template.

3. The system of claim 2, wherein the first posts and the second posts include user-generated content, and wherein the annotation criteria includes labels for labeling the elements including a text label, an author label, and a date label to be associated with each post of the first posts and the second posts.

4. The system of claim 1, wherein the page splitter is configured to receive the first page and the second page including received crawled page data therefor from a crawler.

5. The system of claim 1, wherein the first posts and the second posts include user-generated content.

6. The system of claim 1, wherein site includes a discussion forum and the first page and the second page include user-generated content included within the discussion forum.

7. The system of claim 6, wherein the posts of the first post-type are defined by type based on a characteristic of corresponding users from whom the posts of the first post-type are received.

8. The system of claim 1, wherein the site template is applicable to a page of the site to extract and annotate elements of posts of the page.

9. The system of claim 1, wherein the merger includes a boilerplate merger configured to merge the first boilerplate and the second boilerplate into the boilerplate template based on a measure of similarity therebetween, relative to at least one other page of the site.

10. The system of claim 1, wherein the merger includes a post-type merger configured to merge the posts of the first post-type into the first post-type template based on a measure of similarity therebetween.

11. A computer-implemented method including executing instructions stored on a computer-readable medium, the method comprising:

splitting a first page of a site into first boilerplate and first posts and a second page of the site into second boilerplate and second posts;

associating the first page with the second page, based on a similarity of the first boilerplate and the second boilerplate;

associating at least one of the first posts with a first post-type, at least one of the second posts with the first post-type, and at least one of the second posts with a second post-type;

merging the first boilerplate and the second boilerplate into a boilerplate template;

merging posts of the first post-type from the first page and from the second page into a first post-type template;

merging posts of the second post-type from the second page into a second post-type template; and

merging the boilerplate template, the first post-type template, and the second post-type template into a site template associated with the site.

12. The method of claim 11, comprising:

analyzing merged posts of the first post-type to extract elements therefrom; and

annotating the elements based on annotation criteria for inclusion within the first post-type template.

13. The method of claim 12, wherein the first posts and the second posts include user-generated content, and wherein the annotation criteria includes labels for labeling the elements including a text label, an author label, and a date label to be associated with each post of the first posts and the second posts.

14. The method of claim 11, wherein splitting the first page of the site into first boilerplate and first posts and the second page of the site into second boilerplate and second posts comprises:

receiving the first page and the second page including received crawled page data therefor from a crawler.

15. The method of claim 11, wherein the first posts and the second posts include user-generated content.

16. The method of claim 15, wherein the posts of the first post-type are defined by type based on a characteristic of corresponding users from whom the posts of the first post-type are received.

17. A computer system comprising:

one or more computing devices programmed to perform operations including;

splitting pages of a site containing user-generated content into boilerplate and post data,

aggregating a page group of the split pages, based on similarity of boilerplate contained within the page group,

aggregating groups of the posts within and among the pages of the page group, based on a post-type of posts within each group of posts,

analyzing posts of each post-type and corresponding post group, to extract elements therefrom, and

annotating the elements based on annotation criteria for inclusion within post-type templates corresponding to each post-type and corresponding post group.

18. The computer system of claim 17, wherein one or more of the computing devices are programmed to perform further operations including:

merging the boilerplate of the page group into a boilerplate template; and

merging the posts of each post-type and corresponding post group into a corresponding merge post, for subsequent analysis and annotation thereof to obtain a corresponding post-type template.

19. The computer system of claim 18, wherein one or more of the computing devices are programmed to perform further operations including:

including the boilerplate template and post-type templates into a site template for the site.

20. The computer system of claim 17, wherein posts of a post-type are defined based on a characteristic of corresponding users from whom the posts of the post-type are received.

21. A computer system comprising:

one or more computing devices programmed to perform operations including

splitting a first page of a site into first boilerplate and first posts, and a second page of the site into second boilerplate and second posts;

associating the first page with the second page, based on a similarity of the first boilerplate and the second boilerplate, and configured to associate at least one of the first posts and at least one of the second posts with a first post-type, and at least one of the second posts with a second post-type; and

merging the first boilerplate and the second boilerplate into a boilerplate template, posts of the first post-type from the first page and from the second page into a first post-type template, and posts of the second post-type from the second page into a second post-type template, and further configured to merge the boilerplate template, the first post-type template, and the second post-type template into a site template associated with the site.

22. The system of claim 21, wherein one or more of the computing devices are programmed to perform further operations including:

analyzing merged posts of the first post-type and to extract elements therefrom and to annotate the elements based on annotation criteria for inclusion within the first post-type template.

23. The system of claim 21, wherein the first posts and the second posts include user-generated content.

24. The system of claim 21, wherein the merging includes boilerplate merging configured to merge the first boilerplate and the second boilerplate into the boilerplate template based on a measure of similarity therebetween, relative to at least one other page of the site.

25. The system of claim 21, wherein the merging includes a post-type merger configured to merge the posts of the first post-type into the first post-type template based on a measure of similarity therebetween.