Extracting structured data from weblogs

• US 20060287989A1
• Filed: 06/16/2006
• Published: 12/21/2006
• Est. Priority Date: 06/16/2005
• Status: Active Grant

• Current Assignee: Buzzmetrics, Ltd. (Nielsen Holdings plc)
• Original Assignee: Buzzmetrics, Ltd. (Nielsen Holdings plc)
• Inventors: Glance, Natalie
• Granted Patent: US 9,158,855 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/28 Databases characterised by their database models, e.g. relational or object models
• CPC Class Codes: G06F 16/80 of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML content-based retrieval of web data G06F16/95
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: G06F 40/205 Parsing

Abstract

A method of extracting individual posts from a weblog comprises the steps of: (a) providing a feed associated with the weblog; and (b) screen scraping the weblog into a representation for weblog posts using the feed data containing partial content of the weblog.

Claims

1. A method of extracting individual posts from a weblog, comprising the steps of:

(a) accessing the home page of the weblog;

(b) identifying at least one feed associated with the weblog;

(c) determining whether the feed contains sufficient content for performing feed-guided segmentation;

(d) if the feed contains sufficient content for feed-guided segmentation, determining whether the feed contains full content or partial content of the weblog;

(e) if the feed contains full content of the weblog, mapping the data found in the feed into a representation for weblog posts; and

(f) if the feed contains partial content of the weblog, screen scraping the weblog into a representation for weblog posts using the feed data.

2. The method of claim 1, wherein

the identifying step (b) comprises receiving a feed URL for the weblog from a ping related by a ping server from which the weblog update is collected.

3. The method of claim 1, wherein

the identifying step (b) comprises mapping the weblog'"'"'s URL to a feed URL, if the weblog is hosted by a weblog host that publishes a full content feed for the weblog.

4. The method of claim 1, wherein

the identifying step (b) comprises using RSS auto-discovery for identifying the the webog feed'"'"'s URL from metadata in the HTML for the weblog'"'"'s home page.

5. The method of claim 1, wherein

the identifying step (b) further comprises:

(b1) extracting hyperlinks from the body of the weblog; and

(b2) filtering the set of extracted hyperlinks using a classifier to identify hyperlinks that belong to feeds for the weblog.

6. The method of claim 5, wherein

a set of heuristics is used in the filtering step (b2) to identify a feed for the weblog from the extracted hyperlinks.

7. The method of claim 5, wherein

the filtering step (b2) uses at least one of the following criteria to identify a feed for the weblog:

(i) URLs that allow readers to subscribe to the feed in their RSS reader;

(ii) URLs matching the host name of the blog and having a common feed suffix taken from the set including {“

atom.xml”

, “

.xml”

, “

.rss”

, and “

rdf”

}; and

(iii) URLs matching the domain name of the blog and having a common feed prefixes taken from the set including {“

xml” and

“

rss”

}.

8. The method of claim 1, wherein

in the determining step (c), an item in the feed is deemed to contain sufficient content if it contains a date-posted field and either a content field or a description field.

9. The method of claim 1, wherein

the determining step (d) uses heuristics to determine whether the feed contains full content based on features of content and description text in the feed.

10. The method of claim 9, wherein

the features of content and description text include presence or absence of HTML tags, percent of posts ending in ellipses, and type of feed.

11. The method of claim 1, wherein

the screen scraping step (f) further comprises:

(f1) creating skeletal representations of the weblog'"'"'s posts using data available from the feed.

12. The method of claim 11, wherein

for each post in the weblog, the skeletal representation includes at least one of the following data:

the weblog'"'"'s URL, the date on which the post was posted, partial content of the post, a title of the post, the author of the post, and a permalink.

13. The method of claim 12, wherein

the screen scraping step (f) further comprises:

(f2) removing summarization artifacts from the skeletal representations.

14. The method of claim 12, wherein

the screen scraping step (f) further comprises:

(f3) removing ads from the skeletal representations.

15. The method of claim 12, wherein

the screen scraping step (f) further comprises:

(f4) searching the home page of the weblog for content matching the skeletal representations.

16. The method of claim 15, wherein

the screen scraping step (f) further comprises:

(f5) upon finding content on the home page of the weblog matching a skeletal representation, identifying an enclosing node associated with the matching text in the tidied XHTML for the weblog page;

(f6) copying text contained within the enclosing node to the skeletal representation, thereby using said text as the full content of the post; and

(f7) repeating steps (f5) and (f4) for the skeletal representation of each post in the weblog.

17. The method of claim 1, further comprising the step of:

(g) if the feed contains insufficient partial content to perform screen scraping in step (f), repeating steps (c) through (f) using another feed associated with the weblog.

18. The method of claim 17, further comprising the step of:

(h) if the weblog contains no feeds with sufficient full content or sufficient partial content, performing screen scraping of the weblog into a representation for weblog posts.

19. The method of claim 18, wherein

the screen scraping step (h) further comprises:

(h1) extracting dates from the tidied XHTML for the weblog page;

(h2) sorting the extracted dates into ordered lists, each ordered list corresponding to a unique relative XPath;

(h3) filtering the ordered lists according to a set of heuristics to determine which list corresponds to the actual entry dates of the weblog posts;

(h4) segmenting the weblog into entries, using dates from the list obtained in step (h3) as markers for the entries;

(h5) segmenting each weblog entry into posts using post titles markers; and

(h6) identifying a permalink and author for each post.

20. The method of claim 19, wherein

the set of heuristics in the filtering step (h3) includes at least one of:

(i) keeping only lists whose dates all belong to the current year and/or the past year;

(ii) keeping only non-singleton date lists;

(iii) keeping only lists whose dates conform to a similar format (e.g. MM/dd/YYYY;

(iv) keeping only lists whose dates decrease monotonically;

(v) keeping only lists with most recent dates (but not in the future);

(vi) keeping only lists with longest date string representation;

(vii) keeping only lists with the greatest number of dates; and

(viii) keeping only first list.

21. The method of claim 19, wherein

the segmenting step (h4) uses a set of heuristics to identify the end of each entry.

22. The method of claim 21, wherein

the set of heuristics includes at least one of:

a start of a sidebar on the weblog page, a copyright notice on the weblog page, a form, and a comment.

23. The method of claim 21, wherein

the set of heuristics includes a step of looking for a node in the document object model whose XPath is analogous in structure to the XPath of the last node in a previous weblog entry.

24. The method of claim 19, wherein

the segmenting step (h5) further comprises:

(i) searching the entry'"'"'s nodes to identify a title node;

(ii) assuming that the titles of all subsequent posts have the same relative XPath as the title node identified in step (i); and

(iii) if no title node is identified, assuming that the weblog entry is a single post.

25. The method of claim 24, wherein

a node is identified as a title node if the node'"'"'s class attribute is one of:

title, subtitle, or blogpost.

26. The method of claim 19, wherein

the identifying step (h6) further comprises identifying patterns indicative of an author byline.

27. The method of claim 19, wherein

the identifying step (h6) further comprises identifying hrefs in the post content that match, thereby indicating a permalink.

28. A method of extracting individual posts from a weblog, comprising the steps of:

(a) providing a feed associated with the weblog; and

(b) screen scraping the weblog into a representation for weblog posts using the feed data containing partial content of the weblog.

29. The method of claim 28, wherein

the screen scraping step (b) further comprises:

(b1) creating skeletal representations of the weblog'"'"'s posts using data available from the feed.

30. The method of claim 29, wherein

for each post in the weblog, the skeletal representation includes at least one of the following data:

the weblog'"'"'s URL, the date on which the post was posted, partial content of the post, a title of the post, the author of the post, and a permalink.

31. The method of claim 30, wherein

the screen scraping step (b) further comprises:

(b2) removing summarization artifacts from the skeletal representations.

32. The method of claim 30, wherein

the screen scraping step (b) further comprises:

(b3) removing ads from the skeletal representations.

33. The method of claim 30, wherein

the screen scraping step (b) further comprises:

(b4) searching the home page of the weblog for content matching the skeletal representations.

34. The method of claim 33, wherein

the screen scraping step (b) further comprises:

(b5) upon finding content on the home page of the weblog matching a skeletal representation, identifying an enclosing node associated with the matching text in the tidied XHTML for the weblog page;

(b6) copying text contained within the enclosing node to the skeletal representation, thereby using said text as the full content of the post; and

(b7) repeating steps (b5) and (b4) for the skeletal representation of each post in the weblog.