Method and system for extracting web data

• US 20070033188A1
• Filed: 09/08/2006
• Published: 02/08/2007
• Est. Priority Date: 08/05/2005
• Status: Active Grant

• Current Assignee: Buzzmetrics, Ltd. (Nielsen Holdings plc)
• Original Assignee: Buzzmetrics, Ltd. (Nielsen Holdings plc)
• Inventors: Schler, Jonathan, Tenne, David, Levy, Ori, Navot, Akiva
• Granted Patent: US 7,596,552 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/38 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/383 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06Q 90/00 Systems or methods specially adapted for administrative, commercial, financial, managerial or supervisory purposes, not involving significant data processing
• CPC Class Codes: G06Q 99/00 Subject matter not provided for in other groups of this subclass
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

An apparatus for providing an analysis of attitudes expressed in web content, comprising: a collector for collecting attitude-data in relation to a predetermined subject from one or more pre-selected web site, the attitude-data containing attitudes in relation to the predetermined subject; a processor, associated with the collector, for processing the attitude data so as to generate an attitude analysis; and an outputter, associated with the processor, for outputting the attitude analysis, thereby to provide an indication of attitudes being expressed in the web content in relation to the predetermined subject.

Claims

1. Method for analyzing attitudes expressed in web content, said attitudes being in relation to a predetermined subject, comprising:

automatically collecting attitude data from at least one pre-selected web site, the attitude-data expressing a plurality of attitudes in relation to said predetermined subject;

electronically processing the attitude data so as to generate attitude information indicative of said plurality of attitudes; and

outputting said attitude-information, thereby to provide an analysis of said attitudes in relation to said predetermined subject.

2. The method of claim 1, wherein said collecting further comprises crawling said pre-selected web site.

3. The method of claim 1, wherein said collecting further comprises extracting relevant data from relevant pages of said pre-selected web site.

4. The method of claim 1, wherein said collecting further comprises creating a mark-up language format representation of said attitude data.

5. The method of claim 1, wherein said collecting further comprises integrating the attitude-data into a complete and non-redundant attitude data.

6. The method of claim 1, wherein said processing further comprises an initial preprocessing, for extracting metadata of said attitude data.

7. The method of claim 1, wherein said processing further comprises categorizing text of said attitude-data.

8. The method of claim 7, wherein said categorizing text of said attitude-data, comprises categorization according to style.

9. The method of claim 7, wherein said categorizing text of said attitude-data, comprises categorization according to content.

10. The method of claim 1, wherein said processing further comprises analyzing said attitude-data for finding in said attitude-data at least one relevance-relationship between a phrase and a respective concept.

11. The method of claim 1, wherein said processing further comprises measuring correlations in said attitude-data among phrases having a relevance-relationship with a common concept and between at least one of the phrases and said common concept.

12. The method of claim 1, wherein said processing further comprises extracting at least one key quotation from said attitude-data, said key quotation being descriptive of a relevance-relationship existing in said attitude-data between a concept and respective phrases relating thereto.

13. The method of claim 1, wherein said processing further comprises clustering concepts relating to said attitude-data.

14. The method of claim 1, wherein said collecting comprises collecting attitude-data from all web sites.

15. The method of claim 13, further comprising displaying said clustered concepts and accessing source content via said clusters.

16. Device for interactive setting of a data collection policy using a web page display, comprising a web page displayer, for displaying a web page to a user, operable for defining a data collection policy in relation to said web page.

17. The device of claim 16, wherein said web page displayer is further operable to define a specific data collection policy in relation to a respective region of said web page.