Mining semi-structured social media

• US 9,002,852 B2
• Filed: 11/15/2012
• Issued: 04/07/2015
• Est. Priority Date: 11/15/2012
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Farahat, Ayman Omar, Ahmed, Nesreen Kamel Abuseree
• Primary Examiner(s): HO, BINH VAN
• Application Number:
  US13/677,927
• Publication Number:
  US 20140136541A1
• Time in Patent Office: 873 Days
• Field of Search: 707/740
• US Class Current: 707/740
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking

Abstract

Methods and apparatus for analysis of semi-structured social media are described. A method comprises classifying a plurality of user-generated content entries into a plurality of categories based at least in part on an analysis of respective structured components of at least a subset of the plurality of entries. The method further includes determining, based at least in part on an analysis of additional components of entries of a particular category, a set of representative content elements of entries of the particular category, and generating a report that comprises one or more representative content elements of the particular category.

Claims

1. A method, comprising:

performing, by one or more computing devices;

collecting user-generated content entries from one or more social media data sources;

classifying the user-generated content entries into categories based at least in part on an analysis of respective structured components of at least a subset of the entries, the structured components including data organized in a defined set of fields of respective data types;

determining, based at least in part on a semantic analysis of unstructured components of the entries of a particular category, a set of representative user-generated content elements of the entries of the particular category comprising a set of frequent word combinations, the unstructured components including additional data of arbitrary length and not organized in the defined set of fields of the respective data types;

generating a report that comprises one or more of the representative user-generated content elements of the set of representative user-generated content elements of the entries of the particular category; and

generating, based at least in part on the report, a survey question in requesting additional feedback associated with a particular frequent word combination of the set of frequent word combinations.

2. The method as recited in claim 1, wherein the unstructured components comprise natural language text.

3. The method as recited in claim 1, wherein said collecting the user-generated content entries using at least one of:

(a) a screen-scraping technique on a web page, (b) one or more invocations of an application programming interface (API), or (c) one or more database queries.

4. The method as recited in claim 1, wherein the structured component of a particular entry of the user-generated content entries comprises an indication of at least one of:

(a) a user-selected rating level, (b) a date, (c) a geographical location, (d) a purchase price, or (e) a demographic category of a user.

5. The method as recited in claim 1, wherein the categories include at least one of:

(a) a positive rating category, (b) a negative rating category, or (c) a neutral rating category.

6. The method as recited in claim 1, wherein the semantic analysis of an unstructured component of a particular entry of the particular category comprises an analysis of user-generated text to identify noun and verb phrases.

7. The method as recited in claim 1, wherein the semantic analysis of an unstructured component of a particular entry of the particular category comprises a semantic analysis of user text to identify parts of speech.

8. The method as recited in claim 1, wherein the set of representative user-generated content elements of the entries of the particular category comprises the set of frequent word combinations, wherein the report comprises an indication of a context in which a variant of a frequent word combination of the set of frequent word combinations occurred in a particular user-generated content entry.

9. A system, comprising:

one or more processors; and

a memory comprising program instructions executable by the one or more processors to;

collect user-generated content entries from one or more social media data sources;

classify the user-generated content entries into categories based at least in part on an analysis of respective structured components of at least a subset of the entries, the structured components including data organized in a defined set of fields of respective data types;

determine, based at least in part on a statistical analysis of unstructured components of the entries of a particular category, a set of representative user-generated content elements of the entries of the particular category comprising a set of frequent word combinations, the unstructured components including additional data of arbitrary length and not organized in the defined set of fields of the respective data types;

generate a report that comprises one or more of the representative user-generated content elements of the set of representative user-generated content elements of the entries of the particular category; and

generate, based at least in part on the report, a survey question in requesting additional feedback associated with a particular frequent word combination of the set of frequent word combinations.

10. The system as recited in claim 9, wherein the program instructions are executable by the one or more processors to:

collect the user-generated content entries using at least one of;

(a) a screen-scraping technique on a web page, (b) one or more invocations of an application programming interface (API), or (c) one or more database queries.

11. The system as recited in claim 9, wherein an unstructured component of a particular entry of the particular category comprises natural language text, wherein the statistical analysis of the unstructured component comprises an n-gram analysis of user-generated text.

12. The system as recited in claim 9, wherein the statistical analysis of an unstructured component of a particular entry of the particular category includes tokenizing of user-generated text.

13. The system as recited in claim 9, wherein the set of representative user-generated content elements of the entries of the particular category comprises the set of frequent word combinations, wherein the report comprises a link to one or more contexts in which a variant of a particular frequent word combination of the set of frequent word combinations occurred in one or more of the user-generated content entries.

14. The system as recited in claim 9, wherein semantic analysis of an unstructured component of a particular entry of the particular category comprises a semantic analysis of user text to identify parts of speech.

15. A non-transitory computer-readable storage medium storing program instructions that when executed by a computing device implement:

collecting user-generated content entries from one or more social media data sources;

classifying the user-generated content entries into categories based at least in part on an analysis of respective structured components of at least a subset of the entries, the structured components including data organized in a defined set of fields of respective data types;

determining, based at least in part on a combination of statistics and semantic analysis of unstructured components of the entries of a particular category, a set of representative user-generated content elements of the entries of the particular category comprising a set of frequent word combinations, the unstructured components including additional data of arbitrary length and not organized in the defined set of fields of the respective data types;

generating a report that comprises one or more of the representative user-generated content elements of the set of representative user-generated content elements of the entries of the particular category; and

generating, based at least in part on the report, a survey question in requesting additional feedback associated with a particular frequent word combination of the set of frequent word combinations.

16. The non-transitory computer-readable storage medium as recited in claim 15, wherein collecting the user-generated content entries using at least one of:

(a) a screen-scraping technique on a web page, (b) one or more invocations of an application programming interface (API), or (c) one or more database queries.

17. The non-transitory computer-readable storage medium as recited in claim 15, wherein the structured component of a particular entry of the user-generated content entries comprises an indication of at least one of:

(a) a user-selected rating level, (b) a date, (c) a geographical location, (d) a purchase price, or (e) a demographical category of a user.

18. The non-transitory computer-readable storage medium as recited in claim 15, wherein an unstructured component of a particular entry of the particular category comprises user-generated text, wherein the combination of statistics and semantic analysis of the unstructured component comprises a use of one or more of:

tokenization, word-stemming, stop-word removal, n-gram computation, part-of-speech tagging, and chunking.

19. The non-transitory computer-readable storage medium as recited in claim 15, wherein the combination of statistics and semantic analysis of an unstructured component of a particular entry of the particular category includes identifying noun and verb phrases.

20. The non-transitory computer-readable storage medium as recited in claim 15, wherein the categories comprise a first category and a second category, wherein the respective sets of representative user-generated content elements of the entries of the first category comprise respective sets of frequent word combinations, and wherein the report comprises an indication of a particular frequent word combination identified in the first category that does not appear in a set of frequent word combinations for the second category.