Natural language processing optimized for micro content

• US 8,938,450 B2
• Filed: 08/30/2013
• Issued: 01/20/2015
• Est. Priority Date: 02/17/2012
• Status: Active Grant

• Current Assignee: Blazent Incorporated (ServiceNow Incorporated)
• Original Assignee: Bottlenose, Inc. (ServiceNow Incorporated)
• Inventors: Spivack, Nova, ter Heide, Dominiek
• Primary Examiner(s): Ortiz Ditren, Belix M
• Application Number:
  US14/015,021
• Publication Number:
  US 20140136544A1
• Time in Patent Office: 508 Days
• Field of Search: 707/723, 707/740
• US Class Current: 707/723
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/263 Language identification
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

A system and a method for microcontent natural language processing are presented. The method comprising steps of receiving a microcontent message from a social networking server, tokenizing the microcontent message into one or more text tokens, detecting the language of the microcontent message and selecting the property dictionary for part-of-speech tag, part-of-speech tagging the microcontent message to identify related pronouns and nouns based on the selected dictionary, and extracting topics form the microcontent messages and assigning confidence values to the topics.

Claims

1. A method for microcontent natural language processing comprising:

receiving a plurality of microcontent messages from a social networking server, the plurality of microcontent messages including a microcontent message;

breaking up the microcontent message into one or more text tokens by using a tokenizer module that is configured to process micro-syntax and punctuation in the microcontent message;

performing a part-of-speech (POS) tagging process on the text tokens to identify a linguistic category for each of the text tokens, wherein the POS tagging process for a respective text token is performed using an error-driven transformation-based tagger and based on a definition and a context of the respective text token;

performing a topic extraction on the microcontent message to extract topic metadata for the microcontent message based on the identified linguistic category for each of the text tokens, wherein the extraction is performed without looking up a pre-specified topic in a dictionary of known entities;

associating a topic metadata to the microcontent message based on the extracted topic;

identifying type metadata from the microcontent message based on an ontology of predetermined microcontent types by applying a database of annotation rules to the text tokens of the microcontent message;

associating the identified type metadata to the microcontent message;

analyzing co-occurrence of all available metadatas in the plurality of microcontent messages;

producing a list of trending topics for the all available metadatas based on results from the analyzing; and

compiling a trend database by tracking how the list of trending topics changes over time.

2. The method of claim 1, wherein the ontology of predetermined microcontent types are organized as a relational database including trees of the predetermined microcontent types.

3. The method of claim 2, wherein the annotation rules includes list of names and patterns that help assigning a tree of the trees of the predetermined microcontent types to the microcontent message.

4. The method of claim 1, further comprising:

providing an interface for users to edit the ontology of predetermined microcontent types or the database of annotation rules.

5. The method of claim 1, wherein the step of identifying type metadata of the microcontent message includes:

compiling a matching index including a set of expressions based on the ontology of predetermined microcontent types or the database of annotation rules; and

matching the expressions of the matching index with the microcontent message.

6. The method of claim 1, further comprising:

performing a language detection on the microcontent message to detect language metadata; and

associating the detected language metadata with the microcontent message.

7. The method of claim 6, wherein the language detection is based on a dictionary-based language detector or an NGram language detector.

8. The method of claim 1, further comprising:

performing a speech tagging on the microcontent message to identify nouns in the microcontent message.

9. The method of claim 8, wherein the speech tagging is based on a Brill tagger method.

10. The method of claim 1, further comprising:

assigning a confidence score for each piece of the extracted topic metadata; and

associating the topic metadata and confidence scores with the microcontent message.

11. The method of claim 1, further comprising:

identifying sentiment metadata by performing a sentiment analysis on the microcontent message; and

associating the sentiment metadata with the microcontent message.

12. The method of claim 11, wherein the sentiment analysis is based on a Naï

ve Bayesian classifier that is trained offline to an annotated set of positive, negative and neutral microcontent messages.

17. The method of claim 1, further comprising:

presenting, via a user interface, the list of trending topics as scatters in a quadrant scatter plot, wherein a volume of a respective scatter indicates a size of a trending topic represented by the respective scatter.

18. The method of claim 17, further comprising:

producing an animation of the quadrant scatter plot over a time period to reflect how the trending topics change in the plurality of microcontent messages.

13. A system comprising:

a network component configured for receiving a plurality of microcontent messages from asocial networking server, the plurality of microcontent messages including a microcontent message;

a processor; and

a memory storing instructions which, when executed by the processor, cause the system to perform a process including;

breaking up the microcontent message into one or more text tokens by using a tokenizer module that is configured to process micro-syntax and punctuation in the microcontent message;

performing a part-of-speech (POS) tagging process on the text tokens to identify a linguistic category for each of the text tokens, wherein the POS tagging process for a respective text token is performed using an error-driven transformation-based tagger and based on a definition and a context of the respective text token;

performing a topic extraction on the microcontent message to extract topic metadata for the microcontent message based on the identified linguistic category for each of the text tokens, wherein the extraction is performed without looking up a pre-specified topic in a dictionary of known entities;

associating a topic metadata to the microcontent message based on the extracted topic;

identifying type metadata from the microcontent message based on an ontology of predetermined microcontent types by applying a database of annotation rules to the text tokens of the microcontent message;

associating the identified type metadata to the microcontent message;

analyzing co-occurrence of all available metadatas in the plurality of microcontent messages;

producing a list of trending topics for the all available metadatas based on results from the analyzing; and

compiling a trend database by tracking how the list of trending topics changes over time.

14. The system of claim 13, wherein the step of identifying type metadata of the microcontent message includes:

compiling a matching index including a set of expressions based on the ontology of predetermined microcontent types or the database of annotation rules; and

matching the expressions of the matching index with the microcontent message.

15. The system of claim 13, wherein the process further includes:

performing a language detection on the microcontent message to detect language metadata;

performing a speech tagging on the microcontent message to identify nouns in the microcontent message; and

identifying sentiment metadata by performing a sentiment analysis on the microcontent message.

16. The system of claim 13, wherein the process further includes:

presenting, via a user interface, the list of trending topics as scatters in a quadrant scatter plot, wherein a volume of a respective scatter indicates a size of a trending topic represented by the respective scatter; and

producing an animation of the quadrant scatter plot over a time period to reflect how the trending topics change in the plurality of microcontent messages.