System and method for tokenization of text

• US 20060116862A1
• Filed: 12/01/2004
• Published: 06/01/2006
• Est. Priority Date: 12/01/2004
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Dictaphone Corporation (Microsoft Corporation)
• Inventors: Santisteban, Ana, Dowd, John, Uhrbach, Amy J., Lapshina, Larissa, Rechea, Bernardo, Frankel, Alan, Carus, Alwin B., Carrier, Jill, Cote, William F., Femina, Kathryn Del La, Han, Wensheng
• Granted Patent: US 7,937,263 B2
• Time in Patent Office: Days
• US Class Current: 704/1
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates

Abstract

The present invention pertains to a system and method for the tokenization of text. The featurizer may be configured to receive input text and convert the input text into tokens. According to one aspect of the invention, the tokens may include only one type of character, the characters selected from the group consisting of letters, numbers, and punctuation. The tokenizer may also include a classifier. The classifier may be configured to receive the tokens from the featurizer. Furthermore, the classifier may be configured to analyze the tokens received from the featurizer to determine if the tokens may be input into a predetermined classification model using a preclassifier. If one of the tokens passes the preclassifier, then the token is classified using the predetermined classification model. Additionally, according to a first aspect of the invention, the tokenizer may also include a finalizer. The finalizer may be configured to receive the tokens and may be configured to produce a final output.

Claims