Tokenizer for a natural language processing system
First Claim
1. A method of segmenting a textual input string including characters separated by spaces, comprising:
- receiving the textual input string;
proposing a first segmentation of at least a portion of the input string by segmenting the input string at the spaces to obtain a plurality of tokens;
attempting to validate word boundaries in the first segmentation by submitting the first segmentation to a linguistic knowledge component;
if the first segmentation is not validated, proposing a subsequent segmentation by;
determining whether invalid tokens contain any of a predetermined plurality of multi-character punctuation strings or emoticons;
if so, segmenting the tokens into subtokens based on the multi-character punctuation strings or emoticons;
determining whether invalid tokens contain punctuation marks;
if so, segmenting the tokens into subtokens according to a predetermined precedence hierarchy of punctuation;
determining whether invalid tokens contain both alpha and numeric characters;
if so, segmenting the tokens into subtokens at boundaries between the alpha and numeric characters in the tokens;
submitting the subsequent segmentation to the linguistic knowledge component for validation; and
repeating the steps of proposing a subsequent segmentation and submitting the subsequent segmentation to the linguistic knowledge component until the portion of the input string is validated or the portion of the input string has been segmented according to a predetermined number of segmentation criteria.
2 Assignments
0 Petitions
Accused Products
Abstract
The present invention is a segmenter used in a natural language processing system. The segmenter segments a textual input string into tokens for further natural language processing. In accordance with one feature of the invention, the segmenter includes a tokenizer engine that proposes segmentations and submits them to a linguistic knowledge component for validation. In accordance with another feature of the invention, the segmentation system includes language-specific data that contains a precedence hierarchy for punctuation. If proposed tokens in the input string contain punctuation, they can illustratively be broken into subtokens based on the precedence hierarchy.
-
Citations
10 Claims
-
1. A method of segmenting a textual input string including characters separated by spaces, comprising:
-
receiving the textual input string; proposing a first segmentation of at least a portion of the input string by segmenting the input string at the spaces to obtain a plurality of tokens; attempting to validate word boundaries in the first segmentation by submitting the first segmentation to a linguistic knowledge component; if the first segmentation is not validated, proposing a subsequent segmentation by; determining whether invalid tokens contain any of a predetermined plurality of multi-character punctuation strings or emoticons; if so, segmenting the tokens into subtokens based on the multi-character punctuation strings or emoticons; determining whether invalid tokens contain punctuation marks; if so, segmenting the tokens into subtokens according to a predetermined precedence hierarchy of punctuation; determining whether invalid tokens contain both alpha and numeric characters; if so, segmenting the tokens into subtokens at boundaries between the alpha and numeric characters in the tokens; submitting the subsequent segmentation to the linguistic knowledge component for validation; and repeating the steps of proposing a subsequent segmentation and submitting the subsequent segmentation to the linguistic knowledge component until the portion of the input string is validated or the portion of the input string has been segmented according to a predetermined number of segmentation criteria. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A method of segmenting a textual input string including characters separated by spaces, comprising:
-
receiving the textual input string; proposing a first segmentation of at least a portion of the input string by identifying a token as a group of characters flanked by white spaces or either end of the input string; attempting to validate word boundaries in the first segmentation by submitting the first segmentation to a linguistic knowledge component; if the first segmentation is not validated, proposing a subsequent segmentation by; determining whether invalid tokens contain any of a predetermined plurality of multi-character punctuation strings or emoticons; if so, segmenting the tokens into subtokens based on the multi-character punctuation strings or emoticons; determining whether invalid tokens contain punctuation marks; if so, segmenting the tokens into subtokens according to a predetermined precedence hierarchy of punctuation; determining whether invalid tokens contain both alpha and numeric characters; if so, segmenting the tokens into subtokens at boundaries between the alpha and numeric characters in the tokens; submitting the subsequent segmentation to the linguistic knowledge component for validation; and repeating the steps of proposing a subsequent segmentation and submitting the subsequent segmentation to the linguistic knowledge component until the portion of the input string is validated or the portion of the input string has been segmented according to a predetermined number of segmentation criteria. - View Dependent Claims (9, 10)
-
Specification