METHOD AND APPARATUS FOR NAMED ENTITY RECOGNITION IN NATURAL LANGUAGE

• US 20090326923A1
• Filed: 05/15/2007
• Published: 12/31/2009
• Est. Priority Date: 05/15/2006
• Status: Active Grant

• Current Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Yan, Pengju, Takashi, Tsuzuki, Sun, Yufei
• Granted Patent: US 8,938,385 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32

Abstract

The present invention provides a method for recognizing a named entity included in natural language, comprising the steps of: performing gradual parsing model training with the natural language to obtain a classification model; performing gradual parsing and recognition according to the obtained classification model to obtain information on positions and types of candidate named entities; performing a refusal recognition process for the candidate named entities; and generating a candidate named entity lattice from the refusal-recognition-processed candidate named entities, and searching for a optimal path. The present invention uses a one-class classifier to score or evaluate these results to obtain the most reliable beginning and end borders of the named entities on the basis of the forward and backward parsing and recognizing results obtained only by using the local features.

Claims

1. A method for recognizing a named entity included in natural language, comprising the steps of:

recognizing candidate named entities by a gradual recognizer;

extracting character-based global features of the recognized candidate named entities by a refusal recognizer;

testing said candidate named entities by using said global features; and

if a score of the testing is larger than a predetermined threshold, accepting said candidate named entity, otherwise, refusing the recognition thereof.

2. The method according to claim 1, wherein the step of testing said candidate named entities by using said global features includes steps of using said global features as an input and testing said candidate named entities by using a one-class support vector machine.

3. The method according to claim 1, wherein said step of gradually recognizing includes a forward gradual parsing training step and a backward gradual parsing training step so as to generate a forward gradual classification model and a backward gradual classification model respectively.

4. The method according to claim 1, further comprising a step of forming the candidate named entities recognized by the refusal recognizer as a lattice, and a step of searching for a path having a maximum refusal recognition score on the lattice.

5. The method according to claim 3, wherein said refusal recognition score is equal to a testing score of a one-class support vector machine minus said threshold.

6. A method for recognizing a named entity included in natural language, comprising the steps of:

extracting local features of words or characters included in the center of a feature window by using the feature window;

on the basis of a classification model obtained after training the natural language with a gradual parsing model, gradually parsing and recognizing the natural language so as to obtain information on positions and types of candidate named entities;

extracting global features of the candidate named entities included in the center of the feature window by using the feature window, and;

performing a refusal recognition process for the candidate named entities; and

generating a candidate named entity lattice from the refusal-recognition-processed candidate named entities, and searching for an optimal path.

7. The method according to claim 6, wherein said gradual parsing model training step includes using a forward gradual classification model obtained through performing forward gradual parsing model training, and a backward gradual classification model obtained through performing backward gradual parsing model training to perform training.

8. The method according to claim 6, wherein said gradual parsing and recognizing step includes a step of extracting the local feature of the extracted words, and parsing the extracted words based on the extracted local features.

9. The method according to claim 8, further comprising a step of obtaining a multi-dimensional vector representing all features of the candidate named entity by a feature extracting module, and classifying the obtained multi-dimensional vector by using a gradual classification model.

10. The method according to claim 6, wherein the refusal recognition processing step includes a step of extracting the global feature of characters in the candidate named entity, and performing the refusal recognition process for the obtained global features by using the model obtained through learning on a one-class classifier.

11. The method according to claim 6, wherein the refusal recognition processing step includes a step of extracting the candidate named entities by a refusal recognizer, calculating precision scores of these candidate named entities, and accepting or refusing the candidate named entities according to the calculated scores.

12. The method according to claim 6, wherein a refusal recognizer obtained by using different training sets for different kinds of named entities is used in the refusal recognition processing step.

13. The method according to claim 6, wherein the refusal recognition processing step uses a one-class support vector machine to test the candidate named entities, if a score of the testing is larger than a predetermined threshold, accepting said candidate named entity, otherwise, refusing said candidate named entity.

14. The method according to claim 6, wherein in said refusal recognition processing step, if the forward and backward parsing processes obtain the same candidate named entity, then the candidate named entity is evaluated with a one-class support vector machine and an identical candidate threshold.

15. The method according to claim 6, wherein in said refusal recognition processing step, if one parsing process obtains one candidate named entity and the other parsing process does not obtain any other named entity which intersects with said one candidate named entity in position, then said one candidate named entity is evaluated with a one-class support vector machine and a free threshold.

16. The method according to claim 6, wherein in said refusal recognition processing step, if the forward parsing process obtains one candidate named entity and the backward parsing process obtains another candidate named entity, and these two candidates intersect with each other in position and have the same type, then at most two new candidates of the same type are generated according to the overlapping conditions, and then the at most four candidates are evaluated with a one-class support vector machine and a overlapping candidate threshold.

17. The method according to claim 6, wherein in said refusal recognition processing step, if the two parsing processes obtain two candidate named entities which intersect with each other in position and have different types, then the two candidates are evaluated with a one-class support vector machine and a forward and backward conflicting candidate threshold.

18. An off-line training method for recognizing a named entity included in natural language, comprising the steps of:

performing forward gradual parsing model training to a natural sentence to obtain a forward gradual classification model;

performing backward gradual parsing model training to the natural sentence to obtain a backward gradual classification model; and

performing refusal recognition model training to candidate named entities based on the obtained forward and backward gradual classification models to obtain a refusal recognition model.

19. An on-line recognizing method for recognizing a named entity included in natural language, comprising the steps of:

recognizing the natural language with a forward gradual classification model to obtain a forward recognition result;

recognizing the natural language with a backward gradual classification model to obtain a backward recognition result;

generating a candidate named entity lattice according to said forward and backward recognition results; and

calculating an optimal path according to said generated candidate named entity lattice to output a named entity.

20. An off-line training system for recognizing a named entity included in natural language, comprising:

a local feature extracting device for having a supplied training text generate a named entity training sample denoted by a feature vector and a sample token;

a multiple-class support vector machine training device for training the training text to generate a gradual classification model;

a global feature extracting device for having the named entity training sample generate a character-based refusal recognition training sample denoted by a feature vector and a sample token;

a one-class support vector machine training device for refusal-recognition-training the obtained refusal recognition training sample to generate a refusal recognition classification model;

a training sample memory for storing the training texts used during the training.

21. An on-line recognizing system for recognizing a named entity included in natural language, comprising:

a local feature extracting device for having a provided training sample generate a local feature vector;

a multiple-class support vector machine recognizing device for recognizing the input text according to the local feature vector of the sample to obtain candidate named entities;

a global feature extracting device for extracting a global feature vector of the candidate named entities and the contexts thereof; and

a one-class support vector machine recognizing device for recognizing the input candidate named entities according to the global feature vector of the sample,

wherein said multiple-class support vector machine recognizing device uses a multiple-class classification model to test the input local feature vector so as to obtain its type token, and forms a candidate named entity according to a series of beginning and continuing tokens belonging to the same type of named entity; and

said one-class support vector machine recognizing device uses a one-class classification model to test the input global feature vector so as to obtain its testing score, substrates different thresholds from the obtained testing score to obtain a refusal recognition score, searches for an optimal path according to the refusal recognition score, and accepts the candidate named entities on the optimal path.

22. The system according to claim 21, further comprising:

a refusal recognition scoring device for determining one of the different thresholds according to the candidate named entity recognition results and the positional relationships among the candidate named entities obtained by said one-class support vector machine recognizing device so as to calculate a refusal recognition score, and accepting or refusing the candidate named entity according to the calculated refusal recognition score.

23. The system according to claim 21, further comprising:

an optimal path searching device for searching for an optimal path having a maximum sum of the refusal recognition scores according to the positions and the refusal recognition scores of the candidate named entities.