Morphological analyzer, morphological analysis method, and morphological analysis program

• US 20040243409A1
• Filed: 03/30/2004
• Published: 12/02/2004
• Est. Priority Date: 05/30/2003
• Status: Abandoned Application

• Current Assignee: OKI Electric Industry Company Limited
• Original Assignee: OKI Electric Industry Company Limited
• Inventors: Nakagawa, Tetsuji
• Application Number:
  US10/812,000
• Publication Number:
  US 20040243409A1
• Time in Patent Office: Days
• US Class Current: 704/240
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/53 Processing of non-Latin text kana-to-kanji conversion G06F40/129; vowelisation G06F40/232
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models

Abstract

An input text is analyzed into morphemes by using a prescribed morphological analysis procedure to generate word strings with part-of-speech tags, including form information for parts of speech having forms, as hypotheses. The probabilities of occurrence of each hypothesis in a corpus of text are calculated by use of two or more part-of-speech n-gram models, at least one of which takes the forms of the parts of speech into consideration. Lexicalized models and class models may also be used. The models are weighted and the probabilities are combined according to the weights to obtain a single probability for each hypothesis. The hypothesis with the highest probability is selected as the solution to the morphological analysis. By combining multiple models, this method can resolve ambiguity with a higher degree of accuracy than methods that use only a single model.

Claims

1. A morphological analyzer comprising:

a hypothesis generator for applying a prescribed method of morphological analysis to a text and generating one or more hypotheses as candidate results of the morphological analysis, each hypothesis being a word string with part-of-speech tags, the part-of-speech tags including form information for parts of speech having forms;

a model storage facility storing information for a plurality of part-of-speech n-gram models, at least one of the part-of-speech n-gram models including information about the forms of the parts of speech;

a probability calculator for finding a probability that each said hypothesis will appear in a large corpus of text by using a weighted combination of the information for the part-of-speech n-gram models stored in the model storage facility; and

a solution finder for finding a solution among said hypotheses, based on the probabilities generated by the probability calculator.

2. The morphological analyzer of claim 1, wherein said at least one of the part-of-speech n-gram models including information about forms of parts of speech is a hierarchical part-of-speech n-gram model.

3. The morphological analyzer of claim 2, wherein the hierarchical part-of-speech n-gram model calculates a product of a conditional probability P(w_i|t_i) of occurrence of a word w_i given its part of speech t_i, a conditional probability P(t_i^form|t_i^pos) of occurrence of the part of speech t_i^pos of said word w_i in a form t_i^form shown by said word w_i, and a conditional probability P(t_i^pos|t_i−

N+1 . . . t<sub>i−

1</sub>) of occurrence of the part of speech t_i^pos of said word w_i following a part-of-speech tag string t<sub>i−

N+1</sub> . . . t<sub>i−

1</sub> indicating parts of speech of N−

1 preceding words, where N is a positive integer.

4. The morphological analyzer of claim 1, wherein at least one of the part-of-speech n-gram models is a lexicalized part-of-speech n-gram model.

5. The morphological analyzer of claim 4, wherein the lexicalized part-of-speech n-gram model calculates a product of a conditional probability P(w_i|t_i) of occurrence of a word w_i given its part of speech t_i and a conditional probability P(t_i|w_i−

N+1t<sub>i−

N+1</sub> . . . w<sub>i−

1</sub>t<sub>i−

1</sub>) of occurrence of the part of speech t_i of said word w_i following N−

1 words w<sub>i−

N+1</sub> . . . w<sub>i−

1</sub> having respective parts of speech t<sub>i−

N+1</sub> . . . t<sub>i−

1</sub>, where N is a positive integer.

6. The morphological analyzer of claim 4, wherein the lexicalized part-of-speech n-gram model calculates a conditional probability P(w_it_i|t_i−

N+1 . . . t<sub>i−

1</sub>) of occurrence of a word w_i having a part of speech t_i following a string of N−

1 parts of speech t<sub>i−

N+1</sub> . . . t<sub>i−

1</sub>, where N is a positive integer.

7. The morphological analyzer of claim 4, wherein the lexicalized part-of-speech n-gram model calculates a conditional probability P(w_it_i|w_i−

N+1t<sub>i−

N+1</sub> . . . w<sub>i−

1</sub>t<sub>i−

1</sub>) of occurrence of a word w_i having a part of speech t_i following a string of N−

1 words w<sub>i−

N+1</sub> . . . w<sub>i−

1</sub> having respective parts of speech t<sub>i−

N+1</sub> . . . t<sub>i−

1</sub>, where N is a positive integer.

8. The morphological analyzer of claim 1, wherein at least one of the part-of-speech n-gram models stored in the model storage facility is a class part-of-speech n-gram model.

9. The morphological analyzer of claim 8, wherein the class part-of-speech n-gram model calculates a product of a conditional probability P(w_i|t_i) of occurrence of a word w_i given its part of speech t_i and a conditional probability P(t_i|c_i−

N+1t<sub>i−

N+1</sub> . . . c<sub>i−

1</sub>t<sub>i−

1</sub>) of occurrence of said part of speech t_i following a string of N−

1 words assigned to respective classes c<sub>i−

N+1</sub> . . . c<sub>i−

1</sub> with respective parts of speech t<sub>i−

N+1</sub> . . . t<sub>i−

1</sub>, where N is a positive integer.

10. The morphological analyzer of claim 8, wherein the class part-of-speech n-gram model calculates a product of a conditional probability P(w_it_i|c_i−

N+1t<sub>i−

N+1</sub> . . . c<sub>i−

1</sub>t<sub>i−

1</sub>) of occurrence of a word w_i having a part of speech t_i following a string of N−

1 words in respective classes c<sub>i−

N+1</sub> . . . c<sub>i−

1</sub> with respective parts of speech t<sub>i−

N+1</sub> . . . t<sub>i−

1</sub>, where N is a positive integer.

11. The morphological analyzer of claim 8, wherein the class part-of-speech n-gram model is trained from both a part-of-speech tagged corpus and a part-of-speech untagged corpus.

12. The morphological analyzer of claim 1, further comprising a weight calculation unit using a leave-one-out method to calculate weights of the part-of-speech n-gram models.

13. A method of morphological analysis comprising:

applying a prescribed method of morphological analysis to a text and generating one or more hypotheses as candidate results of the morphological analysis, each hypothesis being a word string with part-of-speech tags, the part-of-speech tags including form information for parts of speech having forms;

calculating probabilities that each said hypothesis will appear in a large corpus of text by using a weighted combination of a plurality of part-of-speech n-gram models, at least one of the part-of-speech n-gram models including information about forms of parts of speech; and

finding a solution among said hypotheses, based on said probabilities.

14. The method of claim 13, wherein said at least one of the part-of-speech n-gram models including information about forms of parts of speech is a hierarchical part-of-speech n-gram model.

15. The method of claim 14, wherein the hierarchical part-of-speech n-gram model calculates a product of a conditional probability P(w_i|t_i) of occurrence of a word w_i given its part of speech t_i, a conditional probability P(t_i^form|t_i^pos) of occurrence of the part of speech t_i^pos of said word w_i in a form t_i^form shown by said word w_i, and a conditional probability P(t_i^pos|t_i−

N+1 . . . t<sub>i−

1</sub>) of occurrence of the part of speech t_i^pos of said word w_i following a part-of-speech tag string t<sub>i−

N+1</sub> . . . t<sub>i−

1</sub> indicating parts of speech of N−

1 preceding words, where N is a positive integer.

16. The method of claim 13, wherein at least one of the part-of-speech n-gram models is a lexicalized part-of-speech n-gram model.

17. The method of claim 13, wherein at least one of the part-of-speech n-gram models is a class part-of-speech n-gram model.

18. The method of claim 17, further comprising training the class part-of-speech n-gram model from both a part-of-speech tagged corpus and a part-of-speech untagged corpus.

19. The method of claim 13, further comprising using a leave-one-out method to calculate weights of the part-of-speech n-gram models.

20. A machine-readable medium storing a program comprising instructions that can be executed by a computing device to carry out morphological analysis by the method of claim 13.