UNSUPERVISED CHINESE WORD SEGMENTATION FOR STATISTICAL MACHINE TRANSLATION

• US 20090326916A1
• Filed: 06/27/2008
• Published: 12/31/2009
• Est. Priority Date: 06/27/2008
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Xu, Jia, Gao, Jianfeng, Toutanova, Kristina Nikolova
• Application Number:
  US12/163,119
• Publication Number:
  US 20090326916A1
• Time in Patent Office: Days
• US Class Current: 704/4
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment
• CPC Class Codes: G06F 40/53 Processing of non-Latin text kana-to-kanji conversion G06F40/129; vowelisation G06F40/232

Abstract

Described is using a generative model in processing an unsegmented sentence into a segmented sentence. A segmenter includes the generative model, which given an unsegmented sentence (e.g., in Chinese) provides candidate segmented sentences to a probability-based decoder that selects the segmented sentence. For example, the segmented (e.g., Chinese-language) sentence may be provided to a statistical machine translator that outputs a translated (e.g., English-language) sentence. The generative model may include a word sub-model that generates hidden words using a word model, a spelling sub-model that generates characters from the hidden words, and an alignment sub-model that generates translated words and alignment data from the characters. The word sub-model may correspond to a unigram model having words and associated frequency data therein, and the alignment sub-model may correspond to a word aligned corpus having source sentence, translated target sentence pairings therein. Training is also described.

Claims

1. In a computing environment, a method comprising:

receiving an unsegmented sentence; and

segmenting the unsegmented sentence into a segmented sentence via a segmenter that includes a generative model.

2. The method of claim 1 wherein the segmenter further includes a decoder that obtains candidate segmented sentences from the generative model and selects as the segmented sentence a candidate segmented sentence based on probability.

3. The method of claim 1 further comprising, providing the segmented sentence to a translator and receiving a translated sentence from the translator.

4. The method of claim 3 wherein the unsegmented sentence comprises a Chinese-language sentence, and wherein the translated sentence comprises an English-language sentence.

5. The method of claim 1 wherein segmenting the unsegmented sentence comprises, generating hidden words using a word model, generating characters from the hidden words, and generating the translated sentence from the characters.

6. The method of claim 1 further comprising, providing the generative model by combining sub-models, including a word sub-model, a spelling sub-model and an alignment sub-model.

7. The method of claim 6 further comprising, generating a hidden string of words via the word sub-model.

8. The method of claim 7 wherein generating the hidden string of words comprises modeling the words as hidden variables, and inferring the hidden variables via Gibbs sampling.

9. The method of claim 7 further comprising, providing the hidden string of words to the spelling sub-model, and generating a string of characters from the hidden string of words via the spelling sub-model.

10. The method of claim 6 further comprising, providing a string of characters to the alignment sub-model.

11. The method of claim 1 further comprising, providing the generative model, including using a parameter set to combine sub-models.

12. The method of claim 11 wherein the parameter set corresponds to word probability, length probability, alignment probability or translation probability, or any combination of word probability, length probability, alignment probability or translation probability.

13. The method of claim 11 further comprising, processing training data to obtain the parameter set.

14. In a computing environment, a system comprising, a generative model, including a word sub-model that generates hidden words using a word model, a spelling sub-model that generates characters from the hidden words, and an alignment sub-model that generates translated words and alignment data from the characters.

15. The system of claim 14 further comprising, a segmenter that includes the generative model for segmenting an unsegmented sentence into candidates, and a decoder for processing the candidates to select the best candidate as a segmented sentence.

16. The system of claim 14 further comprising, means for determining a parameter set used by the generative model in combining sub-models.

17. The system of claim 14 wherein the word sub-model corresponds to a unigram model having words and associated frequency data therein, and wherein the alignment sub-model corresponds to a word aligned corpus having source sentence, translated target sentence pairings therein.

18. In a computing environment, a method comprising, configuring a generative model for use in segmenting an unsegmented sentence, including generating hidden words using a word model, generating characters from the hidden words, and generating candidate segmented sentences from the characters.

19. The method of claim 18 further comprising, selecting a candidate as a segmented sentence based on probability data.

20. The method of claim 18 further comprising, using training data to configure at least part of the generative model.