System and method for productive generation of compound words in statistical machine translation

• US 8,781,810 B2
• Filed: 07/25/2011
• Issued: 07/15/2014
• Est. Priority Date: 07/25/2011
• Status: Expired due to Fees

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Cancedda, Nicola, Stymne, Sara
• Primary Examiner(s): SPOONER, LAMONT M
• Application Number:
  US13/189,738
• Publication Number:
  US 20130030787A1
• Time in Patent Office: 1,086 Days
• Field of Search: 704 2- 8, 715/264
• US Class Current: 704/2
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06F 40/56 Natural language generation

Abstract

A method and a system for making merging decisions for a translation are disclosed which are suited to use where the target language is a productive compounding one. The method includes outputting decisions on merging of pairs of words in a translated text string with a merging system. The merging system can include a set of stored heuristics and/or a merging model. In the case of heuristics, these can include a heuristic by which two consecutive words in the string are considered for merging if the first word of the two consecutive words is recognized as a compound modifier and their observed frequency f₁ as a closed compound word is larger than an observed frequency f₂ of the two consecutive words as a bigram. In the case of a merging model, it can be one that is trained on features associated with pairs of consecutive tokens of text strings in a training set and predetermined merging decisions for the pairs. A translation in the target language is output, based on the merging decisions for the translated text string.

Claims

1. A method for making merging decisions for a translation comprising:

providing a translated text string in a target language of a source text string in a source language;

with a merging system, wherein the merging system is implemented with a computer processor, outputting decisions on merging of pairs of words in the translated text string, the merging system comprising at least one of;

a set of stored heuristics comprising at least a first heuristic by which two consecutive words in the string are considered for merging if an observed frequency f₁ of the two consecutive words as a closed compound word is larger than an observed frequency f₂ of the two consecutive words as a bigram, and

a merging model trained on features associated with pairs of consecutive tokens of text strings in a training set and predetermined merging decisions for the pairs to predict merging decisions for a new translated text string; and

outputting a translation in the target language based on the merging decisions for the translated text string.

2. The method of claim 1, wherein the merging system comprises the set of heuristics.

3. The method of claim 2, wherein in the first heuristic, two consecutive words in the string are considered for merging the first word of the two consecutive words is recognized as a compound modifier and if the observed frequency f₁ of the two consecutive words as a closed compound word is larger than the observed frequency f₂ of the two consecutive words as a bigram.

4. The method of claim 3, wherein in applying the first heuristic a list of recognized compound modifiers is accessed, the list including compound modifiers observed when compound words in a training corpus in the target language were split using predetermined splitting rules.

5. The method of claim 2, wherein the set of heuristics further includes a second heuristic by which two consecutive words in the string are considered for merging based on part of speech tags assigned to the two words, the part of speech tags being selected from a restricted part of speech tagset, at least one of the restricted part of speech tags being configured for identifying a word from the target language as being one which also forms a part of a known closed compound word in the target language.

6. The method of claim 5, wherein two consecutive words in the target text string are merged if at least one of the first and second heuristics considers the two consecutive words for merging.

7. The method of claim 1, wherein the merging system comprises the merging model, the method further comprising extracting features for pairs of consecutive words in the translated target string.

8. The method of claim 7, wherein the features are selected from the group consisting of features based on:

a. a surface word form;

word−

1, word+1, where word−

1 represents a first word of a pair of consecutive words in the translated text string and word+1 represents a second word of the pair, positioned after the first word;

b. parts-of-speech;

POS−

1, POS+1, where POS−

1 is a part of speech assigned to word−

1 and POS+1 is a part of speech assigned to word+1;

c. a character n-gram comprising n characters from the end of word−

1 and m characters from the start of word+1, wherein m+n is at least 2;

d. a character n-gram comprising n characters from the end of word−

1 and m characters from the start of word+1, wherein the character n-grams are normalized by replacing characters are with phonetic approximations, and grouped into groups with similar phonetic distribution; and

e. a frequency, in a training corpus, of at least one of;

a bigram represented by word−

1 word+1,

a closed compound word derived from word−

1 and word+1,

a closed compound word derived from word−

1 and another word, and

a closed compound derived word from word+1 and another word.

9. The method of claim 7, wherein the merging model has been trained automatically on a set of pairs of target text strings, each pair comprising a first target text string and a second target text string in which at least one consecutive pair of the words in the first string are identified to be merged to form a closed compound word.

10. The method of claim 9, wherein at least some of the pairs of target text strings are artificially generated using a heuristic for forming closed compound words.

11. The method of claim 7, wherein the merging model is one which has been trained with a sequence labeling machine learning algorithm.

12. The method of claim 7, further comprising automatically generating training data for training the merging model.

13. The method of claim 12, wherein the automatic generating training data for training the merging model comprises applying a stored set of merging heuristics to decomposed text in the target language and extracting merging decisions therefrom.

14. The method of claim 7, wherein the merging model computes an optimal set of merging decisions over all pairs of words in the target text string.

15. The method of claim 1, wherein each of the decisions on merging includes a decision on whether first and second words are to be merged and optionally, for words which are to be merged, a merging rule which specifies a local modification of at least one of the words at their boundary, the merging rule being selected form a predetermined set of merging rules.

16. The method of claim 1, wherein the providing of the translated text string in the target language of the source text string in the source language comprises, translating the source text string with a decoder which has been trained on a parallel training corpus comprising source language text strings and decomposed target language text strings formed by decomposing closed compound words of target language text strings corresponding to the source language text strings.

17. A computer program product comprising a non-transitory computer-readable recording medium which stores instructions for performing the method of claim 1.

18. A system for making merging decisions comprising memory which stores instructions for performing the method of claim 1 and a processor in communication with the memory for executing the instructions.

19. A translation system comprising:

a decoder which receives as input a source text string in a source language and outputs a translated text string in a target language in a target language, based on the source text string;

a merging system, which receives the translated text string and outputs a translation in the target language based on translated text string, the merging system being configured for outputting decisions on merging of pairs of words in the translated text string, the merging system comprising at least one of;

a set of stored heuristics comprising at least a first heuristic by which two consecutive words in the string are considered for merging if the observed frequency f₁ of the two consecutive words as a closed compound word is larger than an observed frequency f₂ of the two consecutive words as a bigram, and, optionally, if the first word of the two consecutive words is also recognized as a compound modifier, and

a merging model trained on features associated with pairs of consecutive tokens of text strings in a training corpus and predetermined merging decisions for the pairs to predict merging decisions for a new translated text string.

20. The system of claim 19, wherein the merging system comprises software instructions stored in memory and a processor for executing the instructions.

21. The system of claim 19, further comprising memory which stores a list of closed compounds in the training corpus and their associated frequencies f₁ and f₂.

22. The system of claim 19, wherein the translated text strings output by the decoder have fewer closed compound words, on average, than the corresponding translations output by the merging system.