WORD ALIGNMENT METHOD AND SYSTEM FOR IMPROVED VOCABULARY COVERAGE IN STATISTICAL MACHINE TRANSLATION

• US 20110307245A1
• Filed: 06/14/2010
• Published: 12/15/2011
• Est. Priority Date: 06/14/2010
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Cancedda, Nicola, Dymetman, Marc, Hanneman, Gregory Alan
• Granted Patent: US 8,612,205 B2
• Time in Patent Office: Days
• US Class Current: 704/4
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment

Abstract

A system and method for generating word alignments from pairs of aligned text strings are provided. A corpus of text strings provides pairs of text strings, primarily sentences, in source and target languages. A first alignment between a text string pair creates links therebetween. Each link links a single token of the first text string to a single token of the second text string. A second alignment also creates links between the text string pair. In some cases, these links may correspond to bi-phrases. A modified first alignment is generated by selectively modifying links in the first alignment which include a word which is infrequent in the corpus, based on links generated in the second alignment. This results in removing at least some of the links for the infrequent words, allowing more compact and better quality bi-phrases, with higher vocabulary coverage, to be extracted for use in a machine translation system.

Claims

1. A method for generating word alignments from pairs of aligned text strings comprising:

from a corpus of text strings, receiving a pair of text strings comprising a first text string in a first language and a second text string in a second language;

with a first alignment tool, generating a first alignment between the first and second text strings which creates links between the first and second text string, each link linking a single token of the first text string to a single token of the second text string, the tokens of the first and second text strings including words;

with a second alignment tool, generating a second alignment between the first and second text strings which creates links between the first and second text strings, each link linking at least one token of the first text string to at least one token of the second text string,

generating a modified first alignment by selectively modifying links in the first alignment which include a word which is infrequent in the corpus, based on links generated in the second alignment.

2. The method of claim 1, wherein the generation of at least one of the first, second, and modified alignments is performed with a computer processor.

3. The method of claim 1, wherein the generation of the first alignment comprises:

generating a unidirectional source to target alignment;

generating a unidirectional target to source alignment; and

symmetrizing the two unidirectional alignments to generate the first alignment.

4. The method of claim 3, wherein the second alignment tool generates a symmetrical alignment without separately generating unidirectional source to target and target to source alignments.

5. The method of claim 1, wherein the first alignment tool comprises a GIZA++ alignment tool.

6. The method of claim 1, wherein the second alignment tool comprises at least one of an Anymalign alignment tool and a Berkeley alignment tool.

7. The method of claim 1, wherein the second alignment tool enforces a constraint not applied by the first alignment tool which limits which tokens of the first and second text strings are able to be aligned.

8. The method of claim 1, wherein the second alignment tool generates a bi-phrase alignment in which each alignment link includes at least one source token and at least one target token, and wherein for at least some alignments of text strings in the corpus, at least one of the links includes at least one of:

more than one source word from the source text string; and

more than one target word from the target text string.

9. The method of claim 8, wherein the modification retains links in the first alignment existing between the target and source words which are present in a bi-phrase alignment output by the second alignment tool.

10. The method of claim 1, wherein the generation of the modified alignment includes establishing a threshold frequency for infrequent words, and the selective modification of the first alignment is conditional at least on there being at least one link in the first alignment which includes a word which is present in the text corpus at no greater than the threshold frequency.

11. The method of claim 1, wherein the selective modification includes removing links in the first alignment which include an infrequent word and which are not present in the second alignment, either in a word link or in a phrase link which includes more than one word from at ea one of the target and source text strings.

12. The method of claim 1, wherein the selective modification is conditional on there being a bi-phrase identified in the second alignment to be used as a basis for the modification which has at least a threshold frequency k in the corpus or in a set of sub-corpora generated by sampling the corpus.

13. The method of claim 1, wherein the selective modification comprises identifying links in the first alignment to be retained which include the infrequent word and a linked target word where there is a corresponding link present in the second alignment which includes the infrequent word and the same linked target word and identifying for removal, at least a portion of the links in the first alignment which include the infrequent word and a linked target word for which there is no corresponding link between the infrequent word and the linked target word in the second alignment.

14. The method of claim 13, wherein the selective modification further comprises, for linked target words which are in a link to be retained, removing any other links which include that target word and a source word which are not identified for being retained.

15. The method of claim 1, further comprising storing the modified alignment in computer memory.

16. The method of claim 1, further comprising extracting bi-phrases based on the links in the modified alignment.

17. The method of claim 16, further comprising building a bi-phrase table based on the bi-phrases extracted from many of the text strings in the text corpus and a measure of their frequency in the text corpus.

18. The method of claim 17, further comprising constructing a statistical machine translation system incorporating the bi-phrase table.

19. A computer program product encoding instructions, which when executed by a computer, perform the method of claim 1.

20. A system for generating word alignments from word aligned text strings comprising:

memory which stores the first alignment tool, the second alignment tool, and instructions for performing the method of claim 1; and

a processor which executes the instructions.

21. A system for generating word alignments from word aligned text strings comprising:

instructions stored in memory for receiving a source sentence in a source language and a target sentence in a target language from a text corpus, the target sentence having been automatically identified as being a translation of the source sentence;

instructions for generating a first alignment between the source sentence and the target sentence by forming links, including some links that each link a source word with a target word;

instructions for generating a second alignment between the source sentence and the target sentence by forming links, including some links that each link at least one source word with at least one target word, the instructions for generating the second alignment generating alignments for sentence pairs in the corpus which include fewer links, on average, than the instructions for generating a first alignment;

instructions for identifying of links in the second alignment which comprise infrequent words and based on at least some of these identified links, modifying the first alignment to remove links between the infrequent words present in the second alignment links and other words of the first alignment which do not form a part of one of the identified second alignment links.

22. A method for generating word alignments from aligned sentences comprising:

receiving a source sentence in a source language and a target sentence in a target language from a text corpus, the target sentence having been automatically identified as being a translation of the source sentence;

with a processor, generating a word alignment between the source sentence and the target sentence by forming links, including some links that each link a source word with a target word;

generating a second alignment between the source sentence and the target sentence by a method which generates alignments for sentence pairs in the corpus which include fewer links, on average, than the method for generating the first alignment, the second alignment including some links that each link at least one source word with at least one target word;

identifying links in the second alignment which comprise infrequent words and based on at least some of these identified links, modifying the first alignment to remove links between the infrequent words present in the second alignment links and other words of the first alignment which do not form a part of one of the identified second alignment links.