Method and apparatus for evaluating machine translation quality

• US 7,587,307 B2
• Filed: 12/18/2003
• Issued: 09/08/2009
• Est. Priority Date: 12/18/2003
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Yamada, Kenji, Cancedda, Nicola
• Primary Examiner(s): Dorvil, Richemond
• Assistant Examiner(s): Yen, Eric
• Application Number:
  US10/737,972
• Publication Number:
  US 20050137854A1
• Time in Patent Office: 2,091 Days
• Field of Search: 704/2, 704/9
• US Class Current: 704/2
• CPC Class Codes: G06F 16/3347 using vector based model
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 40/51 Translation evaluation

Abstract

Quality of machine translation of natural language is determined by computing a sequence kernel that provides a measure of similarity between a first sequence of symbols representing a machine translation in a target natural language and a second sequence of symbols representing a reference translation in the target natural language. The measure of similarity takes into account the existence of non-contiguous subsequences shared by the first sequence of symbols and the second sequence of symbols. When the similarity measure does not meet an acceptable threshold level, the translation model of the machine translator may be adjusted to improve subsequent translations performed by the machine translator.

Claims

1. A method for computing machine translation performance, comprising:

receiving a sequence of natural language data in a first language;

translating the sequence of natural language data to a second language to define a machine translation of the sequence of natural language data comprising symbols;

receiving a reference translation of the sequence of natural language data in the second language comprising symbols;

with a computer processor, computing a sequence kernel that provides a similarity measure between the machine translation and the reference translation based on occurrences of subsequences that are shared by the machine translation and the reference translation, for a selected subsequence length, including performing an inner product in a feature space of all possible subsequences of a selected subsequence length;

outputting a signal indicating the similarity measure;

wherein the similarity measure accounts for non-contiguous occurrences of subsequences of the selected subsequence length that are shared between the machine translation and the reference translation, in which the non-contiguous subsequences share symbols and comprise a gap of at least one symbol which has been determined not to match a symbol in the gap of the other.

2. The method according to claim 1, wherein the similarity measure penalizes identified non-contiguous occurrences of subsequences shared between the machine translation and the reference translation.

3. The method according to claim 2, wherein the similarity measure penalizes identified non-contiguous occurrences of subsequences shared between the machine translation and the reference translation over a range of subsequence lengths.

4. The method according to claim 1, wherein the similarity measure scores symbols in matching subsequences in the machine translation and the reference translation with a different weight than symbols in gaps that do not match.

5. The method according to claim 1, wherein the similarity measure scores matching non-contiguous subsequences in the machine translation and the reference translation with a different weight depending on what symbol or symbols form the matching non-contiguous subsequences or exist in the gaps thereof.

6. The method according to claim 1, wherein the similarity measure scores non-matching subsequences in the machine translation and the reference translation using synonymy data.

7. The method according to claim 6, wherein the synonymy data is a collection of symbols in the second language that have similar meaning.

8. The method according to claim 1, further comprising transforming the sequence of natural language data in the first language to textual data.

9. The method according to claim 8, wherein the sequence of natural language data in the first language is voice data.

10. The method according to claim 1, wherein the reference translation of the sequence of natural language data in the first language is input by a user.

11. The method according to claim 1, further comprising computing the sequence kernel similarity measure by computing a score for every pair of symbols shared between the machine translation and the reference translation.

12. The method according to claim 11, wherein the score for every pair of symbols is computed using synonymy data.

13. The method according to claim 1, wherein the sequence kernel similarity measure is computed by representing language data using an alphabet given by one of:

a set of characters, a set of syllables, a set of words, and a set of word stems.

14. The method according to claim 1, further comprising:

using a machine translator having a translation model to define the machine translation of the sequence of natural language data;

wherein the machine translator is adapted to adjust parameters of the machine translator responsive to receiving the output signal indicating the similarity measure is below a threshold performance level.

15. The method according to claim 1, further comprising using a translation model for translating the sequence of natural language data to the second language.

16. The method according to claim 15, wherein the translation model is updated using the sequence of natural language data and the reference translation in response to receiving the signal indicating the similarity measure is below a threshold performance level.

17. The method according to claim 1, wherein the sequence kernel is given by the following equation:

18. The method according to claim 1, wherein the sequence kernel is given by the following equation:

19. A method for computing machine translation performance, comprising:

receiving into computer readable memory a sequence of natural language data in a first language;

translating the sequence of natural language data to a second language to define a machine translation of the sequence of natural language data;

receiving a reference translation of the sequence of natural language data in the second language;

computing a sequence kernel that provides a similarity measure between the machine translation and the reference translation which includes performing an inner product in a feature space of all possible subsequences of a selected subsequence length;

outputting a signal indicating the similarity measure;

wherein the similarity measure accounts for non-contiguous occurrences of subsequences shared between the machine translation and the reference translation and wherein non-consecutive sequences with gaps, the gaps comprising at least one symbol, are scored differently depending on what symbol or symbols occupies the gaps.

20. A method for computing machine translation performance, comprising:

receiving a sequence of natural language data in a first language;

translating the sequence of natural language data to a second language to define a machine translation of the sequence of natural language data comprising symbols;

receiving a reference translation of the sequence of natural language data in the second language comprising symbols;

computing a sequence kernel that provides a similarity measure between the machine translation and the reference translation;

outputting a signal indicating the similarity measure;

wherein the similarity measure accounts for non-contiguous occurrences of subsequences shared between the machine translation and the reference translation, in which the non-contiguous subsequences share symbols and comprise a gap of at least one symbol which has been determined not to match a symbol in the gap of the other, and

wherein the sequence kernel is given by the following equation;

21. A system for computing machine translation performance, comprising:

means for receiving a sequence of natural language data in a first language;

means for translating the sequence of natural language data to a second language to define a machine translation of the sequence of natural language data comprising symbols;

means for receiving a reference translation of the sequence of natural language data in the second language comprising symbols;

means including a computer processor for computing a sequence kernel that provides a similarity measure between the machine translation and the reference translation which includes performing an inner product in a feature space of all possible subsequences of a selected subsequence length;

means for outputting a signal indicating the similarity measure;

wherein the similarity measure computed by said computing means accounts for non-contiguous occurrences of subsequences shared between the machine translation and the reference translation in which the non-contiguous subsequences share symbols and comprise a gap of at least one symbol which has been determined not to match a symbol in the gap of the other.

22. An article of manufacture for use in a machine comprising:

a) a memory;

b) instructions stored in the memory for a method of computing machine translation performance, the method comprising;

receiving a sequence of natural language data in a first language;

translating the sequence of natural language data to a second language to define a machine translation of the sequence of natural language data comprising symbols;

receiving a reference translation of the sequence of natural language data in the second language comprising symbols;

computing a sequence kernel that provides a similarity measure between the machine translation and the reference translation which includes performing an inner product in a feature space of all possible subsequences of a selected subsequence length;

outputting a signal indicating the similarity measure;

wherein the similarity measure accounts for non-contiguous occurrences of subsequences shared between the machine translation and the reference translation, in which the non-contiguous subsequences share symbols and comprise a gap of at least one symbol which has been determined not to match a symbol in the gap of the other.