Method and system for automatic management of reputation of translators

• US 10,402,498 B2
• Filed: 10/16/2018
• Issued: 09/03/2019
• Est. Priority Date: 05/25/2012
• Status: Active Grant

• Current Assignee: SDL Inc (RWS Holdings Plc)
• Original Assignee: SDL Inc (RWS Holdings Plc)
• Inventors: Marcu, Daniel, Dreyer, Markus
• Primary Examiner(s): Sharma, Neeraj
• Application Number:
  US16/161,651
• Publication Number:
  US 20190042566A1
• Time in Patent Office: 322 Days
• Field of Search: None
• CPC Class Codes: G06F 40/51 Translation evaluation
• CPC Class Codes: G06Q 10/0639 Performance analysis of employees; Performance analysis of enterprise or organisation operations

Abstract

The present invention provides a method that includes receiving a result word set in a target language representing a translation of a test word set in a source language. When the result word set is not in a set of acceptable translations, the method includes measuring a minimum number of edits to transform the result word set into a transform word set. The transform word set is in the set of acceptable translations. A system is provided that includes a receiver to receive a result word set and a counter to measure a minimum number of edits to transform the result word set into a transform word set. A method is provided that includes automatically determining a translation ability of a human translator based on a test result. The method also includes adjusting the translation ability of the human translator based on historical data of translations performed by the human translator.

Claims

1. A method for saving processor computation time and memory of a computer system during automated scoring of a language translation using computation of a hybrid translation edit rate (HyTER) score, the method comprising:

receiving a result word set in a target language representing a translation of a test word set in a source language and an exponentially sized reference set;

generating a translation hypothesis for the result word set;

developing a search space for automated computation of a HyTER score for the translation hypothesis using a Levenshtein distance calculation between pairs of the search space comprising allowed permutations of the translation hypothesis within a fixed window size and parts of the exponentially sized reference set, the search space comprising a lazy composition;

identifying a pair in the search space having a minimum edit distance and highest HyTER score from the automated computation of the HyTER score using the Levenshtein distance calculations within the fixed window size; and

outputting the automatically computed HyTER score and the allowed permutation of the translation hypothesis for the identified pair in the search space having the minimum edit distance and highest HyTER score, wherein the Levenshtein distance calculation is performed using the fixed window size so as to save the processor computation time and the memory of the computer system used for automated computation of the HyTER score.

2. The method according to claim 1, further comprising developing the search space for automated computation of the HyTER score, wherein the lazy composition is a weighted finite-state acceptor that represents a set of allowed permutations of the translation hypothesis and associated distance costs.

3. The method according to claim 1, further comprising calculating the HyTER score for the pairs in the search space to identify a pair in the search space having a minimum edit distance.

4. The method according to claim 1, further comprising reducing a number of pairs for the lazy composition for which the Levenshtein distance is calculated, using the fixed window constraints so as to save processor computation time and computer memory used for automated calculations of the HyTER score.

5. The method of claim 1, wherein calculating the HyTER score for each of the pairs in the search space further comprises saving computation time and memory by not explicitly constructing parts of the lazy composition.

6. The method according to claim 1, wherein the Levenshtein distance is calculated so as to save processor computation time and computer memory used for automated calculations of the HyTER score by constraining a number of paths constructed by the processor on demand by a weighted finite-state acceptor using a fixed window size, and not constructing permutation paths of the composition outside a window.

7. The method of claim 1, wherein the result word set is generated by a machine translation system.

8. The method of claim 7, wherein the translation hypothesis is provided by a machine translation system, and further comprising evaluating a quality of the machine translation system based on the minimum number of edits.

9. The method of claim 1, wherein when the translation hypothesis is in a set of acceptable translations of the exponentially sized reference set, the translation hypothesis is given a perfect score.

10. The method according to claim 1, wherein the exponentially sized reference set is encoded as a Recursive Transition Network stored in memory of the computing environment and expanded by the processor of the computing environment on demand.

11. The method of claim 10, wherein the minimum number of edits is determined by counting a number of substitutions, deletions, insertions, and moves required to transform the translation hypothesis into each encoded acceptable translation of the exponentially sized reference set of meaning equivalents expanded on demand from the Recursive Transition Network.

12. The method of claim 11, further comprising determining a normalized minimum number of edits by dividing the minimum number of edits by a number of words in the transformed word set.

13. The method of claim 1, further comprising forming a set of acceptable translations by combining at least a first subset of acceptable translations of the test word set provided by a first translator with a second subset of acceptable translations of the test word set provided by a second translator.

14. The method of claim 13, further comprising:

identifying at least first and second sub-parts of the test word set;

combining a first subset of acceptable translations of the first sub-part of the test word set provided by the first translator with a second subset of acceptable translations of the first sub-part of the test word set provided by the second translator;

combining a first subset of acceptable translations of the second sub-part of the test word set provided by the first translator with a second subset of acceptable translations of the second sub-part of the test word set provided by the second translator;

combining each one of the first and second subsets of acceptable translations of the first sub-part of the test word set with each one of the first and second subsets of acceptable translations of the second sub-part of the test word set to form a third subset of acceptable translations of the word set;

and adding the third subset of acceptable translations to the set of acceptable translations.

15. A system for saving processor computation time and computer memory of the system during automated scoring of a language translation using computation of a hybrid translation edit rate (HyTER) score, the system comprising:

a memory for storing executable instructions, a result word set in a target language representing a translation of a test word set in a source language, and an exponentially sized reference set; and

a processor for executing the instructions stored in the memory, the executable instructions comprising;

receiving a result word set in a target language representing a translation of a test word set in a source language and an exponentially sized reference set;

generating a translation hypothesis for the result word set;

developing a search space for automated computation of a HyTER score for the translation hypothesis using a Levenshtein distance calculation between pairs of the search space comprising allowed permutations of the translation hypothesis within a fixed window and parts of the exponentially sized reference set, the search space comprising a lazy composition,

identifying a pair in the search space having a minimum edit distance and highest HyTER score from the automated computation of the HyTER score using the Levenshtein distance calculations within the fixed window; and

outputting the automatically computed HyTER score and the allowed permutation of the translation hypothesis for the identified pair in the search space having a minimum edit distance and highest HyTER score, wherein the Levenshtein distance calculation is performed using the fixed window so as to save the processor computation time and the computer memory of the system used for automated calculations of the HyTER score.

16. The system of claim 15, wherein the result word set is received from a human translator, and wherein a translation ability of the human translator based on the HyTER score is output to the human translator.

17. The system of claim 16, wherein a test result is stored in the memory as an indicator of a translation ability of the human translator, and wherein the translation ability of the human translator is adjusted based on at least one of:

price data related to at least one translation completed by the human translator;

an average time to complete translations by the human translator;

a customer satisfaction rating of the human translator;

a number of translations completed by the human translator; and

a percentage of projects completed on-time by the human translator.

18. The system of claim 15, further comprising a machine translator interface for receiving the result word set from a machine translator, wherein a quality of the machine translator is evaluated based on the minimum number of edits.

19. The system of claim 18, wherein when the minimum edit distance for the identified pair is zero, the result word set is given a perfect HyTER score.

20. The system of claim 19, wherein the minimum number of edits to transform the result word set into the transform word set comprises a minimum number of substitutions, deletions, insertions, and moves, and further comprising a transformer to identify the minimum number of substitutions, deletions, insertions, and moves.