Providing machine-generated translations and corresponding trust levels

• US 8,380,486 B2
• Filed: 10/01/2009
• Issued: 02/19/2013
• Est. Priority Date: 10/01/2009
• Status: Active Grant

• Current Assignee: SDL PLC (RWS Holdings Plc)
• Original Assignee: Language Weaver, Inc. (RWS Holdings Plc)
• Inventors: Soricut, Radu, Viswanathan, Narayanaswamy, Marcu, Daniel
• Primary Examiner(s): Neway, Samuel G
• Application Number:
  US12/572,021
• Publication Number:
  US 20110082683A1
• Time in Patent Office: 1,237 Days
• Field of Search: 704 1- 10
• US Class Current: 704/2
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation

Abstract

A quality-prediction engine predicts a trust level associated with translational accuracy of a machine-generated translation. Training a quality-prediction may include translating a document in a source language to a target language by executing a machine-translation engine stored in memory to obtain a machine-generated translation. The training may further include comparing the machine-generated translation with a human-generated translation of the document. The human-generated translation is in the target language. Additionally, the training may include generating a mapping between features of the machine-generated translation and features of the human-generated translation based on the comparison.

Claims

1. A method for training a quality-prediction engine, the method comprising:

translating a document in a source language to a target language by executing a machine-translation engine stored in memory to obtain a machine-generated translation;

comparing the machine-generated translation with a human-generated translation of the document, the human-generated translation in the target language;

generating a mapping between features of the machine-generated translation and features of the human-generated translation based on the comparison, the mapping allowing determination of trust levels associated with translational accuracy of future machine-generated translations that lack corresponding human-generated translations; and

calibrating the quality prediction engine, wherein calibrating the quality-prediction engine includes;

obtaining a plurality of opinions for a plurality of sample translations generated by execution of the machine-translation engine, each of the opinions from a human and indicating a perceived trust level of corresponding sample translations;

using the quality-prediction engine to determine a trust level of each of the plurality of sample translations;

determining a relationship between the plurality of opinions and the trust levels of the plurality of sample translations; and

tuning the mapping to minimize any difference between the plurality of opinions and the trust levels of the plurality of sample translations.

2. The method of claim 1, wherein the translational accuracy represents an alignment-estimation between the machine-generated translation and a prospective human-generated translation.

3. The method of claim 1, wherein the trust level of each of the plurality of sample translations is indicated as one of a numerical scale, a term based scale, a star-rating scale, or an analog scale.

4. The method of claim 1, wherein determining the trust level of each of the plurality of sample translations includes analyzing discrete units of the plurality of sample translations to determine a trust level for each discrete unit.

5. The method of claim 4, wherein a discrete unit comprises one of a sentence, a word, a phrase, or a paragraph.

6. The method of claim 1, wherein the quality-prediction engine overestimates the trust level of the plurality of sample translations.

7. The method of claim 1, wherein the quality-prediction engine underestimates the trust level of the plurality of sample translations.

8. A method for training a quality-prediction engine, the method comprising:

translating a document in a source language to a target language by executing a machine-translation engine stored in memory to obtain a machine-generated translation;

comparing the machine-generated translation with a human-generated translation of the document, the human-generated translation in the target language;

generating a mapping between features of the machine-generated translation and features of the human-generated translation based on the comparison, the mapping allowing determination of trust levels associated with translational accuracy of future machine-generated translations that lack corresponding human-generated translations; and

calibrating the quality prediction engine, wherein calibrating the quality prediction engine includes;

obtaining a plurality of opinions for a plurality of sample translations generated by execution of the machine-translation engine, each of the opinions from a human and indicating a perceived trust level of corresponding sample translations;

using the quality-prediction engine to determine a trust level of each of the plurality of sample translations;

determining a relationship between the plurality of opinions and the trust levels of the plurality of sample translations; and

tuning the mapping to minimize any difference between the plurality of opinions and the trust levels of the plurality of sample translations, and wherein calibrating the quality-prediction engine is automatically triggered to ensure that determined trust levels are continually consistent with user feedback.

9. The method of claim 8, wherein calibrating the quality-prediction engine includes obtaining user feedback.

10. A system for training a quality-prediction engine, the system comprising:

a processor;

a machine-translation engine stored in memory and executable by a processor to translate a document in a source language to a target language to obtain a machine-generated translation;

a feature-comparison module stored in memory and executable by a processor to compare the machine-generated translation with a human-generated translation of the document, the human-generated translation in the target language;

a mapping module stored in memory and executable by a processor to generate a mapping between features of the machine-generated translation and features of the human-generated translation based on the comparison, the mapping allowing determination of trust levels associated with translational accuracy of future machine-generated translations that lack corresponding human-generated translations; and

a calibration module stored in memory and executable by a processor to calibrate the quality-prediction engine;

wherein the calibration module;

obtains a plurality of opinions for a plurality of sample translations generated by execution of the machine-translation engine, each of the opinions from a human and indicating a perceived trust level of corresponding sample translations;

uses the quality-prediction engine to determine a trust level of each of the plurality of sample translations;

determines a relationship between the plurality of opinions and the trust levels of the plurality of sample translations; and

tunes the mapping to minimize any difference between the plurality of opinions and the trust levels of the plurality of sample translations.

11. The system of claim 10, wherein the translational accuracy represents an alignment-estimation between the machine-generated translation and a prospective human-generated translation.

12. The system of claim 10, wherein the quality-prediction engine indicates the trust level of each of the plurality of sample translations as one of a numerical scale, a term based scale, a star-rating scale, or an analog scale.

13. The system of claim 10, wherein the quality prediction engine includes an analysis module stored in memory and executable by a processor to analyze discrete units of the plurality of sample translations to determine a trust level for each discrete unit.

14. The system of claim 13, wherein a discrete unit comprises one of a sentence, a word, a phrase, or a paragraph.

15. The system of claim 10, wherein the quality-prediction engine overestimates the trust level of the plurality of sample translations.

16. The system of claim 10, wherein the quality-prediction engine underestimates the trust level of the plurality of sample translations.

17. A system for training a quality-prediction engine, the system comprising:

a processor;

a machine-translation engine stored in memory and executable by a processor to translate a document in a source language to a target language to obtain a machine-generated translation;

a feature-comparison module stored in memory and executable by a processor to compare the machine-generated translation with a human-generated translation of the document, the human-generated translation in the target language;

a mapping module stored in memory and executable by a processor to generate a mapping between features of the machine-generated translation and features of the human-generated translation based on the comparison, the mapping allowing determination of trust levels associated with translational accuracy of future machine-generated translations that lack corresponding human-generated translations; and

a calibration module stored in memory and executable by a processor to calibrate the quality-prediction engine;

wherein the calibration module;

obtains a plurality of opinions for a plurality of sample translations generated by execution of the machine-translation engine, each of the opinions from a human and indicating a perceived trust level of corresponding sample translations;

uses the quality-prediction engine to determine a trust level of each of the plurality of sample translations;

determines a relationship between the plurality of opinions and the trust levels of the plurality of sample translations; and

tunes the mapping to minimize any difference between the plurality of opinions and the trust levels of the plurality of sample translations;

wherein the quality-prediction engine is automatically calibrated to ensure that determined trust levels are continually consistent with user feedback.

18. The system of claim 17, further comprising an interface module stored in memory and executable by a processor to obtain user feedback.

19. A non-transitory computer readable storage medium having a program embodied thereon, the program executable by a processor to perform a method for training a quality-prediction engine, the method comprising:

translating a document in a source language to a target language using a machine-translation engine to obtain a machine-generated translation;

comparing the machine-generated translation with a human-generated translation of the document, the human-generated translation in the target language;

generating a mapping between features of the machine-generated translation and features of the human-generated translation based on the comparison, the mapping allowing determination of trust levels associated with translational accuracy of future machine-generated translations that lack corresponding human-generated translations; and

calibrating the quality prediction engine, wherein calibrating the quality-prediction engine includes;

obtaining a plurality of opinions for a plurality of sample translations generated by execution of the machine-translation engine, each of the opinions from a human and indicating a perceived trust level of corresponding sample translations;

using the quality-prediction engine to determine a trust level of each of the plurality of sample translations;

determining a relationship between the plurality of opinions and the trust levels of the plurality of sample translations; and

tuning the mapping to minimize any difference between the plurality of opinions and the trust levels of the plurality of sample translations.

20. A non-transitory computer readable storage medium having a program embodied thereon, the program executable by a processor to perform a method for training a quality-prediction engine, the method comprising:

translating a document in a source language to a target language using a machine-translation engine to obtain a machine-generated translation;

comparing the machine-generated translation with a human-generated translation of the document, the human-generated translation in the target language;

generating a mapping between features of the machine-generated translation and features of the human-generated translation based on the comparison, the mapping allowing determination of trust levels associated with translational accuracy of future machine-generated translations that lack corresponding human-generated translations; and

calibrating the quality prediction engine, wherein calibrating the quality-prediction engine includes;

obtaining a plurality of opinions for a plurality of sample translations generated by execution of the machine-translation engine, each of the opinions from a human and indicating a perceived trust level of corresponding sample translations;

using the quality-prediction engine to determine a trust level of each of the plurality of sample translations;

determining a relationship between the plurality of opinions and the trust levels of the plurality of sample translations; and

tuning the mapping to minimize any difference between the plurality of opinions and the trust levels of the plurality of sample translations, wherein calibrating the quality-prediction engine is automatically triggered to ensure that determined trust levels are continually consistent with user feedback.