Diagnostic evaluation of machine translators

• US 8,185,377 B2
• Filed: 06/27/2008
• Issued: 05/22/2012
• Est. Priority Date: 08/11/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhou, Ming
• Primary Examiner(s): Vo, Huyen X.
• Application Number:
  US12/147,750
• Publication Number:
  US 20090157386A1
• Time in Patent Office: 1,425 Days
• Field of Search: 704 1- 10, 704/270, 704/277, 704/270.1, 704/251, 704/257, 704/272
• US Class Current: 704/9
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30

Abstract

A system for evaluating translation quality of a machine translator is discussed. The system includes a bilingual data generator configured to intermittently access a wide area network and generate a bilingual corpus from data received from the wide area network. The method also includes an example extraction component configured to receive an ontology input indicative of a plurality of ontological categories of evaluation and to extract evaluation examples from the bilingual corpus based on the ontology input. The system further includes an evaluation component configured to evaluate translation results from translation by a machine translator of the evaluation examples and to score the translation results according to the ontological categories.

Claims

1. A system for evaluating translation quality of a machine translator, comprising:

a bilingual data generator configured to intermittently access a wide area network and generate a bilingual corpus from data received from the wide area network;

a linguistic analysis component configured to add otherwise latent linguistic information to the bilingual corpus to obtain an augmented bilingual corpus;

an example extraction component configured to receive an ontology input indicative of a plurality of ontological categories of evaluation and to extract evaluation examples from the augmented bilingual corpus based on the ontology input; and

a processor that is component of a computing device that utilizes an evaluation component to evaluate translation results from translation by a machine translator of the evaluation examples and to score the translation results according to the ontological categories.

2. The system of claim 1 wherein the bilingual corpus includes source data representative of text in a source language and corresponding target data representative of text in a target language and wherein the example extraction component identifies a plurality of checkpoints with each checkpoint indicative of a portion of the source data.

3. The system of claim 2 wherein the evaluation component is configured to score the translation results by producing a plurality of checkpoint scores and wherein each checkpoint score is indicative of the quality of the translation results for a given checkpoint.

4. The system of claim 3 wherein the evaluation component is configured to develop a total score based on the plurality of checkpoint scores.

5. The system of claim 3 wherein at least one of the checkpoint scores is indicative of the syntactic similarity between the translation results and the target data associated with the checkpoint.

6. The system of claim 3 wherein at least one of the checkpoint scores is indicative of a morphological similarity between the translation results and the target data associated with the checkpoint.

7. A method of evaluating translation quality of a machine translator, comprising:

intermittently generating and updating a bilingual corpus, augmented with linguistic information generated from performing semantic analysis of the bilingual corpus;

when the bilingual corpus is generated, extracting evaluation examples, each evaluation example being extracted to test the machine translator in one of a plurality of predefined categories of evaluation;

receiving translation results from the machine translator, for the evaluation examples; and

utilizing a processor that is a component of a computing device to evaluate the translation results in each of the predefined categories of evaluation; and

storing evaluation results for display to a user.

8. The method of claim 7 and further comprising:

displaying the evaluation examples for editing by the user.

9. The system of claim 8 wherein at least one of the checkpoint scores is indicative of a match between the translation results and the target data associated with the checkpoint.

10. The method of claim 7 and further comprising:

displaying the evaluation results to the user in a hierarchical structure that indicates a relationship among the categories of evaluation.

11. The method of claim 7 wherein extracting evaluation examples comprises:

exposing the evaluation examples to a community over a network; and

receiving additional evaluation examples from the community over the network.

12. The method of claim 7 wherein generating and intermittently updating the bilingual corpus comprises:

augmenting the bilingual corpus with the linguistic information to more specifically assign each of the evaluation examples to an individual category of evaluation.

13. The method of claim 7 wherein receiving the translation results comprises:

receiving the translation results over a network and further comprising displaying the translation results to the user over a wide area network.

14. The method of claim 7 and further comprising:

repeating the step of extracting each time the bilingual corpus is updated.

15. The method of claim 7 and further comprising:

calculating a total score for the machine translator.

16. The method of claim 7 wherein intermittently generating and updating the bilingual corpus comprises:

receiving from a data source a representation of textual data in a source language and a corresponding target language.

17. The method of claim 16 wherein intermittently generating and updating the bilingual corpus further comprises:

extracting a plurality of translation pairs from the representation with each translation pair including data in the source language and data corresponding to a translation in the target language.

18. The method of claim 17 wherein intermittently generating and updating the bilingual corpus further comprises:

selecting a checkpoint from the plurality of translation pairs by assigning an ontological category to the source language portion of the translation pair; and

storing the checkpoint in an evaluation corpus.

19. The method of claim 18 wherein selecting the checkpoint from the plurality of translation pairs includes employing a grammar rule to select the checkpoint.

20. The method of claim 18 wherein selecting the checkpoint from the plurality of translation pairs includes employing a dictionary to select the checkpoint.