Language capability assessment and training apparatus and techniques

• US 10,319,252 B2
• Filed: 11/09/2005
• Issued: 06/11/2019
• Est. Priority Date: 11/09/2005
• Status: Active Grant

• Current Assignee: SDL PLC (RWS Holdings Plc)
• Original Assignee: SDL PLC (RWS Holdings Plc)
• Inventors: Galley, Michel, Knight, Kevin, Marcu, Daniel
• Primary Examiner(s): Utama, Robert J
• Application Number:
  US11/272,460
• Publication Number:
  US 20070122792A1
• Time in Patent Office: 4,962 Days
• Field of Search: 434118, 434247, 434322, 434323, 434350, 434353, 434362, 704 1- 10, 3118, 3247, 3322, 3323, 3350, 3353, 3362
• CPC Class Codes: G09B 5/00 Electrically-operated educational appliances working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

A learning system for a text-to-text application such as a machine translation system. The system has questions, and a matrix of correct answers to those questions. Any of the many different correct answers within the matrix can be considered as perfectly correct answers to the question. The system operates by displaying a question, which may be a phrase to be translated, and obtaining an answer to the question from the user. The answer is compared against the matrix and scored. Feedback may also be provided to the user.

Claims

1. A method for analyzing user language capability, the method comprising:

generating an answer key for a text to text based question regarding translation of a source text string from a source language to a target language, the source text string including a plurality of words, the answer key comprising a plurality of answers, the plurality of answers all consisting of completely correct answers to the text to text based question, each of the completely correct answers including a phrase having a plurality of words in the target language;

storing the answer key for the text to text based question in a packed format in an answer database on a computer, the packed format of each correct answer including a plurality of paths, the packed format of the answer key including certain common paths that are not stored repeatedly;

presenting the text to text based question regarding the source text to a user using the computer;

accepting, at the computer, an answer to the text to text based question as input from the user, the answer including a text translation of the source text, the text translation representing the user'"'"'s answer to the text to text based question regarding the source text for which the answer key has been determined; and

executing a program resident on a storage medium to cause the computer to;

search selected paths of the answer key without analyzing all of the plurality of different paths representing correct answers stored in the answer key,

determine an error cost for each selected path in the answer key, the error cost based on errors between the text translation and the correct answer represented by the analyzed path,

present an answer to the user representing a path having a lowest error cost in the answer key,

determine from the error cost if the presented answer is an exact match, and

if the presented answer is not an exact match, display feedback to the user based on errors between the text translation and the presented answer, the feedback including an error score based on the error cost of the presented answer.

2. A method as in claim 1, wherein the error score for the displayed answer is based on a cost assigned to each kind of error.

3. A method as in claim 2, wherein said scoring the answers comprises counting an error only once even when multiple parts are affected by said error.

4. A method as in claim 1, wherein the displayed feedback is based on a specific kind of error to help the user in understanding more about said error.

5. A method as in claim 1, wherein packed format of the answer key comprises a matrix of prestored correct answers, which includes millions of different correct answers.

6. A method as in claim 5, wherein said generating the answer key comprises forming a user interface which facilitates entry of the many different answers.

7. A method as in claim 6, wherein said user interface includes an associated tool that tests certain ones of said many different answers.

8. A method as in claim 1, wherein said packed format is a recursive transition network.

9. A method as in claim 1, wherein analyzing said paths of the answer key further comprises representing correct answers by maintaining a list including most promising single hypotheses through the packed database, and following the most promising hypothesis.

10. A method as in claim 3, wherein said errors include word insertion errors, word deletion errors, word substitution errors, word permutation errors, word stemming errors, and paraphrasing errors.

29. A method as in claim 1, wherein the user is a learning system for a text-to-text machine translation system.

30. A method as in claim 29, wherein the learning system and computer communicate the presented text-to-text based question and answer to the text-to-text based question over a network.

11. A method for analyzing user language capability, the method comprising:

displaying a first text having a plurality of words in a first language to a user;

determining an answer key for translation of the first text from the first language to a second language, the answer key comprising a plurality of completely correct answers to a text to text based question about the first text, each of the completely correct answers having a phrase including a plurality of words in the second language;

storing the answer key for the translation of the first text, each of the completely correct answers in the answer key stored in a matrix of prestored answers;

presenting to the user the text to text based question about translation of the first text between the first language and the second language;

accepting input of a text translation of the first text in the second language from the user as a response to the text to text based question about translation, the text translation generated by the user; and

executing a program resident on a storage medium to cause a machine to;

search selected paths of the answer key, the search performed without analyzing all of the plurality of different paths representing correct answers stored in the answer key using a speed up technique over a complete estimator,

determine an error cost for each path analyzed in the answer key,

present a best answer to the user representing a path having a lowest error cost in the answer key,

determine from the error cost if the presented answer is an exact match, and

if the presented answer is not an exact match, presenting feedback to the user based on errors between the text translation and the displayed answer, the feedback including an error score based on the error cost of the displayed answer.

12. A method as in claim 11, further comprising providing feedback to the user indicating specific information about the kind of errors made in said translation.

13. A method as in claim 12, wherein a calculation of the error score includes scoring each of a plurality of errors as being a single kind of error even when said error affects more than one word.

14. A method as in claim 13, wherein said calculation of the error score comprises continued considering a single error for each of word insertion, word deletion, word substitution, word permutation, word stemming, and paraphrasing.

15. A method as in claim 11, wherein said search comprises following a most promising path through the matrix of correct answers.

16. An apparatus for analyzing user language capability, the apparatus comprising:

an interface part operating to;

accept a text translation as input from a user representing the user'"'"'s answer to a text to text based question about translation of a phrase having a plurality of words, the translation between a first language and a second language, and

accept a plurality of different predetermined correct answers to the text to text based question, each of the plurality of correct answers including a plurality of parts, and each of the correct answers predetermined to be completely correct;

a database, in which said plurality of different predetermined correct answers are stored in a packed format, where at least two or more of the plurality of said answers rely on common information for one or more parts of said correct answers; and

a machine, which operates to;

analyze a plurality of paths representing correct answers of the stored correct answers without analyzing paths of all of the stored correct answers to determine error costs of the analyzed paths,

identify an analyzed path having a lowest error cost,

display a correct answer corresponding to the identified path to the user, wherein;

if the error cost for the displayed correct answer is zero, the display indicates to the user that the text translation input is a correct answer, and

if the error cost for the displayed correct answer is not zero, the display includes an error score based on the error cost.

17. An apparatus as in claim 16, wherein said machine calculates error scores for each of the plurality of analyzed paths representing correct answers according to a kind of said errors.

18. An apparatus as in claim 16, wherein said machine counts an error between the displayed answer and said text translation input only once even when multiple parts of the displayed answer are affected by said error.

19. An apparatus as in claim 18, wherein said machine determines, from said error, feedback to help the user in understanding more about said error.

20. An apparatus as in claim 16, wherein the answer key comprises a matrix which includes each of said plurality of different correct answers.

21. An apparatus as in claim 20, further comprising an answer matrix user interface that allows entry of details of said answer matrix, and which facilitates entry of the plurality of different correct answers.

22. An apparatus as in claim 21, wherein said answer matrix user interface includes an associated tool that tests certain ones of said plurality of different correct answers.

23. An apparatus as in claim 16, wherein said packed format is a recursive transition network.

24. An apparatus as in claim 16, wherein said machine operates by analyzing each of said plurality of paths representing correct answers of the stored correct answers by maintaining a list including a most promising single hypotheses through the packed format of the correct answers in the database, and following the most promising hypothesis.

25. An apparatus as in claim 18, wherein said errors include word insertion errors, word deletion errors, word substitution errors, word permutation errors, word stemming errors, and paraphrasing errors.

26. An apparatus for analyzing user bi-lingual language capability, the apparatus comprising:

a machine that includes a memory for storing answer keys in a matrix, each of the answer keys including a plurality of prestored answers to a translation question, each of the prestored answers in the matrix consisting of completely correct answers to a text-to-text based question, each of the completely correct answers including a phrase having a plurality of words,

said machine operating to produce signals indicative of a user interface that operates to display a first text in a first language to a user, and accepts a text translation as input of the text in a second language from the user as a translation,

said machine determining either an exact match or a closest match between the translation and any pre-stored answer to the question in the matrix, where

a plurality of the paths representing correct answers of the stored correct answers is analyzed without analyzing paths of all of the stored correct answers to determine error costs of the analyzed paths;

an analyzed path is identified as having a lowest error cost for all of the stored correct answers;

a correct answer corresponding to the identified path is displayed to the user;

if the error cost for the displayed correct answer is zero, the text translation input is indicated as being a correct answer; and

if the error cost for the displayed correct answer is not zero, an error score is displayed to the user as feedback to indicate an error and represent the error cost.

27. An apparatus as in claim 26, wherein said machine further provides feedback to the user indicating information about specific kinds of errors made in said translation.

28. An apparatus as in claim 26, wherein said machine calculates an error cost for each of a plurality of paths and one or more kinds of errors, each of the one or more kinds of errors considered as being a single error even when said error affects more than one word.