Methods and apparatus for context adaptation of speech-to-speech translation systems

• US 20080059147A1
• Filed: 09/01/2006
• Published: 03/06/2008
• Est. Priority Date: 09/01/2006
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gu, Liang, Kuo, Hong-Kwang Jeff, Zhou, Bowen, Gao, Yuqing, Afify, Mohamed A.
• Granted Patent: US 7,860,705 B2
• Time in Patent Office: Days
• US Class Current: 704/5
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A technique for context adaptation of a speech-to-speech translation system is provided. A plurality of sets of paralinguistic attribute values is obtained from a plurality of input signals. Each set of the plurality of sets of paralinguistic attribute values is extracted from a corresponding input signal of the plurality of input signals via a corresponding classifier of a plurality of classifiers. A final set of paralinguistic attribute values is generated for the plurality of input signals from the plurality of sets of paralinguistic attribute values. Performance of at least one of a speech recognition module, a translation module and a text-to-speech module of the speech-to-speech translation system is modified in accordance with the final set of paralinguistic attribute values for the plurality of input signals.

Claims

1. A method of context adaptation of a speech-to-speech translation system comprising the steps of:

extracting a plurality of sets of paralinguistic attribute values from a plurality of input signals, wherein each set of the plurality of sets of paralinguistic attribute values is extracted from a corresponding input signal of the plurality of input signals via a corresponding classifier of a plurality of classifiers;

generating a final set of paralinguistic attribute values for the plurality of input signals from the plurality of sets of paralinguistic attribute values; and

modifying performance of at least one of a speech recognition module, a translation module and a text-to-speech module of the speech-to-speech translation system in accordance with the final set of paralinguistic attribute values for the plurality of input signals.

2. The method of claim 1, wherein the step of extracting a plurality of sets of paralinguistic attribute values comprises the steps of:

receiving each of the plurality of input signals at a corresponding one of the plurality of classifiers;

determining a value for each of a plurality of paralinguistic attributes from each input signal;

outputting a set of paralinguistic attribute values from each of the plurality of classifiers.

3. The method of claim 1, wherein, in the step of extracting a plurality of sets of paralinguistic attribute values, the paralinguistic attribute values comprise values for at least one of gender, accent, age, intonation, emotion, social background and educational level of a speaker.

4. The method of claim 1, wherein, in the step of extracting a plurality of sets of paralinguistic attribute values, the plurality of classifiers comprise at least one of a speech signal classifier, a visual signal classifier, a text input classifier, and a pointing device classifier.

5. The method of claim 4, wherein an input signal of the text input classifier comprises at least one of text entered by an operator and text obtained as feedback from the speech recognition module.

6. The method of claim 1, wherein the step of generating a final set of paralinguistic attribute values comprises the steps of:

receiving the plurality of sets of paralinguistic attribute values from the plurality of classifiers; and

combining values of common paralinguistic attributes from the plurality of sets of paralinguistic attribute values to generate a final set of paralinguistic attribute values.

7. The method of claim 6, wherein, in the step of combining values of common paralinguistic attributes, each set of paralinguistic attribute values comprise a plurality of confidence values, and each of the plurality of confidence values corresponds to a paralinguistic attribute value, and wherein the plurality of confidence values are utilized in combining values.

8. The method of claim 6, wherein, in the step of combining values of common paralinguistic attributes, each set of paralinguistic attribute values is associated with a corresponding usefulness factor for each classifier, and the usefulness factor is utilized in combining values.

9. The method of claim 1, wherein, in the step of generating a final set of paralinguistic attribute values, the final set of paralinguistic attribute values define a social context of the plurality of input signals.

10. The method of claim 1, wherein, in the step of generating a final set of paralinguistic attribute values, the final set of paralinguistic attribute values enable question detection.

11. The method of claim 1, wherein the step of modifying performance comprises the steps of:

constructing one or more models in at least one of the speech recognition module and the translation module in accordance with the final set of paralinguistic attribute values, wherein each model is conditioned on different paralinguistic attributes; and

dynamically selecting an appropriate model from the one or more models during operation of at least one of the speech recognition module and the translation module.

12. The method of claim 1, wherein the step of modifying performance comprises the step of accessing an expression database to generate appropriate expression in the text-to-speech module based on the final set of paralinguistic attribute values.

13. The method of claim 1, wherein the step of modifying performance comprises the step of obtaining an appropriate pronunciation in the text-to-speech module based on the final set of paralinguistic attribute values.

14. A context adaptable speech-to-speech translation system comprising:

a plurality of classifiers, wherein each of the plurality of classifiers receives a corresponding input signal and generates a corresponding set of paralinguistic attribute values;

a fusion module that receives a plurality of sets of paralinguistic attribute values from the plurality of classifiers and generates a final set of paralinguistic attribute values; and

speech-to-speech translation modules comprising a speech recognition module, a translation module, and a text-to-speech module, wherein performance of at least one of the speech recognition module, the translation module and the text-to-speech module are modified in accordance with the final set of paralinguistic attribute values for the plurality of input signals.

15. The context adaptable speech-to-speech translation system of claim 14, wherein the each of the plurality of classifiers receive a corresponding one of the plurality of input signals, determine a value for each of a plurality of paralinguistic attributes from each input signal, and output a set of paralinguistic attribute values from each of the plurality of classifiers.

16. The context adaptable speech-to-speech translation system of claim 14, wherein the fusion module receives the plurality of sets of paralinguistic attribute values from the plurality of classifiers, and combines values of common paralinguistic attributes from the plurality of sets of paralinguistic attribute values to generate a final set of paralinguistic attribute values.

17. The context adaptable speech-to-speech translation system of claim 14, wherein the speech-to-speech translation modules construct one or more models in at least one of the speech recognition module and the translation module in accordance with the final set of paralinguistic attribute values, wherein each model is conditioned on different paralinguistic attributes, and dynamically select an appropriate model from the one or more models during operation of at least one of the speech recognition module and the translation module.

18. The context adaptable speech-to-speech translation system of claim 14, wherein the speech-to-speech translation modules access an expression database to generate appropriate expression in the text-to-speech module based on the final set of paralinguistic attribute values.

19. The context adaptable speech-to-speech translation system of claim 14, wherein the speech-to-speech translation modules obtain an appropriate pronunciation in the text-to-speech module based on the final set of paralinguistic attribute values.

20. An article of manufacture for context adaptation of a speech-to-speech translation system, comprising a machine readable medium containing one or more programs which when executed implement the steps of:

extracting a plurality of sets of paralinguistic attribute values from a plurality of input signals, wherein each set of the plurality of sets of paralinguistic attribute values is extracted from a corresponding input signal of the plurality of input signals via a corresponding classifier of a plurality of classifiers;

generating a final set of paralinguistic attribute values for the plurality of input signals from the plurality of sets of paralinguistic attribute values; and

modifying performance of at least one of a speech recognition module, a translation module and a text-to-speech module of the speech-to-speech translation system in accordance with the final set of paralinguistic attribute values for the plurality of input signals.