Mobile speech-to-speech interpretation system

• US 8,478,578 B2
• Filed: 01/09/2009
• Issued: 07/02/2013
• Est. Priority Date: 01/09/2008
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Fluential, LLC
• Inventors: Ehsani, Farzad, Master, Demitrios, Zuber, Elaine Drom
• Primary Examiner(s): SAINT CYR, LEONARD
• Application Number:
  US12/351,793
• Publication Number:
  US 20090177461A1
• Time in Patent Office: 1,635 Days
• Field of Search: 704 2- 10, 704/246, 704/247, 704/251, 704/252
• US Class Current: 704/2
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06F 40/51 Translation evaluation
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications

Abstract

Interpretation from a first language to a second language via one or more communication devices is performed through a communication network (e.g. phone network or the internet) using a server for performing recognition and interpretation tasks, comprising the steps of: receiving an input speech utterance in a first language on a first mobile communication device; conditioning said input speech utterance; first transmitting said conditioned input speech utterance to a server; recognizing said first transmitted speech utterance to generate one or more recognition results; interpreting said recognition results to generate one or more interpretation results in an interlingua; mapping the interlingua to a second language in a first selected format; second transmitting said interpretation results in the first selected format to a second mobile communication device; and presenting said interpretation results in a second selected format on said second communication device.

Claims

1. A method for interpreting from a first language to a second language, comprising the steps of:

receiving an input speech utterance in a first language on a first mobile communication device;

conditioning said input speech utterance;

first transmitting said conditioned input speech utterance to a server;

recognizing said first transmitted speech utterance to generate a set of first recognition results;

optimizing said recognition results by rescoring word-recognition lattice results in conjunction with interpretation lattice results to generate a set of second recognition results;

interpreting said set of second recognition results by generating one or more interpretation results comprising abstract paraphrased specific concepts;

translating said abstract paraphrased specific concepts into a form of a language-independent abstract interlingua;

mapping the abstract paraphrased specific concepts from the language-independent interlingua to a second language in a first selected format;

second transmitting said interpretation results in the first selected format to a second mobile communication device; and

presenting said interpretation results in a second selected format on said second communication device.

2. The method of claim 1 wherein in said conditioning step, features of said input speech utterance are extracted.

3. The method of claim 1 wherein in said first transmitting step, said first transmission is performed using a data connection.

4. The method of claim 3 wherein said data connection is a digital network.

5. The method of claim 1 wherein in said first transmitting step, users'"'"' profile data and updates thereof are transmitted with said compressed input speech.

6. The method of claim 5 wherein said user profile data includes one or more of:

domain selection, voice identification data, location data, and users'"'"' preferences.

7. The method of claim 1 wherein said second communication device is the first communication device.

8. The method of claim 1 wherein the format includes text, audio, images and videos.

9. The method of claim 8 wherein the video format is presented in a sign language format.

10. The method of claim 1 wherein in said recognition step, mobile interference models are utilized to improve the recognition results.

11. The method of claim 1 wherein in said interpreting step, interpretation is accomplished through one or more of:

classification engine, RBMT, TBMT, and SMT.

12. The method of claim 1 wherein in said interpreting step, said recognition results is interpreted to generate one or more interpretation results in said second language, thereby bypassing said mapping step.

13. A method for interpreting from a first language to a second language, comprising the steps of:

receiving an input speech utterance in a first language on a first mobile communication device;

conditioning said input speech utterance;

first transmitting said conditioned input speech utterance to a server;

recognizing said first transmitted speech utterance to generate by said server a set of first recognition results;

optimizing the one or more recognition results by rescoring word-recognition lattice results based on corresponding interpretation lattice results to generate a set of second recognition results;

interpreting said set of second recognition results by generating by said server one or more interpretation results comprising abstract paraphrased specific concepts;

translating said abstract paraphrased specific concepts into a form of a language-independent abstract interlingua;

mapping the abstract paraphrased specific concepts from the language-independent interlingua to a second language in a first selected format;

second transmitting said interpretation results in the first selected format to a second mobile communication device; and

presenting said interpretation results in a second selected format on said second communication device.

14. The method of claim 13 wherein in said conditioning step, features of said input speech utterance are extracted.

15. The method of claim 13 wherein in said first transmitting step, users'"'"' profile data and updates thereof are transmitted with said compressed input speech.

16. The method of claim 15 wherein said user profile data includes one or more of:

domain selection, voice identification data, location data, and users'"'"' preferences.

17. The method of claim 13 wherein said second communication device is the first communication device.

18. The method of claim 13 wherein the format includes text, audio, images and videos.

19. The method of claim 18 wherein the video format is presented in a sign language format.

20. The method of claim 13 wherein in said recognition step, mobile interference models are utilized to improve the recognition results.

21. The method of claim 13 wherein in said interpreting step, interpretation is accomplished through one or more of:

classification engine, RBMT, TBMT, and SMT.

22. The method of claim 13 wherein in said interpreting step, said recognition results is interpreted to generate one or more interpretation results in said second language, thereby bypassing said mapping step.

23. The method of claim 13 wherein in said first transmitting step, said first transmission is performed using a data connection.

24. The method of claim 23 wherein said data connection is a digital network.