TERMINAL DEVICE AND HANDS-FREE DEVICE FOR HANDS-FREE AUTOMATIC INTERPRETATION SERVICE, AND HANDS-FREE AUTOMATIC INTERPRETATION SERVICE METHOD

• US 20160210283A1
• Filed: 04/30/2014
• Published: 07/21/2016
• Est. Priority Date: 08/28/2013
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: KIM, Ki-Hyun, WANG, Ji-Hyun, LEE, Dam-Heo, KIM, Sang-Hun, KIM, Dong-Hyun, YUN, Seung, LEE, Min-Kyu, CHOI, Mu-Yeol
• Granted Patent: US 10,216,729 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 40/51 Translation evaluation
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: H04M 1/6041 Portable telephones adapted for handsfree use

Abstract

A user terminal, hands-free device and method for hands-free automatic interpretation service. The user terminal includes an interpretation environment initialization unit, an interpretation intermediation unit, and an interpretation processing unit. The interpretation environment initialization unit performs pairing with a hands-free device in response to a request from the hands-free device, and initializes an interpretation environment. The interpretation intermediation unit sends interpretation results obtained by interpreting a user'"'"'s voice information received from the hands-free device to a counterpart terminal, and receives interpretation results obtained by interpreting a counterpart'"'"'s voice information from the counterpart terminal. The interpretation processing unit synthesizes the interpretation results of the counterpart into a voice form based on the initialized interpretation environment when the interpretation results are received from the counterpart terminal, and sends the synthesized voice information to the hands-free device.

Claims

1. A user terminal for hands-free automatic interpretation service, comprising:

an interpretation environment initialization unit configured to perform pairing with a hands-free device in response to a request from the hands-free device, and to initialize an interpretation environment;

an interpretation intermediation unit configured to send interpretation results obtained by interpreting a user'"'"'s voice information received from the hands-free device to a counterpart terminal, and to receive interpretation results obtained by interpreting a counterpart'"'"'s voice information from the counterpart terminal; and

an interpretation processing unit configured to synthesize the interpretation results of the counterpart into a voice form based on the initialized interpretation environment when the interpretation results are received from the counterpart terminal, and to send the synthesized voice information to the hands-free device.

2. The user terminal of claim 1, wherein the interpretation environment initialization unit comprises:

an environment information reception unit configured to receive interpretation environment information, including one or more of pieces of information about a language and gender of the counterpart, from the counterpart terminal; and

an interpretation environment setting unit configured to set interpretation environment information, including one or more of information about an interpretation language and voice classification information, based on the received interpretation environment information.

3. The user terminal of claim 2, wherein the interpretation processing unit comprises a voice synthesis unit configured to synthesize the interpretation results into a male or female voice based on the set voice classification information when the interpretation results are received from the counterpart terminal.

4. The user terminal of claim 3, wherein, when additional information is further received from the counterpart terminal, the voice synthesis unit synthesizes the interpretation results into a male or female voice based on one or more of an articulation speed and emotional state of the user into the interpretation results based on the received additional information.

5. The user terminal of claim 1, further comprising an additional information collection unit configured to collect additional information including one or more of user state information and personal information;

wherein the user state information includes one or more of an articulation speed and an emotional state extracted from the user'"'"'s voice information; and

wherein the personal information includes one or more of information about a name, nationality and an image input by the user.

6. The user terminal of claim 1, wherein the interpretation intermediation unit comprises:

a server intermediation unit configured to send the voice information to an interpretation server when the voice information is received from the hands-free device, and to receive the interpretation results from the interpretation server; and

a terminal intermediation unit configured to send the received interpretation results to the counterpart terminal in accordance with a predetermined interpretation protocol.

7. The user terminal of claim 6, further comprising a situation information collection unit configured to collect situation information including one or more of information about a current location of the user and information about a background sound;

wherein the server intermediation unit sends the voice information received from the hands-free device and the collected situation information to the interpretation server.

8. The user terminal of claim 1, wherein:

the interpretation results obtained by interpreting the user'"'"'s voice information include one or more of interpretation data, corresponding to the voice information interpreted in a text or voice form, and interpretation error correction data; and

the error correction data includes one or more of similarly intended sentences and N-best recognition data.

9. The user terminal of claim 8, wherein the interpretation processing unit comprises an interpretation correction unit configured to, if the error correction data is included in the interpretation results, correct the interpretation data based on one or more pieces of information that belong to the error correction data and that are selected by the user.

10. A method for hands-free automatic interpretation service, the method being performed by a user terminal, the method comprising:

performing pairing with a hands-free device in response to a request from the hands-free device, and initializing an interpretation environment;

sending received voice information to an interpretation server when the voice information of a user is received from the hands-free device, and receiving interpretation results from the interpretation server;

sending the received interpretation results to a counterpart terminal,

receiving interpretation results obtained by interpreting a counterpart'"'"'s voice information from the counterpart terminal;

synthesizing the interpretation results received from the counterpart terminal into a voice form based on the initialized interpretation environment; and

sending the synthesized voice information to the hands-free device.

11. The method of claim 10, wherein initializing the interpretation environment comprises:

receiving interpretation environment information, including one or more of pieces of information about a language and gender of the counterpart, from the counterpart terminal; and

setting interpretation environment information, including one or more of information about an interpretation language and voice classification information, based on the received interpretation environment information.

12. The method of claim 11, wherein synthesizing the interpretation results into the voice comprises, when the interpretation results are received from the counterpart terminal, synthesizing the interpretation results into a male or female voice based on the set voice classification information.

13. The method of claim 12, wherein synthesizing the interpretation results into the voice comprises, when the additional information is further received from the counterpart terminal, synthesizing the interpretation results based on received additional information based on one or more of an articulation speed and emotional state of the user into the interpretation results.

14. The method of claim 10, further comprising collecting situation information including one or more of information about a current location of the user and information about a background sound;

wherein sending the received voice information of the user to the interpretation server comprises sending the voice information, received from the hands-free device, and the collected situation information to the interpretation server.

15. The method of claim 10, further comprising:

if the error correction data is included in the interpretation results received from the interpretation server, outputting error correction data to a display;

receiving one or more pieces of selection information, belonging to the error correction data, from the user; and

correcting interpretation data based on the received selection information.

16. A hands-free device for hands-free automatic interpretation service including a microphone and a speaker, the hands-free device comprising:

an interpretation function performance unit configured to perform pairing with a user terminal in response to an interpretation request from a user, and to request the user terminal to initialize an interpretation environment; and

a control unit configured to control a communication unit so that the communication unit sends voice information of the user, received via the microphone, to the user terminal, and to control the communication unit so that the communication unit outputs voice information, received from the user terminal, via the speaker.

17. The hands-free device of claim 16, wherein the interpretation function performance unit runs an interpretation app installed on the user terminal and requests the user terminal to initialize the interpretation environment, in response to the interpretation request from the user.

18. The hands-free device of claim 16, wherein the interpretation function performance unit detects a counterpart terminal, that is, an object terminal of interpretation at a close location, and sends information about the detected counterpart terminal to the user terminal via the communication unit so that an interpretation environment with the counterpart terminal is initialized.

19. The hands-free device of claim 16, further comprising a motion recognition unit configured to recognize a motion of the user using a motion recognition sensor using one or more of an acceleration sensor and a gyro sensor;

wherein the control unit controls a selection of error correction data output to a display of the user terminal based on the recognized motion.

20. The hands-free device of claim 16, wherein the microphone comprises one or more of a first microphone placed near a mouth of the user, a second microphone placed near an ear of the user and a bone conduction microphone.