Speech recognition system and speech recognition method thereof

• US 10,699,718 B2
• Filed: 03/11/2016
• Issued: 06/30/2020
• Est. Priority Date: 03/13/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kim, Tae-yoon, Kim, Sang-ha, Han, Chang-woo, Lee, Jae-won
• Primary Examiner(s): Washburn, Daniel C
• Assistant Examiner(s): Nguyen, Timothy
• Application Number:
  US15/067,341
• Publication Number:
  US 20160267913A1
• Time in Patent Office: 1,572 Days
• Field of Search: 704252
• CPC Class Codes: G06F 1/3215 Monitoring of peripheral devices
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 17/20 Pattern transformations or operations aimed at increasing system robustness, e.g. against channel noise or different working conditions
• CPC Class Codes: G10L 17/24 the user being prompted to utter a password or a predefined phrase
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A device detects a wake-up keyword from a received speech signal of a user by using a wake-up keyword model, and transmits a wake-up keyword detection/non-detection signal and the received speech signal of the user to a speech recognition server. The speech recognition server performs a recognition process on the speech signal of the user by setting a speech recognition model according to the detection or non-detection of the wake-up keyword.

Claims

1. A device comprising:

a memory that stores a speech recognition model, and a plurality of personalized wake-up keyword models, the plurality of personalized wake-up keyword models being stored in association with context information; and

a communication interface configured to communicate with an external speech recognition server;

a processor configured to;

obtain user speech data;

identify a personalized wake-up keyword model, from among the plurality of personalized wake-up keyword models, that corresponds to current context information that is related to a current context of the device, the current context information including at least one of;

location information, weather information, or detected environment information;

identify, using the identified personalized wake-up keyword model that corresponds to the current context information, whether a wake-up keyword corresponding to a personalized wake-up keyword that corresponds to the current context is included in the obtained user speech data;

transmit, to the external speech recognition server via the communication interface, the obtained user speech data, information that indicates whether the device identified the personalized wake-up keyword that corresponds to the current context information in the obtained user speech data, and the current context information;

receive, from the external speech recognition server via the communication interface, a speech recognition result of the user speech data based on the transmitting of the obtained user speech data, the information that indicates whether the device identified the personalized wake-up keyword that corresponds to the current context information in the obtained user speech data, and the current context information; and

control the device according to the speech recognition result, wherein the processor is further configured to;

recognize, in the obtained user speech data, a speech signal for wake-up keyword registering, by using the speech recognition model; and

based on recognizing the speech signal for the wake-up keyword registering, identify whether the speech signal is valid as a new wake-up keyword model, based on a speech matching rate of recognition of the speech signal.

2. The device of claim 1, further comprising an environment information sensor configured to detect the detected environment information, which is environment information about an environment of the device, wherein the plurality of personalized wake-up keyword models are based on various environment information, and the processor is further configured to detect the personalized wake-up keyword from the obtained speech data by using a personalized wake-up keyword model that corresponds to the detected environment information of the device, from among the plurality of personalized wake-up keyword models.

3. The device of claim 2, wherein the detected environment information includes position information of the device.

4. The device of claim 2, wherein the processor is configured to recognize user identification information of a user based on the obtained user speech data of the user, transmit the user identification information and the environment information of the device to the external speech recognition server via the communication interface, and receive, from the external speech recognition server, the speech recognition result based on the user identification information of the user, and the detected environment information.

5. The device of claim 1, further comprising a user input interface configured to receive a user input, wherein the plurality of personalized wake-up keyword models include candidate personalized wake-up keyword models, and

the processor is further configured to register a candidate personalized wake-up keyword model, which is selected among the stored personalized wake-up keyword models, according to the user input received through the user input interface.

6. The device of claim 1, wherein the processor is configured to receive the plurality of personalized wake-up keyword models from the external speech recognition server via the communication interface and register the received personalized wake-up keyword models in the memory in advance.

7. The device of claim 1, wherein the processor is further configured to register the personalized wake-up keyword model in the memory in advance.

8. The device of claim 1, wherein the current context information includes the location information, the location information including at least one of physical location information or logical location information.

9. The device of claim 1, further comprising at least one sensor.

10. The device of claim 9, wherein the at least one sensor includes at least one of:

a global positioning system (GPS) sensor that detects the location information or an environment sensor that detects the environment information.

11. A speech recognition server comprising:

a memory configured to store a speech recognition model, and a plurality of personalized wake-up keyword models;

a communication interface configured to communicate with an external device; and

a processor configured to;

obtain, from the external device, user speech data, indication information that indicates whether the external device identified a personalized wake-up keyword that corresponds to current context information in the obtained user speech data, and the current context information, the current context information including at least one of;

location information, weather information, or detected environment information; and

based on the obtained indication information indicating that the external device identified the personalized wake-up keyword that corresponds to the current context information in the obtained user speech data;

combine the speech recognition model with a personalized wake-up keyword model and recognize the obtained user speech data using the combined model;

generate a speech recognition result by removing the personalized wake-up keyword from the obtained user speech data; and

transmit the speech recognition result, from which the personalized wake-up keyword has been removed, to the external device, wherein the processor is further configured to;

recognize, in the obtained user speech data, a speech signal for wake-up keyword registering, by using the speech recognition model; and

based on recognizing the speech signal for the wake-up keyword registering, identify whether the speech signal is valid as a new wake-up keyword model, based on a speech matching rate of recognition of the speech signal.

12. The speech recognition server of claim 11, wherein

the personalized wake-up keyword model is one of a plurality of wake-up keyword models which are based on various environment information, and

the combining of the models includes combining the speech recognition model with a personalized wake-up keyword model corresponding to environment information of the external device included in the obtained current context information, from among the plurality of wake-up keyword models.

13. The speech recognition server of claim 12, wherein the combining of the models includes combining the speech recognition model with a personalized keyword corresponding to user identification information included in the obtained current context information.

14. A speech recognition method, which is performed by a device, the speech recognition method comprising:

obtaining, via a microphone, user speech data;

identifying a personalized wake-up keyword model, from among a plurality of stored personalized wake-up keyword models, that corresponds to current context information that is related to a current context of the device, the current context information including at least one of;

location information, weather information, or detected environment information;

identifying, using the identified personalized wake-up keyword model that corresponds to the current context information, whether a wake-up keyword corresponding to a personalized wake-up keyword that corresponds to the current context information is included in the obtained user speech data;

transmitting, to an external speech recognition server, the obtained user speech data, information that indicates whether the device identified the personalized wake-up keyword that corresponds to the current context information in the obtained user speech data, and the current context information;

receiving, from the external speech recognition server via a communication interface, a speech recognition result of the user speech data based on the transmitting of the obtained user speech data, the information that indicates whether the device identified the personalized wake-up keyword that corresponds to the current context information in the obtained user speech data, and the current context information; and

controlling the device according to the speech recognition result, wherein a wake-up keyword model registered in the external speech recognition server is synchronized with the wake-up keyword model registered in the device, wherein the speech recognition method further comprises;

recognizing, in the obtained user speech data, a speech signal for wake-up keyword registering, by using a speech recognition model; and

based on recognizing the speech signal for the wake-up keyword registering, identifying whether the speech signal is valid as a new wake-up keyword model, based on a speech matching rate of recognition of the speech signal.

15. The speech recognition method of claim 14, wherein

the plurality of stored personalized wake-up keyword models are based on various environment information, the speech recognition method further comprises detecting the environment information, which is environment information about an environment of the device, and transmitting the detected environment information to the external speech recognition server, and

the identifying the wake-up keyword comprises detecting the wake-up keyword from the user speech data by using a personalized wake-up keyword model corresponding to the detected environment information, from among the plurality of stored personalized wake-up keyword models.

16. The speech recognition method of claim 15, further comprising recognizing user identification information based on the obtained user speech data, wherein the transmitted current context information comprises the user identification information and the environment information.

17. A non-transitory computer-readable recording medium having stored thereon a program which, when executed by a computer system, causes the computer system to perform the speech recognition method of claim 14.