Method for operating speech recognition service and electronic device supporting the same

• US 10,706,847 B2
• Filed: 03/28/2018
• Issued: 07/07/2020
• Est. Priority Date: 03/28/2017
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Seo, Jang Seok, Kang, Sang Ki, Ku, Han Jun, Park, Sung Pa, Rhee, In Jong, Yi, Ji Soo, Hong, Yoo Jin, Kim, Kyung Tae, Kim, Ji Hyun, Jeon, Yong Joon
• Primary Examiner(s): McFadden, Susan I
• Application Number:
  US15/938,094
• Publication Number:
  US 20180286400A1
• Time in Patent Office: 832 Days
• Field of Search: 704257
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/226 using non-speech characteristics

Abstract

An electronic device includes a communication module, a sensor module, a microphone, a memory, a display, and a processor. The processor is configured to determine whether a user is in proximity to the electronic device, to transmit at least one of voice input information or information on the proximity of the user to the external device, to receive at least one action associated with the execution of the function of the electronic device corresponding to a recognition result from the external device, based on voice input recognition of the external device, to output content associated with execution and/or processing of each of the at least one action, when the user is spaced apart from the electronic device, and to prevent at least a portion of the at least one content from being output in a state that the user is within the specified proximity to the electronic device.

Claims

1. An electronic system comprising:

an electronic device including a housing including a first surface, a viewable display, a proximity sensor disposed on the first surface and adjacent to the display, a microphone, a first speaker disposed on the first surface and adjacent to the display, a second speaker disposed to be spaced apart from the first speaker, a wireless communication circuit, a first processor electrically connected with the display, the proximity sensor, the microphone, the first speaker, the second speaker, and the wireless communication circuit, and a first memory electrically connected with the first processor, and

a first server including a network interface, a second processor electrically connected with the network interface, and a second memory electrically connected with the second processor,

wherein the second memory includes an automatic speech recognition (ASR) module comprising processing circuitry and program elements, a natural language understanding (NLU) module comprising processing circuitry and program elements, and a natural language generator (NLG) module comprising processing circuitry and program elements,

wherein the first memory stores instructions that, when executed, cause the first processor to;

receive proximity data from the proximity sensor;

receive a voice input using the microphone; and

provide, to the second processor, first data associated with the voice input and second data associated with the proximity data through the wireless communication circuit,

wherein the second memory stores instructions that, when executed, cause the second processor to;

process the first data using at least one of the ASR module and/or the NLU module;

generate a natural language response based on at least a portion of the second data using the NLG module;

generate a path rule including a sequence of at least one function based on the first data and the second data; and

provide the natural language response and the path rule to the first processor, and

wherein the first memory further stores instructions that, when executed, cause the first processor to;

output at least a portion of the natural language response through the first speaker and/or through the display and the second speaker by executing the at least one function based on the path rule.

2. The electronic system of claim 1, wherein the instructions in the second memory cause the second processor to:

generate a first natural language response when the second data indicates that an object is within a specified proximity of the proximity sensor.

3. The electronic system of claim 2, wherein the instructions in the first memory cause the first processor to:

output at least a portion of the first natural language response through the first speaker during a turn-off state of the display.

4. The electronic system of claim 3, wherein the instructions in the first memory cause the first processor to:

monitor third data provided by the proximity sensor; and

output the at least a portion of the first natural language response using the first speaker and/or using the display and the second speaker, based on at least a portion of the third data.

5. An electronic device configured to support a speech recognition service, the electronic device comprising:

a communication module comprising communication circuitry configured to communicate with an external device;

a sensor module comprising sensing circuitry configured to sense a proximity of an object to the electronic device;

a microphone configured to receive a voice input;

a memory configured to store information associated with operation of the speech recognition service;

a display configured to output a screen associated with execution of a function of the electronic device; and

a processor electrically connected with the communication module, the sensor module, the microphone, the memory, and the display,

wherein the processor is configured to;

determine whether a user and the electronic device are within a specified proximity to each other using the sensor module;

control the electronic device to transmit, to the external device, voice input information and information on the proximity between the user and the electronic device;

control the electronic device to receive a path rule including a sequence of at least one action associated with the execution of the function of the electronic device corresponding to a recognition result from the external device based on voice input recognition of the external device;

control the electronic device to output at least one content associated with execution and/or processing of each of the at least one action based on the path rule, when the user and the electronic device are spaced apart from each other; and

control the electronic device to prevent at least a portion of the at least one content from being output when the user and the electronic device are within the specified proximity to each other.

6. The electronic device of claim 5, wherein the processor is configured to:

control the display to be turned off when it is determined that the user and the electronic device are within the specified proximity to each other; and

process a screen associated with the execution and/or the processing of each of the at least one action in a background of the display which is turned off.

7. The electronic device of claim 6, wherein the processor is configured to:

control the display, which is turned off, to be turned on, when a state between the electronic device and the user is changed from a proximity state, in which the user and the electronic device are within the specified proximity to each other, to a spacing state in which the user and the electronic device are spaced apart from each other; and

control the electronic device to output a screen associated with execution and/or processing of a final action of the at least one action.

8. The electronic device of claim 6, wherein the processor is configured to:

control the display, which is turned off, to be turned on, when a state between the electronic device and the user is changed from a proximity state, in which the user and the electronic device are within the specified proximity to each other, to a spacing state in which the user and the electronic device are spaced apart from each other; and

control the electronic device to output a screen associated with execution or processing of an action of the at least one action, which is executed at a time that the state between the electronic device and the user is changed to the spacing state.

9. The electronic device of claim 5, further comprising:

a hardware key disposed in one region of the electronic device,

wherein the processor is configured to;

control activation of the speech recognition service, based on the electronic device receiving a signal of an input applied to the hardware key.

10. The electronic device of claim 5, wherein the processor is configured to:

control the electronic device to receive voice response information associated with a specific action of the at least one action, from the external device; and

control the electronic device to output the voice response information at a time that the specific action is executed and/or completed, when a sound output function is provided in the electronic device.

11. The electronic device of claim 10, further comprising:

a receiver module comprising receiver circuitry disposed in a first region of the electronic device; and

a speaker disposed in a second region of the electronic device.

12. The electronic device of claim 11, wherein the processor is configured to:

control the electronic device to output the voice response information using the receiver module, when it is determined that the electronic device and the user are within the specified proximity to each other.

13. The electronic device of claim 11, wherein the processor is configured to:

control the electronic device to output the voice response information using the speaker, when it is determined that the electronic device and the user are spaced apart from each other.

14. The electronic device of claim 11, further comprising:

a hardware key disposed in a region of the electronic device,

wherein the processor is configured to;

control the electronic device to output specified voice information using the receiver module, when receiving a signal of an input applied to the hardware key, in a state that the sound output function is provided in the electronic device and the electronic device and the user are within the specified proximity to each other.

15. A method for operating a speech recognition service of an electronic device, the method comprising:

receiving a voice input;

determining whether the electronic device and a user are within a specified proximity to each other;

transmitting, to an external device which supports the operation of the speech recognition service, voice input information and information on the proximity between the user and the electronic device;

receiving a path rule including a sequence of at least one action associated with execution of a function of the electronic device corresponding to a recognition result of the voice input information from the external device;

outputting at least one content associated with execution or processing of each of the at least one action based on the path rule, when the electronic device and the user are spaced apart from each other; and

preventing at least a portion of the at least one content from being output when the electronic device and the user are within the specified proximity to each other.

16. The method of claim 15, wherein the preventing of the at least a portion of the at least one content includes:

controlling a display of the electronic device to be turned off when it is determined that the electronic device and the user are within the specified proximity to each other; and

processing a screen associated with the execution and/or the processing of each of the at least one action in background of the display which is turned off.

17. The method of claim 16, wherein the preventing of the at least a portion of the at least one content includes:

controlling the display, which is turned off, to be turned on, when a state between the electronic device and the user is changed from a proximity state, in which the user and the electronic device are within the specified proximity to each other, to a spacing state in which the user and the electronic device are spaced apart from each other; and

outputting a screen associated with execution or processing of a final action of the at least one action.

18. The method of claim 16, wherein the preventing of the at least a portion of the at least one content includes:

controlling the display, which is turned off, to be turned on, when a state between the electronic device and the user is changed from a proximity state, in which the user and the electronic device are within the specified proximity to each other, to a spacing state in which the user and the electronic device are spaced apart from each other; and

outputting a screen associated with execution or processing of an action of the at least one action, which is executed at a time that the state between the electronic device and the user is changed to the spacing state.

19. The method of claim 15, wherein the receiving of the at least one action includes:

receiving voice response information associated with a specific action of the at least one action from the external device, and

wherein the preventing of the at least a portion of the at least one content includes;

outputting the voice response information at a time that the specific action is executed and/or completed, when a sound output function is provided in the electronic device.

20. The method of claim 19, wherein the outputting of the voice response information includes:

outputting the voice response information using a receiver module comprising receiver circuitry disposed in a first region of the electronic device, when the electronic device and the user are within the specified proximity to each other; and

outputting the voice response information using a speaker disposed in a second region of the electronic device, when the electronic device and the user are spaced apart from each other.