Electronic device and method for processing user speech

• US 10,803,862 B2
• Filed: 04/19/2018
• Issued: 10/13/2020
• Est. Priority Date: 04/19/2017
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Jang, Dong-Ho, Song, Ga-Jin, Kim, Kyung-Tae, Kim, Kyu-Young, Jeon, Yong Joon, Kim, Min-Sung, Kim, Jin-Woong, Lee, Kwang-Bin, Lee, Ji-Young, Jun, Jae-Choon, Jaygarl, Ho-Jun
• Primary Examiner(s): Gay, Sonia L
• Application Number:
  US15/957,452
• Publication Number:
  US 20180308480A1
• Time in Patent Office: 908 Days
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

An electronic device and method are provided for processing user speech. The electronic device includes a housing, a touchscreen display disposed in the housing and exposed through a first portion of the housing, a microphone disposed in the housing and exposed through a second portion of the housing, at least one speaker disposed in the housing and exposed through a third portion of the housing, a wireless communication circuit disposed in the housing, at least one processor disposed in the housing and electrically connected with the touchscreen display, the microphone, the speaker, and the wireless communication circuit, and a memory disposed in the housing and electrically connected with the processor, wherein the memory stores instructions executed to enable the at least one processor to receive a first user input through the touchscreen display or the microphone, the first user input including a request to perform tasks using the electronic device, send data associated with the first user input through the wireless communication circuit to an external server, receive a response from the external server through the wireless communication circuit, the response including information about a plurality of states of the electronic device to perform the tasks in an order, perform the tasks by causing the electronic device to have the plurality of states having the order after receiving the response, receive a second user input through the touchscreen display or the microphone, the second user input including a request to cancel at least one of the performed tasks, and cause the electronic device to return to one of the plurality of states, based on the second user input.

Claims

1. An electronic device, comprising:

a housing;

a touchscreen display disposed in the housing and exposed through a first portion of the housing;

a microphone disposed in the housing and exposed through a second portion of the housing;

at least one speaker disposed in the housing and exposed through a third portion of the housing;

a wireless communication circuit disposed in the housing;

at least one processor disposed in the housing and electrically connected with the touchscreen display, the microphone, the speaker, and the wireless communication circuit; and

a memory disposed in the housing and electrically connected with the processor, wherein the memory stores instructions executed to enable the at least one processor to;

receive a first user input through the touchscreen display or the microphone, the first user input including a request to perform tasks using the electronic device,

send data associated with the first user input through the wireless communication circuit to an external server,

receive a response from the external server through the wireless communication circuit, the response including information about a plurality of states of the electronic device to perform the tasks in an order,

perform the tasks by causing the electronic device to have the plurality of states having the order after receiving the response,

receive a second user input through the touchscreen display or the microphone, the second user input including a request to cancel at least one of the performed tasks,

identify a first state of the plurality of states corresponding to the at least one of the performed tasks based on the second user input, and

cancel the at least one of the performed tasks by causing the electronic device to return to the first state of the plurality of states.

2. The electronic device of claim 1, further comprising a physical button exposed on the housing, wherein the instructions are executed to enable the processor to receive the second user input through the physical button.

3. The electronic device of claim 2, wherein the physical button is disposed on a surface, of the housing, where the touchscreen display is disposed.

4. The electronic device of claim 2, wherein the physical button is disposed on a surface, of the housing, where the touchscreen display is not disposed.

5. The electronic device of claim 1, wherein the instructions, when executed, further instruct the processor to:

receive the second user input through the microphone,

send data for the second user input to the external server through the wireless communication circuit, and

receive another response from the external server through the wireless communication circuit, and

wherein the another response causes the electronic device to return to the first state of the plurality of states.

6. The electronic device of claim 1, wherein the instructions, when executed, further instruct the processor to display, on the display, at least one of a screen or an image associated with the plurality of states.

7. The electronic device of claim 1, wherein the second user input includes a request for canceling all of the performed tasks, and

wherein the instructions, when executed, further instruct the processor to cause the electronic device to return to a state before the electronic device has the plurality of states, based on the second user input.

8. An electronic device, comprising:

a housing;

a touchscreen display disposed in the housing and exposed through a first portion of the housing;

a microphone disposed in the housing and exposed through a second portion of the housing;

at least one speaker disposed in the housing and exposed through a third portion of the housing;

a wireless communication circuit disposed in the housing;

at least one processor disposed in the housing and electrically connected with the touchscreen display, the microphone, the speaker, and the wireless communication circuit; and

a memory disposed in the housing and electrically connected with the processor, wherein the memory stores instructions executed to enable the at least one processor to;

receive a first user input through the touchscreen display or the microphone, the first user input including a request for the electronic device to perform a first task,

send first data associated with the first user input through the wireless communication circuit to an external server,

receive a first response from the external server through the wireless communication circuit, the first response including first information about a plurality of first states having an order for the electronic device to perform the first task,

perform the first task by causing the electronic device to have the plurality of first states having the order after receiving the first response,

receive a second user input through the touchscreen display or the microphone, the second user input including a second request for the electronic device to perform a second task, and

send second data associated with the second user input, along with at least part of the first information, to the external server, through the wireless communication circuit, and

receive a second response from the external server through the wireless communication circuit, the second response including second information about at least part of the first information for the electronic device to perform the second task.

9. The electronic device of claim 8, wherein the second user input includes a request for canceling at least part of the first task performed.

10. The electronic device of claim 9, wherein the instructions, when executed, further instruct the processor to cause the electronic device to return to one of the plurality of first states, based on at least part of the second user input.

11. An electronic device, comprising:

a housing;

a touchscreen display disposed in the housing and exposed through a first portion of the housing;

a microphone disposed in the housing and exposed through a second portion of the housing;

at least one speaker disposed in the housing and exposed through a third portion of the housing;

a wireless communication circuit disposed in the housing;

at least one processor disposed in the housing and electrically connected with the touchscreen display, the microphone, the speaker, and the wireless communication circuit; and

a memory disposed in the housing and electrically connected with the processor, wherein

the memory stores instructions executed to enable the at least one processor to;

control the touchscreen display to display a first screen,

obtain first user speech through the microphone while displaying the first screen,

control the touchscreen display to display a second screen associated with a first task of a plurality of tasks corresponding to the first user speech instead of the first screen,

control the touchscreen display to display a third screen associated with a second task of the plurality of tasks corresponding to the first user speech, instead of the second screen,

obtain a cancellation command to cancel at least one of the plurality of tasks, while displaying the third screen,

identify a task prior in order to the last task among the plurality of tasks based on a result of analyzing the cancellation command,

select the first screen or the second screen associated with the identified task, and

control the touchscreen display to display the selected screen.

12. The electronic device of claim 11, further comprising a hardware button,

wherein the instructions, when executed, cause the processor to obtain the cancellation command through at least one of the hardware button, the microphone, or the touchscreen display.

13. The electronic device of claim 11, wherein the cancellation command includes a request to cancel all of the plurality of tasks, and

wherein the instructions, when executed, instruct the processor to:

cancel all of the plurality of tasks, and

control the touchscreen display to display the first screen.

14. The electronic device of claim 13, wherein the cancellation command further includes a request to perform at least one third task, at least partially different from the plurality of tasks, and

wherein the instructions, when executed, further instruct the processor to:

cancel all of the plurality of tasks,

control the touchscreen display to display the first screen, and

control the touchscreen display to display a fourth screen associated with the at least one third task, instead of the first screen.

15. The electronic device of claim 11, wherein the cancellation command includes a request to canceling at least one of the plurality of tasks, and

wherein the instructions, when executed, further instruct the processor to:

cancel the at least one of the plurality of tasks, and

control the touchscreen display to display the second screen.

16. The electronic device of claim 15, wherein the cancellation command further includes a request for performing at least one fourth task, at least partially different from the plurality of tasks, and

wherein the instructions, when executed, further instructs the processor to:

cancel the at least one of the plurality of tasks,

control the touchscreen display to display the second screen, and

control the touchscreen display to display a fifth screen associated with the at least one fourth task, instead of the second screen.

17. The electronic device of claim 15, wherein the cancellation command further includes a request to perform at least one fourth task, at least partially different from the plurality of tasks, and

wherein the instructions, when executed, further instruct the processor to:

cancel the at least one of the plurality of tasks, and

while canceling, display a sixth screen for performing the at least one fourth task.

18. The electronic device of claim 15, wherein the second screen comprises one of:

a screen displayed immediately before the third screen is displayed;

a screen corresponding to a state for inputting an additional user input to perform another task, and

a screen displayed immediately after a used application is varied.

19. The electronic device of claim 11, wherein the instructions, when executed, further instruct the processor to:

send data for the first user speech to an external server through the wireless communication circuit,

receive the plurality of tasks corresponding to the first user speech through the wireless communication circuit,

send data for the cancellation command to the external server through the wireless communication circuit, and

receive the result of analyzing the cancellation command through the wireless communication circuit.

20. The electronic device of claim 11, wherein the instructions, when executed, further instruct the processor to:

perform an automatic speech recognition (ASR) process on data for the first user speech to obtain first text, and

perform a natural language understanding (NLU) process on the first text to obtain the plurality of tasks.

21. An electronic device, comprising:

a housing;

a touchscreen display disposed in the housing and exposed through a first portion of the housing;

a microphone disposed in the housing and exposed through a second portion of the housing;

at least one speaker disposed in the housing and exposed through a third portion of the housing;

a wireless communication circuit disposed in the housing;

at least one processor disposed in the housing and electrically connected with the touchscreen display, the microphone, the speaker, and the wireless communication circuit; and

a memory disposed in the housing and electrically connected with the processor, wherein the memory stores instructions executed to enable the at least one processor to;

control the touchscreen display to display a first screen,

obtain first user speech through the microphone while displaying the first screen,

control the touchscreen display to display a third screen associated with a last task of a plurality of tasks corresponding to the first user speech, instead of the first screen,

obtain a cancellation command to cancel at least one of the plurality of tasks, while displaying the third screen,

identify a task prior in order to the last task among the plurality of tasks based on a result of analyzing the cancellation command,

select the first screen or a second screen associated with the identified task, and

control the touchscreen display to display the selected screen.