Electronic device for performing operation corresponding to voice input

• US 10,540,973 B2
• Filed: 05/30/2018
• Issued: 01/21/2020
• Est. Priority Date: 06/27/2017
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Jaygarl, Ho Jun, Kim, Jin Woong, No, Jae Gun, Yang, Jae Young
• Primary Examiner(s): Nguyen, Quynh H
• Application Number:
  US15/992,906
• Publication Number:
  US 20180374481A1
• Time in Patent Office: 601 Days
• Field of Search: 704265, 704235
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

An electronic device is provided. The electronic device includes a housing, a speaker, a microphone, a display, a communication circuit, at least one processor operatively connected to the speaker, the microphone, the display, and the communication circuit, and a memory operatively connected to the at least one processor, and storing instructions to cause the at least one processor to receive a first user utterance, including a request to perform a task by using an application program, through the microphone, transmit first data associated with the first user utterance to an external server, receive second data including a text generated by an automatic speech recognition (ASR) module, from the external server, identify the application program based on the second data, receive a first response including information of a sequence of states of the electronic device from the external server to perform at least part of the task, and perform the at least part of the task by controlling the electronic device to enter the sequence of the states by using the application program.

Claims

1. An electronic device comprising:

a housing;

a speaker disposed in a first area of the housing;

a microphone disposed in a second area of the housing;

a display disposed in a third area of the housing;

a communication circuit disposed inside the housing or attached to the housing;

at least one processor disposed inside the housing and operatively connected to the speaker, the microphone, the display, and the communication circuit; and

a memory disposed inside the housing, operatively connected to the at least one processor, and configured to store a plurality of application programs and instructions,

wherein the instructions, when executed, cause the at least one processor to;

receive, through the microphone, a user utterance including a request to perform a task by using at least one of the plurality of application programs,

transmit, through the communication circuit, first data associated with the user utterance to an external server,

receive, from the external server through the communication circuit, second data including a text generated by an automatic speech recognition (ASR) module,

in response to receiving the second data including the text, identify the at least one of the plurality of application programs based on at least part of the second data including the text,

prepare to execute the identified at least one of the plurality of application programs while the external server creates or determines a sequence of states of the electronic device,

after identifying the at least one of the plurality of application programs, receive, from the external server through the communication circuit, a first response including information associated with the sequence of states of the electronic device to perform at least part of the task, and

in response to receiving the first response, perform the at least part of the task by controlling the electronic device to enter the sequence of the states by using the at least one of the plurality of application programs.

2. The electronic device of claim 1,

wherein the memory is further configured to store a service component associated with an intelligence assistant, and

wherein the instructions, when executed, further cause the at least one processor to, before receiving the first response, establish communication between the at least one of the plurality of application programs and the service component.

3. The electronic device of claim 2, wherein the communication includes inter-process communication (IPC).

4. The electronic device of claim 3,

wherein the instructions comprise an operating system, and

wherein the instructions, when executed, further cause the at least one processor to start the communication by using a binder mechanism or framework.

5. The electronic device of claim 1,

wherein the plurality of application programs include a first type application program and a second type application program, and

wherein the instructions, when executed, further cause the at least one processor to:

when the at least one of the plurality of application programs comprises the first type application program, perform the at least part of the task based on the sequence of states; and

when the at least one of the plurality of application programs comprises the second type application program, perform the at least part of the task based on an actor received from the external server, the actor comprising a program for controlling the second type application or an application programming interface (API) set.

6. The electronic device of claim 1, wherein the instructions, when executed, further cause the at least one processor to identify the at least one of the plurality of application programs based on a category in which the first user utterance is included.

7. The electronic device of claim 1, wherein the instructions, when executed, further cause the at least one processor to identify the at least one of the plurality of application programs based on another portion adjacent to a specified portion of the text.

8. An electronic device comprising:

a housing;

a speaker disposed in a first area of the housing;

a microphone disposed in a second area of the housing;

a display disposed in a third area of the housing;

a communication circuit disposed inside the housing or attached to the housing;

at least one processor disposed inside the housing and operatively connected to the speaker, the microphone, the display, and the communication circuit; and

a memory disposed inside the housing, operatively connected to the at least one processor, and configured to store a plurality of application programs and instructions,

wherein the instructions, when executed, cause the at least one processor to;

receive, through the microphone, a user utterance,

transmit, through the communication circuit, the user utterance to an external server,

receive, through the communication circuit from the external server, text data corresponding to the user utterance,

in response to receiving the text data, identify a first application program of the plurality of application programs based on at least part of the text data,

in response to identifying the first application program, establish communication between the first application program and a control program capable of executing the plurality of application programs while the external server creates or determines a sequence of states of the electronic device,

after establishing the communication between the first application program and the control program, receive, through the communication circuit from the external server, the sequence of states of the electronic device,

compare the first application program and a second application program executable based on the sequence of states, and

execute the first application program or the second application program based on a result of the comparing.

9. The electronic device of claim 8, wherein the instructions, when executed, further cause the at least one processor to, when the first application program corresponds to the second application program, execute the first application program.

10. The electronic device of claim 8, wherein the instructions, when executed, further cause the at least one processor to, when the first application program does not correspond to the second application program, establish communication between the second application program and the control program.

11. The electronic device of claim 10, wherein the instructions, when executed, further cause the at least one processor to, when the communication between the second application program and the control program is established, execute the second application program.

12. The electronic device of claim 8, wherein the instructions, when executed, further cause the at least one processor to receive, through the communication circuit, data indicating whether the user utterance is terminated.

13. The electronic device of claim 8, wherein the instructions, when executed, further cause the at least one processor to sequentially receive, through the communication circuit, pieces of text data generated based on an order in which the user utterance is entered into the electronic device.

14. The electronic device of claim 8,

wherein the text data includes identification data corresponding to each of the plurality of application programs, and

wherein the instructions, when executed, further cause the at least one processor to identify the first application program based on the identification data.

15. The electronic device of claim 8, wherein the instructions, when executed, further cause the at least one processor to identify at least one of the plurality of application programs based on a category in which the user utterance is included.

16. An electronic device comprising:

a housing;

a speaker disposed in a first area of the housing;

a microphone disposed in a second area of the housing;

a display disposed in a third area of the housing;

a communication circuit disposed inside the housing or attached to the housing;

at least one processor disposed inside the housing and operatively connected to the speaker, the microphone, the display, and the communication circuit; and

a memory disposed inside the housing, operatively connected to the at least one processor, and configured to store a plurality of application programs and instructions,

wherein the instructions, when executed, cause the at least one processor to;

receive, through the microphone, a first user utterance including a request to perform a task by using at least one of the plurality of application programs,

in response to receiving the user utterance, compare the first user utterance with a second user utterance stored in the memory, and

when a result of the comparing indicates that a degree of matching between the first user utterance and the second user utterance is at a threshold level, perform at least part of the task based on a sequence of states of the electronic device corresponding to the second user utterance.

17. The electronic device of claim 16, wherein the instructions, when executed, further cause the at least one processor to:

receive, from an external server through the communication circuit, data indicating whether the sequence of states is refined, and

when the sequence of states is not refined, perform the at least part of the task based on the sequence of states.

18. The electronic device of claim 17, wherein the instructions, when executed, further cause the at least one processor to:

when the sequence of states is refined, receive, from the external server through the communication circuit, the refined sequence of states of the electronic device, and

perform the at least part of the task based on the refined sequence of states of the electronic device.

19. The electronic device of claim 16,

wherein the first user utterance includes first pulse code modulation (PCM) data corresponding to the first user utterance,

wherein the second user utterance includes second PCM data corresponding to the second user utterance, and

wherein the instructions, when executed, further cause the at least one processor to compare the first PCM data with the second PCM data.

20. The electronic device of claim 16, wherein the instructions, when executed, further cause the at least one processor to receive text data corresponding to the first user utterance.