Electronic device and method of executing function of electronic device

• US 10,547,729 B2
• Filed: 03/05/2018
• Issued: 01/28/2020
• Est. Priority Date: 03/27/2017
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Choi, Il-Hwan, Kang, Sang-Wook, Kim, Young-Hoon, Park, Seung-Hoon, Seo, Hyun-Ju, Um, Tae-Kwang, Lee, Jae-Yong, Choi, Jin-Ho, Lee, Da-Som, Yeo, Jae-Yung
• Primary Examiner(s): Khan, Suhail
• Application Number:
  US15/911,329
• Publication Number:
  US 20180278740A1
• Time in Patent Office: 694 Days
• Field of Search: None
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: H04M 1/724 User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 1/72469 for operating the device by selecting functions from two or more displayed items, e.g. menus or icons
• CPC Class Codes: H04M 2250/22 including a touch pad, a touch sensor or a touch detector digitizers for touch screens G06F3/041
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00

Abstract

An artificial intelligence system and method are disclosed herein. The system includes a processor which implements the method, including: receiving by an input unit a first user input including a request to execute a task using at least one of the electronic device or an external device, transmitting by a wireless communication unit first data associated with the first user input to an external server, receiving a first response from the external server including information associated with at least one of the first user input and a sequence of electronic device states for performing at least a portion of the task, receiving a second user input assigning at least one of a voice command and a touch operation received by a touch screen display as the request to perform the task, and transmitting second data associated with the second user input to the external server.

Claims

1. An electronic device comprising:

a housing;

a touch screen display that is located inside the housing and is exposed through a first portion of the housing;

a microphone that is located inside the housing and is exposed through a second portion of the housing;

at least one speaker that is located inside the housing and is exposed through a third portion of the housing;

a wireless communication circuit that is located inside the housing;

a processor that is located inside the housing and is electrically connected to the touch screen display, the microphone, the speaker, and the wireless communication circuit; and

a memory that is located inside the housing and is electrically connected to the processor,

wherein the memory stores instructions that, when executed, cause the processor to;

receive a first user input via at least one of the touch screen display and the microphone, the first user input including a request to perform a task using at least one of the electronic device and an external device,

transmit first data associated with the first user input to an external server via the wireless communication circuit,

receive a first response from the external server via the wireless communication circuit, the first response including information associated with the first user input and/or sequence information of states of the electronic device for performing at least a portion of the task, wherein the sequence information corresponds to at least one representative dialog stored in the memory, and wherein when an input corresponding to the at least one representative dialog is received from a user by the processor, at least a portion of the task is performed,

after receiving the first response, in response to receiving a second user input including an utterance and/or a user operation via the touch screen display, set a command shortcut to be mapped to the at least one representative dialog, the command shortcut including a text which is acquired based on the utterance and/or the user operation, wherein after the command shortcut is set, when an input including the utterance and/or the user operation is received from the user by the processor, at least a portion of the task is performed, and

transmit second data associated with the second user input to the external server via the communication circuit.

2. The electronic device of claim 1, wherein the instructions further cause the processor to:

receive a third user input via the at least one of the touch screen display or the microphone after the second data is transmitted, the third user input including the utterance and/or the user operation,

transmit third data associated with the third user input to the external server via the wireless communication circuit,

receive a second response from the external server via the wireless communication circuit, the second response including the sequence information of the states of the electronic device for performing at least the portion of the task, and

perform at least the portion of the task according to the sequence information.

3. The electronic device of claim 1, wherein the instructions further cause the processor to:

store a log associated with the first user input received via the touch screen display, and

transmit the log as a part of the first data.

4. The electronic device of claim 1, wherein the utterance includes an utterance personalized by the user, and

wherein the user operation includes a text input and/or an input selecting a graphic user interface by the user.

5. The electronic device of claim 1, wherein:

the at least one representative dialog includes a request to perform the task, and

the instructions further cause the processor to;

receive a voice via the microphone, and

transmit fourth data including another representative dialog corresponding to the voice to the external server.

6. The electronic device of claim 5, wherein the instructions further cause the processor to:

transmit the voice to the external server via the wireless communication circuit,

receive text data corresponding to the voice from the external server via the wireless communication circuit, and

identify the other representative dialog matched with the command shortcut if the text data corresponds to the command shortcut.

7. The electronic device of claim 5, wherein the instructions further cause the processor to:

receive the sequence information of the states of the electronic device that corresponds to the second data and is used to perform at least the portion of the task via the wireless communication circuit after the second data is transmitted, and

perform at least the portion of the task.

8. The electronic device of claim 1, wherein:

the at least one representative dialog includes a request to perform the task, and

the instructions further cause the processor to;

receiving a text input via the touch screen display, and

transmit fifth data including another representative dialog corresponding to the text input to the external server.

9. The electronic device of claim 8, wherein the instructions further cause the processor to identify the other representative dialog matched with the command shortcut if the text input corresponds to the command shortcut.

10. A method of executing an operation of an electronic device, the method comprising:

receiving a first user input, wherein the first user input includes a request to perform a task using at least one of the electronic device or an external device;

transmitting first data associated with the first user input to an external server;

receiving a first response from the external server, wherein the first response includes information associated with the first user input and/or sequence information of states of the electronic device for performing at least a portion of the task, wherein the sequence information corresponds to at least one representative dialog, and wherein when an input corresponding to the at least one representative dialog is received from a user, at least a portion of the task is performed;

after receiving the first response, in response to receiving a second user input including an utterance and/or a user operation via a touch screen display of the electronic device, setting a command shortcut to be mapped to the at least one representative dialog, the command shortcut including a text which is acquired based on the utterance and/or the user operation, wherein after the command shortcut is set, when an input including the utterance and/or the user operation is received from the user, at least a portion of the task is performed; and

transmitting second data associated with the second user input to the external server.

11. The method of claim 10, further comprising:

receiving a third user input after the second data is transmitted, wherein the third user input includes the utterance and/or the user operation;

transmitting third data associated with the third user input to the external server;

receiving a second response from the external server, wherein the second response includes the sequence information of the states of the electronic device for performing at least the portion of the task; and

performing at least the portion of the task according to the sequence information.

12. The method of claim 10, further comprising:

storing a log associated with the first user input received via the touch screen display of the electronic device, and

transmitting the log as a part of the first data.

13. The method of claim 10, wherein the utterance second user includes an utterance personalized by a user, and

wherein the user operation includes a text input and/or an input selecting a graphic user interface by the user.

14. The method of claim 10, further comprising:

storing the at least one representative dialog including a request to perform the task;

receiving a voice via a microphone; and

transmitting fourth data including another representative dialog corresponding to the voice to the external server.

15. The method of claim 14, further comprising:

transmitting the voice to the external server;

receiving text data corresponding to the voice from the external server; and

identifying the other representative dialog matched with the command shortcut if the text data corresponds to the command shortcut.

16. A non-transitory machine-readable storage medium recording a program to perform a method of executing an operation of an electronic device, the method comprising:

receiving a first user input, wherein the first user input includes a request to perform a task using at least one of the electronic device or an external device;

transmitting first data associated with the first user input to an external server;

receiving a first response from the external server, wherein the first response includes information associated with the first user input and/or sequence information of states of the electronic device for performing at least a portion of the task, wherein the sequence information corresponds to at least one representative dialog, and wherein when an input corresponding to the at least one representative dialog is received from a user, at least a portion of the task is performed;

after receiving the first response, in response to receiving a second user input including an utterance and/or a user operation via a touch screen display of the electronic device, setting a command shortcut to be mapped to the at least one representative dialog, the command shortcut including a text which is acquired based on the utterance and/or the user operation, wherein after the command shortcut is set, when an input including the utterance and/or the user operation is received from the user, at least a portion of the task is performed; and

transmitting second data associated with the second user input to the external server.