ELECTRONIC DEVICE FOR PROCESSING MULTI-MODAL INPUT, METHOD FOR PROCESSING MULTI-MODAL INPUT AND SEVER FOR PROCESSING MULTI-MODAL INPUT

• US 20180150280A1
• Filed: 11/28/2017
• Published: 05/31/2018
• Est. Priority Date: 11/28/2016
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: RHEE, In Jong, LEE, Ji Min, KANG, Sang Ki, KU, Han Jun, PARK, Sung Pa, SEO, Jang Seok, SONG, In Wook, AHN, Won Ick, WOO, Kyoung Gu, YI, Ji Soo, JEON, Chang Kyun, JAYGARL, Ho Jun, CHOI, Il Hwan, HONG, Yoo Jin, KIM, Ji Hyun, YEO, Jae Yung
• Granted Patent: US 10,191,718 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04R 1/028 associated with devices performing functions other than acoustics, e.g. electric candles

Abstract

An electronic device is provided. The electronic device includes a housing, a touchscreen display exposed through a first portion of the housing, a microphone disposed at a second portion of the housing, a speaker disposed at a third portion of the housing, a memory disposed inside the housing, a processor disposed inside the housing, and electrically connected to the display, the microphone, the speaker, and the memory. The memory is configured to store a plurality of application programs, each of which includes a graphic user interface (GUI).

Claims

1. An electronic device comprising:

a housing;

a touchscreen display exposed through a first portion of the housing;

a microphone disposed at a second portion of the housing;

a speaker disposed at a third portion of the housing;

a memory disposed inside the housing; and

a processor disposed inside the housing, and electrically connected to the display, the microphone, the speaker, and the memory;

wherein the memory is configured to store a plurality of application programs, each of which includes a graphic user interface (GUI), and

wherein the memory is further configured to store instructions that, when executed, cause the processor to;

control the microphone to receive an utterance from a user;

generate or receive text data based on the utterance;

select one of the applications programs, based at least partly on the text data;

launch the selected application program;

control to display the GUI of the selected application program on the display;

control to display at least part of a sequence of automatically providing an input into the GUI on the display, based at least partly on the text data;

cause the selected application program to perform an operation that utilizes at least one parameter, using the input as the at least one parameter; and

control to display a result of the operation on the display.

2. The electronic device of claim 1, wherein the instructions cause the processor to download at least one of the plurality of application programs from a server.

3. The electronic device of claim 1, wherein the instructions are part of a software framework that provides an application programming interface to the plurality of application programs.

4. The electronic device of claim 1, wherein the instructions further cause the processor to:

control to display, on the display, one or more identifiers, each of which represents a respective one of the plurality of application programs, based at least partly on the text data;

receive a selection of one of the identifiers from the user via the display or the microphone; and

select the one of the application programs, based on the received selection.

5. The electronic device of claim 4, wherein the identifiers are in the form of at least one of an icon, a symbol, an image, or text.

6. The electronic device of claim 1, wherein, when the text data lacks a parameter for the selected application program to perform the operation, the instructions further cause the processor to:

control to display a cursor in an input area of the GUI for receiving the parameter;

receive text input for the parameter via the display; and

cause the application program to perform the operation, using the text input.

7. The electronic device of claim 1, wherein the instructions further cause the processor to:

generate or receive intent of the utterance and the parameter for performing the intent based on the text data;

generate or receive a path rule including a sequence of the operation of the selected application program based on the intent and the parameter;

if the one application program is selected, select the one application program depending on the path rule;

if the selected application program performs the operation, perform the operation included in the path rule depending on the sequence of the path rule; and

if the result of the operation is displayed on the display, control to display the result of the operation included in the path rule on the display depending on the performed sequence.

8. The electronic device of claim 7, wherein the one path rule includes the operation of the selected application program and the parameter utilized to perform the operation.

9. An electronic device comprising:

a housing;

a touchscreen display exposed through a first portion of the housing;

a microphone disposed at a second portion of the housing;

a speaker disposed at a third portion of the housing;

a memory disposed inside the housing; and

a processor disposed inside the housing, and electrically connected to the display, the microphone, the speaker, and the memory;

wherein the memory is configured to store a first plurality of application programs and a second plurality of application programs, wherein each of the first and second plurality of application programs includes a graphic user interface (GUI), and

wherein the memory is further configured to store instructions that, when executed, cause the processor to;

control the microphone to receive an utterance from a user;

generate or receive text data based on the utterance;

select a first application program from the first plurality of application programs, based at least partly on the text data;

launch the selected first application program;

control to display the GUI of the selected first application program on the display;

cause the selected first application program to perform a first operation that utilizes at least one first parameter, based at least partly on the text data, while displaying at least part of a sequence of performing the first operation on the display;

control to display a result of the first operation on the display;

after displaying the result, control to display one or more identifiers on the display, based at least partly on the text data, wherein each of the identifiers represents a respective one of the second plurality of application programs;

receive a selection of one of the identifiers from the user via the display or the microphone;

select a second application program from the second plurality of application programs, based on the received selection; and

cause the selected second application program to perform a second operation that utilizes at least one second parameter, using at least part of the result of the first operation.

10. The electronic device of claim 9, wherein the instructions cause the processor to download at least one of the first or second plurality of application programs from a server.

11. The electronic device of claim 9, wherein the instructions are part of a software framework that provides an application programming interface to the first and second pluralities of application programs.

12. The electronic device of claim 9, wherein the instructions further cause the processor to:

after selecting the second application program, control to display the GUI of the selected second application program on the display;

cause the selected second application program to perform the second operation, while displaying at least part of a sequence of performing the second operation on the display; and

control to display a result of the second operation on the display.

13. The electronic device of claim 9, wherein the identifiers are in the form of at least one of an icon, a symbol, an image, or text.

14. The electronic device of claim 9, wherein, when the text data lacks a first parameter for the first application program to perform the first operation, the instructions further cause the processor to:

control to display an input area of the GUI of the first application program on the display for receiving the first parameter;

receive a first input for the first parameter via the display or the microphone; and

cause the first application program to perform the operation, using the first input.

15. The electronic device of claim 9, wherein, when the text data lacks a second parameter for the second application program to perform the second operation, the instructions further cause the processor to:

control to display an input area of the GUI of the second application program on the display for receiving the second parameter;

receive a second input for the second parameter via the display or the microphone; and

cause the second application program to perform the operation, using the second input.

16. An electronic device comprising:

a housing;

a touchscreen display exposed through a first portion of the housing;

a microphone disposed at a second portion of the housing;

a speaker disposed at a third portion of the housing;

a memory disposed inside the housing; and

a processor disposed inside the housing, and electrically connected to the display, the microphone, the speaker, and the memory;

wherein the memory is configured to store a plurality of application programs, each of which includes a graphic user interface (GUI), and

wherein the memory is further configured to store instructions that, when executed, cause the processor to;

control the microphone to receive a verbal request from a user;

generate or receive text data based on the request;

select one of the applications programs, based at least partly on the text data;

determine a portion of an operation that can be performed automatically on the selected application program, using the text data;

launch the selected application program;

control to display the GUI of the selected application program on the display;

control to display at least part of a sequence of automatically performing the portion of the operation, based at least partly on the text data, without completing the operation;

receive a user input via the display or microphone; and

cause the application program to complete the operation, using the received user input.

17. The electronic device of claim 16, wherein the instructions are part of a software framework that provides an application programming interface provided to the plurality of application programs.

18. The electronic device of claim 16, wherein the instructions further cause the processor to:

control to store user information in the memory; and

when the text data lacks a parameter for the selected application program to perform the operation, propose the parameter to be added to the operation based on the stored user information.

19. A voice input processing method, the method comprising:

receiving an utterance from a user via a microphone;

generating or receiving text data based on the utterance;

selecting one application program based at least partly on the text data;

launching the selected application program;

displaying a graphic user interface (GUI) of the selected application program on a display;

displaying at least part of a sequence of automatically providing an input into the GUI on the display, based at least partly on the text data;

causing the selected application program to perform an operation that utilizes at least one parameter, using the input as the at least one parameter; and

displaying a result of the operation on the display.