DEVICE AND METHOD FOR PERFORMING FUNCTIONS

• US 20160034253A1
• Filed: 07/30/2015
• Published: 02/04/2016
• Est. Priority Date: 07/31/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: CHOI, Hyok-sung, BANG, Han-min
• Granted Patent: US 10,127,011 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/5044 considering hardware capabilities
• CPC Class Codes: G06F 9/5055 considering software capabilities, i.e. software resources associated or available to the machine
• CPC Class Codes: H04M 1/724 User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04M 1/7243 with interactive means for internal management of messages
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00

Abstract

Provided is a device including a display, an audio inputter, and a controller. The display displays at least one screen page of an application that is being executed. The audio inputter receives a voice command of a user. The controller performs an operation corresponding to the voice command by using screen page transition information for transition between application screen pages corresponding to the voice command, which is obtained from information about user interface (UI) elements included in the application screen pages of the application. Each of the UI elements performs a predetermined function when selected by the user.

Claims

1. A device comprising:

a display configured to display at least one screen page of an application;

an audio inputter configured to receive audio of a user that includes a first voice command related to a first application and a second voice command related to a second application; and

a controller configured to;

perform a first operation corresponding to the first voice command by using first screen page transition information for transitioning between screen pages of the first application, which is obtained from information about user interface (UI) elements included in the screen pages of the first application corresponding to the first voice command,

provide the second application with performance result data of the first operation as input data of the second application corresponding to the second voice command, and

perform a second operation corresponding to the second voice command by using second screen page transition information for transitioning between screen pages of the second application, which is obtained from information about UI elements included in the screen pages of the second application corresponding to the second voice command,

wherein each of the UI elements is configured to perform a predetermined function in response to being selected by the user.

2. The device of claim 1, further comprising a memory configured to store the performance result data of the first operation.

3. The device of claim 2, wherein the controller is further configured to receive the performance result data of the first operation from the memory as the input data of the second application.

4. The device of claim 1, wherein the controller is further configured to determine a first transition path between the screen pages of the first application corresponding to the first voice command by using the first screen page transition information, and determine a second transition path between the screen pages of the second application corresponding to the second voice command by using the second screen page transition information.

5. The device of claim 4, wherein the controller is further configured to display the screen pages of the first application and the screen pages of the second application on the display sequentially based on the first transition path and the second transition path.

6. The device of claim 5, wherein the controller is further configured to generate an input event for controlling at least one UI element connecting the screen pages of the first application and the screen pages of the second application, to sequentially display the screen pages of the first application and the screen pages of the second application.

7. The device of claim 1, wherein the information about the UI elements comprises at least one of identification information of the UI elements, type information of input events for controlling the UI elements, application screen information connected with the UI elements, and function information connected with the UI elements.

8. The device of claim 1, wherein the controller is further configured to generate the information about the UI elements included in the screen pages of the first application by using information acquired to render the screen pages of the first application.

9. The device of claim 1, wherein the controller is further configured to identify a current page displayed on the display based on at least one of the information about the UI elements included in the screen pages of the first application and the information about the UI elements included in the screen pages of the second application.

10. The device of claim 1, wherein

the first screen page transition information comprises at least one of identification information of the UI elements connecting the screen pages of the first application and input event information for controlling the UI elements connecting the screen pages of the first application, and

the second screen page transition information comprises at least one of identification information of the UI elements connecting the screen pages of the second application and input event information for controlling the UI elements connecting the screen pages of the second application.

11. The device of claim 1, wherein the audio inputter provides the audio to a server and receives analysis results of the audio from the server.

12. A method of performing a function by a device, the method comprising:

receiving an audio of a user that includes a first voice command related to a first application and a second voice command related to a second application;

performing a first operation corresponding to the first voice command by using first screen page transition information for transitioning between screen pages of the first application, which is obtained from information about user interface (UI) elements included in the screen pages of the first application corresponding to the first voice command;

providing the second application with performance result data of the first operation as input data of the second application corresponding to the second voice command; and

performing a second operation corresponding to the second voice command by using second screen page transition information for transitioning between screen pages of the second application, which is obtained from information about UI elements included in the screen pages of the second application corresponding to the second voice command,

wherein each of the UI elements performs a predetermined function in response to being selected by the user.

13. The method of claim 12, wherein the performing the first operation corresponding to the first voice command comprises storing the performance result data of the first operation in a memory.

14. The method of claim 13, wherein the performing the second operation corresponding to the second voice command comprises receiving the performance result data of the first operation from the memory as the input data of the second application.

15. The method of claim 12, wherein

the performing the first operation corresponding to the first voice command comprises determining a first transition path between the screen pages of the first application corresponding to the first voice command by using the first screen page transition information, and

the performing the second operation corresponding to the second voice command comprises determining a second transition path between the screen pages of the second application corresponding to the second voice command by using the second screen page transition information.

16. The method of claim 15, further comprising displaying the screen pages of the first application and the screen pages of the second application sequentially based on the first transition path and the second transition path.

17. The method of claim 16, wherein the displaying of the screen pages of the first application and the screen pages of the second application sequentially based on the first transition path and the second transition path comprises generating an input event for controlling at least one UI element connecting the screen pages of the first application and the screen pages of the second application.

18. The method of claim 12, further comprising generating the information about the UI elements included in the screen pages of the first application by using information acquired for rendering the screen pages of the first application.

19. The method of claim 12, wherein the receiving the audio comprises:

providing the audio to a server; and

receiving analysis results of the audio from the server.

20. A non-transitory computer-readable recording medium having recorded thereon a program executable by a computer for performing the method of claim 12.

21. A method of performing a function by a device, the method comprising:

receiving a user input for requesting performance of at least one function;

comparing information about user interface (UI) elements included in a current page to information about UI elements of screen component information of at least one application; and

determining a path for transitioning from the current page to a target page, which is a page for providing the at least one function, based on the comparison;

transitioning to the target page according to the determined path; and

performing the at least one function through the target page.

22. The method of claim 21, wherein the at least one function comprises at least one of a message transmission function, an alarm function, a phone call function, a content upload function, an image capture function, a sound recording function, and an information search function.

23. The method of claim 21, wherein the user input comprises at least one of a voice input, a gesture input, a motion input, a bending input, a key input, an eye input, and a combination input.

24. The method of claim 21, wherein the transitioning to the target page comprises generating an input event for displaying the target page.

25. The method of claim 21, further comprising:

at least one intermediate page, and

wherein the determined path comprises the current page, the at least one intermediate page, and the target page.