Device and method for performing functions

• US 10,127,011 B2
• Filed: 07/30/2015
• Issued: 11/13/2018
• Est. Priority Date: 07/31/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Bang, Han-min, Choi, Hyok-sung
• Primary Examiner(s): Belousov, Andrey
• Application Number:
  US14/813,959
• Publication Number:
  US 20160034253A1
• Time in Patent Office: 1,202 Days
• Field of Search: 715728
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/5044 considering hardware capabilities
• CPC Class Codes: G06F 9/5055 considering software capabilities, i.e. software resources associated or available to the machine
• CPC Class Codes: H04M 1/724 User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04M 1/7243 with interactive means for internal management of messages
• CPC Class Codes: H04M 2250/74 with voice recognition means

Abstract

Provided is a device including a display, an audio inputter, and a controller. The display displays at least one screen page of an application that is being executed. The audio inputter receives a voice command of a user. The controller performs an operation corresponding to the voice command by using screen page transition information for transition between application screen pages corresponding to the voice command, which is obtained from information about user interface (UI) elements included in the application screen pages of the application. Each of the UI elements performs a predetermined function when selected by the user.

Claims

1. A device comprising:

a display configured to display a first screen of an application comprising a first user interface (UI) element;

a microphone configured to receive a voice command for the application; and

a processor configured to;

determine a second screen of the application, comprising a second UI element, for performing a function corresponding to the voice command, based on a screen component information of the application, by using each of page identification information of the first screen and the second screen, transition information, and action information that defines an action corresponding to a combination of screen pages by using function information of the first UI element and function information of the second UI element,

control the display to display the second screen by using the transition information for transitioning from the first screen to the second screen, and

perform the function corresponding to the voice command on the second screen,

wherein the transition information is obtained based on the first screen and the second screen.

2. The device of claim 1, wherein the processor is further configured to determine a transition path between the first screen and the second screen based on the transition information.

3. The device of claim 2, wherein the processor is further configured to determine a plurality of transition paths for transfer from the first screen to the second screen, and select a shortest path among the plurality of transition paths.

4. The device of claim 2, wherein the determined transition path comprises the first screen, at least one intermediate screen, and the second screen, and

wherein the processor is further configured to sequentially display the first screen, the at least one intermediate screen, and the second screen on the display based on the transition path.

5. The device of claim 4, wherein the processor is further configured to generate an input event for controlling at least one UI element connecting the first screen, the at least one intermediate screen, and the second screen to sequentially display the first screen, the at least one intermediate screen, and the second screen.

6. The device of claim 5, wherein the processor is further configured to generate information about UI elements included in the first screen and the second screen by using information acquired to render the first screen and the second screen.

7. The device of claim 1, wherein the transition information comprises at least one of identification information of UI elements connecting the first screen and the second screen and input event information for controlling the UI elements connecting the first screen and the second screen.

8. The device of claim 1, wherein the processor is further configured to provide the voice command to a server and receive analysis results of the voice command from the server.

9. The device of claim 1, wherein the processor is further configured to identify the first screen displayed on the display based on information about UI elements included in the first screen.

10. The device of claim 1, further comprising a memory configured to store performance result data of the function.

11. A method of performing a function by a device, the method comprising:

displaying a first screen of an application comprising a first user interface (UI) element;

receiving a voice command for the application;

determining a second screen of the application, comprising a second UI element, for performing a function corresponding to the voice command based on a screen component information of the application, by using each of page identification information of the first screen and the second screen, transition information, and action information that defines an action corresponding to a combination of screen pages by using function information of the first UI element and function information of the second UI element;

displaying the second screen by using the transition information for transitioning from the first screen to the second screen; and

performing the function corresponding to the voice command on the second screen,

wherein the transition information is obtained based on the first screen and the second screen.

12. The method of claim 11, wherein the performing the function corresponding to the voice command comprises determining a transition path between the first screen and the second screen based on the transition information.

13. The method of claim 12, wherein the determining the transition path comprises:

determining a plurality of transition paths for transfer from the first screen to the second screen; and

selecting a shortest path among the plurality of transition paths.

14. The method of claim 12, wherein the determined transition path comprises the first screen, at least one intermediate screen, and the second screen.

15. The method of claim 14, wherein the displaying the first screen and the second screen comprises generating an input event for controlling at least one UI element connecting the first screen, the at least one intermediate screen, and the second screen to sequentially display the first screen, the at least one intermediate screen, and the second screen.

16. The method of claim 15, further comprising generating information about UI elements included in the first screen and the second screen by using information acquired to render the first screen and the second screen.

17. The method of claim 11, wherein the receiving the voice command comprises:

providing the voice command to a server; and

receiving analysis results of the voice command from the server.

18. The method of claim 11, wherein the performing the function corresponding to the voice command comprises identifying the first screen displayed on a display based on information about UI elements included in the first screen.

19. A non-transitory computer-readable recording medium having recorded thereon a program executable by a computer for performing the method of claim 11.

20. A method of performing a function by a device, the method comprising:

receiving a user input for requesting performance of at least one function;

reading screen component information of at least one application from a storage;

comparing information about a first user interface (UI) element included in a current page to information about UI elements of the screen component information of the at least one application;

determining a target page, comprising a second UI element, corresponding to the user input according to the screen component information of the at least one application, by using each of page identification information of the current page and the target page, transition information, and action information that defines an action corresponding to a combination of screen pages by using function information of the first UI element and function information of the second UI element;

determining a path for transitioning automatically from the current page to the target page by using the transition information, wherein the target page is a page for providing the at least one function, based on a result of the comparing;

transitioning to the target page according to the determined path; and

performing the at least one function through the target page.

21. The method of claim 20, wherein the at least one function comprises at least one of a message transmission function, an alarm function, a phone call function, a content upload function, an image capture function, a sound recording function, and an information search function.

22. The method of claim 20, wherein the user input comprises at least one of a voice input, a gesture input, a motion input, a bending input, a key input, an eye input, and a combination input.

23. The method of claim 20, wherein the transitioning to the target page comprises generating an input event for displaying the target page.

24. The method of claim 20, further comprising:

at least one intermediate page, and

wherein the determined path comprises the current page, the at least one intermediate page, and the target page.