DISPLAYING GUI ELEMENTS ON NATURAL USER INTERFACES

• US 20110093821A1
• Filed: 10/20/2009
• Published: 04/21/2011
• Est. Priority Date: 10/20/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wigdor, Daniel J., Hoover, Paul Armistead
• Granted Patent: US 8,261,212 B2
• Time in Patent Office: Days
• US Class Current: 715/863
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

A computing system for displaying a GUI element on a natural user interface is described herein. The computing system includes a display configured to display a natural user interface of a program executed on the computing system, and a gesture sensor configured to detect a gesture input directed at the natural user interface by a user. The computing system also includes a processor configured to execute a gesture-recognizing module for recognizing a registration phase, an operation phase, and a termination phase of the gesture input, and a gesture assist module configured to first display a GUI element overlaid upon the natural user interface in response to recognition of the registration phase. The GUI element includes a visual or audio operation cue to prompt the user to carry out the operation phase of the gesture input, and a selector manipulatable by the user via the operation phase of the gesture.

Claims

1. A computing system for displaying a GUI element on a natural user interface on a display, the computing system comprising:

a display configured to display a natural user interface of a program executed on the computing system;

a gesture sensor configured to detect a gesture input directed at the natural user interface by a user; and

a processor configured to execute;

a gesture-recognizing module configured to recognize a registration phase, an operation phase, and a termination phase of the gesture input; and

a gesture assist module configured to display a GUI element overlaid upon the natural user interface, the GUI element being first displayed in response to recognition of the registration phase by the gesture-recognizing module, and including a visual or audio operation cue to prompt the user to carry out the operation phase of the gesture input, and a selector manipulatable by the user via the operation phase of the gesture.

2. The computing system of claim 1, where the gesture-recognizing module includes a gesture library for matching the registration phase of the gesture input to the GUI element.

3. The computing system of claim 2, where the gesture assist module is configured to specify an appearance and/or a position of the GUI element for display.

4. The computing system of claim 3 where the gesture assist module is configured to specify the position of the GUI element based on a program that has focus.

5. The computing system of claim 3, where the gesture assist module is further configured to specify the appearance of the GUI element based on a program that has focus such that the GUI element is program-specific.

6. The computing system of claim 1, where the processor is further configured to execute a natural user interface control module configured to update the natural user interface responsive to recognition of the operation phase.

7. The computing system of claim 6, where the gesture assist module is further configured to update the GUI element to reflect the update of the natural user interface.

8. The computing system of claim 1, where the gesture sensor includes a touch screen configured to detect a touch input from a user.

9. The computing system of claim 8, where the gesture recognizing module is configured to recognize a multi-touch gesture input.

10. The computing system of claim 9, where the gesture sensor includes a depth-sensing camera configured to detect a three-dimensional gesture input from a user.

11. A method for displaying a GUI element on a natural user interface on a display, the method comprising:

displaying, on a display, a natural user interface of a program executed on the computing system;

recognizing, at a gesture-recognizing module executable by a processor, a registration phase of a gesture input directed at the natural user interface by a user;

displaying a GUI element associated with the registration phase for a first time, by a gesture assist module executable by the processor, the displaying including overlaying the GUI element upon the natural user interface in response to the recognition of the registration phase, and the GUI element including a visual or audio operation cue to prompt the user to carry out the operation phase of the gesture input, and a selector manipulatable by the user via performance of the operation phase of the gesture;

recognizing an operation phase of the gesture input at the gesture-recognizing module; and

updating the natural user interface, responsive to recognition of the operation phase, to provide the user with feedback regarding an effect of the operation phase.

12. The method of claim 11, further comprising matching the registration phase of the gesture input to a GUI element using a gesture library at the gesture-recognizing module, responsive to recognizing the registration phase.

13. The method of claim 12, where displaying the GUI element includes specifying a GUI element appearance and/or a GUI element position.

14. The method of claim 11, further comprising determining a parameter adjustment associated with the operation phase by the program, where the updating of the natural user interface is based on the parameter adjustment.

15. The method of claim 14, further comprising updating the GUI element based on the operation phase to reflect the updating of the natural user interface.

16. The method of claim 14, further comprising modifying the GUI element based on a program that has focus such that the GUI element overlaid on the natural user interface is program-specific.

17. The method of claim 11, further comprising recognizing a termination phase of the gesture input, and in response, stopping the display of the GUI element.

18. The method of claim 11, further comprising recognizing an initiation phase of a gesture input before recognizing the registration phase and, in response to recognizing the initiation phase, displaying a mode selection cue to assist a user in performing the registration phase.

19. A computer-readable medium having instructions stored thereon that are executable by a processor to:

display, on a display, a natural user interface of a program executed on the computing system;

recognize, at a gesture-recognizing module, a registration phase of a gesture input directed at the natural user interface by a user;

display a GUI element associated with the registration phase for a first time, by a gesture assist module, the displaying including overlaying the GUI element upon the natural user interface in response to the recognition of the registration phase, and the GUI element including a visual or audio operation cue to prompt the user to carry out the operation phase of the gesture input, and a selector manipulatable by the user via performance of the operation phase of the gesture;

recognize an operation phase of the gesture input at the gesture-recognizing module; and

update the natural user interface, responsive to recognition of the operation phase, to provide the user with feedback regarding an effect of the operation phase.

20. The computer-readable medium of claim 19, wherein displaying of the GUI element includes specifying a GUI element appearance and/or a GUI element position based on a program that has focus.