Displaying GUI elements on natural user interfaces

• US 8,261,212 B2
• Filed: 10/20/2009
• Issued: 09/04/2012
• Est. Priority Date: 10/20/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wigdor, Daniel J., Hoover, Paul Armistead
• Primary Examiner(s): Belousov, Andrey
• Application Number:
  US12/582,156
• Publication Number:
  US 20110093821A1
• Time in Patent Office: 1,050 Days
• Field of Search: 715/863, 715/790
• US Class Current: 715/863
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

A computing system for displaying a GUI element on a natural user interface is described herein. The computing system includes a display configured to display a natural user interface of a program executed on the computing system, and a gesture sensor configured to detect a gesture input directed at the natural user interface by a user. The computing system also includes a processor configured to execute a gesture-recognizing module for recognizing a registration phase, an operation phase, and a termination phase of the gesture input, and a gesture assist module configured to first display a GUI element overlaid upon the natural user interface in response to recognition of the registration phase. The GUI element includes a visual or audio operation cue to prompt the user to carry out the operation phase of the gesture input, and a selector manipulatable by the user via the operation phase of the gesture.

Claims

1. A computing system for displaying a GUI element on a natural user interface on a display, the computing system comprising:

a display configured to display a natural user interface of a program executed on the computing system;

a gesture sensor configured to detect a gesture input directed at the natural user interface by a user; and

a processor configured to execute;

a gesture-recognizing module configured to recognize a registration phase, an operation phase, and a termination phase of the gesture input;

a gesture assist module configured to display a GUI element overlaid upon the natural user interface, the GUI element being first displayed in response to recognition of the registration phase by the gesture-recognizing module, and including a visual or audio operation cue to prompt the user to carry out the operation phase of the gesture input, and a selector manipulatable by the user via the operation phase of the gesture;

where upon receiving user input via the selector the gesture-recognizing module is configured to send the operation phase to the program, and the program is configured to determine a parameter adjustment based upon the user input received in the operation phase;

where the processor is further configured to execute a natural user interface control module configured to perform a natural user interface update responsive to recognition of the operation phase and based on the parameter adjustment determined by the program;

where the gesture assist module is further configured to receive the parameter adjustment from the program and perform a GUI element update based on the parameter adjustment; and

where the natural user interface update and GUI element update are sent to the display.

2. The computing system of claim 1, where the gesture-recognizing module includes a gesture library for matching the registration phase of the gesture input to the GUI element.

3. The computing system of claim 2, where the gesture assist module is configured to specify an appearance and/or a position of the GUI element for display.

4. The computing system of claim 3, where the gesture assist module is configured to specify the position of the GUI element based on a program that has focus.

5. The computing system of claim 3, where the gesture assist module is further configured to specify the appearance of the GUI element based on a program that has focus such that the GUI element is program-specific.

6. The computing system of claim 1, where the gesture sensor includes a touch screen configured to detect a touch input from a user.

7. The computing system of claim 6, where the gesture recognizing module is configured to recognize a multi-touch gesture input.

8. The computing system of claim 7, where the gesture sensor includes a depth-sensing camera configured to detect a three-dimensional gesture input from a user.

9. A method for displaying a GUI element on a natural user interface on a display, the method comprising:

displaying, on a display, a natural user interface of a program executed on the computing system;

recognizing, at a gesture-recognizing module executable by a processor, a registration phase of a gesture input directed at the natural user interface by a user;

displaying a GUI element associated with the registration phase for a first time, by a gesture assist module executable by the processor, the displaying including overlaying the GUI element upon the natural user interface in response to the recognition of the registration phase, and the GUI element including a visual or audio operation cue to prompt the user to carry out an operation phase of the gesture input, and a selector manipulatable by the user via performance of the operation phase of the gesture input;

recognizing the operation phase of the gesture input at the gesture-recognizing module;

updating the natural user interface, responsive to recognition of the operation phase, to provide the user with feedback regarding an effect of the operation phase;

where, upon receiving user input via the selector, the gesture-recognizing module is configured to send the operation phase to the program, and the program is configured to determine a parameter adjustment based upon the user input received in the operation phase;

where the processor is further configured to execute a natural user interface control module configured to perform a natural user interface update responsive to recognition of the operation phase and based on the parameter adjustment determined by the program;

where the gesture assist module is further configured to receive the parameter adjustment from the program and perform a GUI element update based on the parameter adjustment; and

where the natural user interface update and GUI element update are sent to the display.

10. The method of claim 9, further comprising matching the registration phase of the gesture input to a GUI element using a gesture library at the gesture-recognizing module, responsive to recognizing the registration phase.

11. The method of claim 10, where displaying the GUI element includes specifying a GUI element appearance and/or a GUI element position.

12. The method of claim 9, further comprising determining a parameter adjustment associated with the operation phase by the program, where the updating of the natural user interface is based on the parameter adjustment.

13. The method of claim 12, further comprising updating the GUI element based on the operation phase to reflect the updating of the natural user interface.

14. The method of claim 12, further comprising modifying the GUI element based on a program that has focus such that the GUI element overlaid on the natural user interface is program-specific.

15. The method of claim 9, further comprising recognizing a termination phase of the gesture input, and in response, stopping the display of the GUI element.

16. The method of claim 9, further comprising recognizing an initiation phase of the gesture input before recognizing the registration phase and, in response to recognizing the initiation phase, displaying a mode selection cue to assist a user in performing the registration phase.

17. A computer-readable medium having instructions stored thereon that are executable by a processor to:

display, on a display, a natural user interface of a program executed on the computing system;

recognize, at a gesture-recognizing module, a registration phase of a gesture input directed at the natural user interface by a user;

display a GUI element associated with the registration phase for a first time, by a gesture assist module, the displaying including overlaying the GUI element upon the natural user interface in response to the recognition of the registration phase, and the GUI element including a visual or audio operation cue to prompt the user to carry out an operation phase of the gesture input, and a selector manipulatable by the user via performance of the operation phase of the gesture input;

recognize the operation phase of the gesture input at the gesture-recognizing module;

update the natural user interface, responsive to recognition of the operation phase, to provide the user with feedback regarding an effect of the operation phase;

where, upon receiving user input via the selector, the gesture-recognizing module is configured to send the operation phase to the program, and the program is configured to determine a parameter adjustment based upon the user input received in the operation phase;

where the processor is further configured to execute a natural user interface control module configured to perform a natural user interface update responsive to recognition of the operation phase and based on the parameter adjustment determined by the program;

where the gesture assist module is further configured to receive the parameter adjustment from the program and perform a GUI element update based on the parameter adjustment; and

where the natural user interface update and GUI element update are sent to the display.

18. The computer-readable medium of claim 17, wherein displaying of the GUI element includes specifying a GUI element appearance and/or a GUI element position based on a program that has focus.