Gesture library for natural user input

• US 9,971,491 B2
• Filed: 01/09/2014
• Issued: 05/15/2018
• Est. Priority Date: 01/09/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Schwesinger, Mark, Yang, Emily, Kapur, Jay, Paolantonio, Sergio, Klein, Christian, Murillo, Oscar
• Primary Examiner(s): Trapanese, William C
• Application Number:
  US14/151,744
• Publication Number:
  US 20150193107A1
• Time in Patent Office: 1,587 Days
• Field of Search: None
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A method to decode natural user input from a human subject. The method includes detection of a gesture and concurrent grip state of the subject. If the grip state is closed during the gesture, then a user-interface (UI) canvas of the computer system is transformed based on the gesture. If the grip state is open during the gesture, then a UI object arranged on the UI canvas is activated based on the gesture.

Claims

1. Enacted in a computing machine operatively coupled to a display and to a vision system having at least one camera, a method to decode natural user input (NUI) from a human subject, the method comprising:

displaying an operating system (OS) shell of an OS of the computing machine, the OS shell including a user-interface (UI) element for each of a plurality of applications of the computing machine,

detecting in three dimensions a hand gesture and concurrent grip state of the human subject while displaying the OS shell;

modifying an extent of visibility of the OS shell based on the gesture if the grip state is close-fisted, wherein modifying the extent of visibility includes displaying a larger portion of the OS shell than currently displayed, to enable different applications to be launched; and

launching an application of the computing machine if the gesture is directed to a UI element presented on the OS shell and associated with the application and if the grip state is open-handed.

2. The method of claim 1, wherein the gesture includes motion of at least one hand of the subject.

3. The method of claim 1, further comprising redrawing the OS shell when the extent of visibility of the OS shell is modified.

4. The method of claim 1, wherein the open-handed grip-state gesture is a push gesture.

5. The method of claim 1, wherein the UI element is one or more of a tile, icon, or control.

6. The method of claim 1, wherein detecting the grip state includes resolving a change in the grip state, wherein a closing grip-state change triggers transition to a mode of modifying the extent of visibility of the OS shell, and wherein an opening grip-state change triggers transition to a mode of launching the application.

7. The method of claim 1 wherein displaying the larger portion of the OS shell exposes a plurality of different UI elements, which correspond to the different applications to be launched.

8. The method of claim 1 wherein displaying the larger portion of the OS shell exposes a greater number of UI elements than were exposed prior to the close-fisted gesture.

9. A computing machine configured to decode natural user input (NUI) from a human subject, the computing machine operatively coupled to a display and to a vision system having at least one camera and configured to detect a hand gesture of the subject in three dimensions, the computing machine comprising:

an operating system (OS) accessible to a user of the computing machine via an OS shell, the OS shell being configured to present a user-interface (UI) element for each of a plurality of applications of the computing machine, the OS having an application programming interface (API) to provide, based on the detected hand gesture, input to one or more processes of the computing machine, the input including;

input to signal engagement of the subject as a user of the computing machine,

input to signal disengagement of the subject as a user of the computing machine,

input to shift subsequent input focus to the OS shell upon detection of a close-fisted gesture, wherein shifting subsequent input focus includes reducing a process currently executing on the computing machine to an icon and displaying a larger portion of the OS shell than currently displayed, and

input to launch an application of the computing machine upon detection of an open-handed push gesture directed to a UI element presented on the OS shell and associated with the application.

10. The computing machine of claim 9, wherein the open-handed gesture includes a press motion followed by a release motion.

11. The computing machine of claim 9, wherein the input to modify an extent of visibility of the OS is provided upon detection in the vision system of a two-handed stretch or compression gesture with both fists closed.

12. The computing machine of claim 9, wherein the input to signal engagement of the human subject as a user of the computing machine is provided upon detection of a raised hand and open palm of the subject as the subject faces the display.

13. The computing machine of claim 9, wherein the input to signal disengagement of the human subject as a user of the computing machine is provided upon detection of both hands of the subject dropping below a pre-defined zone of interaction.

14. The computing machine of claim 9, wherein the input to the one or more processes further includes input to return to the OS shell from a currently running application, and wherein such input is provided upon detection of a two-handed zoom-out gesture by the subject.

15. The computing machine of claim 9, wherein the input to the one or more processes further includes input to open or expand the UI element, and wherein such input is provided upon detection of a two-handed zoom-in gesture by the subject.

16. Enacted in a computer system operatively coupled to a vision system, a method to decode natural user input (NUI) from a human subject, the method comprising:

displaying an operating system (OS) shell of an OS of the computing machine, the OS shell including a user-interface (UI) element for each of a plurality of applications of the computing machine;

from a set of gestural inputs detectable by the vision system, selecting a context-appropriate subset based on an operating condition of the computer system;

from a set of actions executable in the computer system, selecting a context-appropriate subset based on the operating condition of the computer system, the operating condition including a grip state of the human subject;

detecting a gestural input from the context-appropriate subset of gestural inputs while displaying the OS shell; and

executing an action from the context-appropriate subset of executable actions responsive to the detected gestural input, the action including modifying an extent of visibility of the OS shell based on the gesture if the grip state is close-fisted, wherein modifying the extent of visibility includes displaying a larger portion of the OS shell than currently displayed, to enable different applications to be launched; and

launching an application of the computing machine if the gesture is directed to a UI element presented on the OS shell and associated with the application and if the grip state is open-handed.

17. The method of claim 16, wherein the action includes seizing display or input focus from the process currently running.