Gestures and gesture modifiers for manipulating a user-interface

• US 8,457,353 B2
• Filed: 05/18/2010
• Issued: 06/04/2013
• Est. Priority Date: 05/18/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Reville, Brendan, Vassigh, Ali, Klein, Christian, Poulos, Adam, Andersen, Jordan, Fleischman, Zack
• Primary Examiner(s): Wang, Claire X
• Assistant Examiner(s): NAKHJAVAN, SHERVIN K
• Application Number:
  US12/782,380
• Publication Number:
  US 20110289456A1
• Time in Patent Office: 1,113 Days
• Field of Search: 382/103, 382/106, 382/107, 382/154, 348/113, 348/114, 348/169
• US Class Current: 382/103
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

Gesture modifiers are provided for modifying and enhancing the control of a user-interface such as that provided by an operating system or application of a general computing system or multimedia console. Symbolic gesture movements are performed by a user in mid-air. A capture device generates depth images and a three-dimensional representation of a capture area including a human target. The human target is tracked using skeletal mapping to capture the mid-air motion of the user. Skeletal mapping data is used to identify movements corresponding to pre-defined gestures using gesture filters. Detection of a viable gesture can trigger one or more user-interface actions or controls. Gesture modifiers are provided to modify the user-interface action triggered by detection of a gesture and/or to aid in the identification of gestures.

Claims

1. A method of operating a user-interface using mid-air motion of a human target, comprising:

receiving a plurality of images from a capture device, the plurality of images including the human target;

tracking movement of the human target from the plurality of images using skeletal mapping of the human target;

determining from the skeletal mapping whether the movement of the human target satisfies one or more filters for a first mid-air gesture, the first mid-air gesture is performed by a first hand of the user;

determining whether a modifier for the first mid-air gesture is present by determining whether a second hand of the human target satisfies a position parameter of a first gesture modifier filter, the position parameter defines a plurality of ranges for a position of the second hand;

if the movement of the human target satisfies the one or more filters and the modifier is not present, performing a first user-interface action corresponding to the first mid-air gesture, the first user-interface action includes navigating through a list of menu items by a first amount; and

if the movement of the human target satisfies the one or more filters and the modifier is present, performing a second user-interface action corresponding to the first mid-air gesture, the second user-interface action includes navigating through the list of menu items by a second amount when the position of the second hand is in a first range and navigating through the list of menu items by a third amount when the position of the second hand is in a second range.

2. A method according to claim 1, wherein the second user-interface action is a modified version of the first user-interface action.

3. A method according to claim 1, wherein:

the position parameter defines a virtual quadrant for the second hand above the user'"'"'s waist.

4. A method according to claim 1, wherein determining whether the movement of the human target satisfies the one or more filters for the mid-air gesture includes:

determining whether a position of the first hand of the human target satisfies a starting position parameter,

determining whether a direction of movement of the first hand from the starting position satisfies a directional parameter, and

determining whether a distance traveled by the first hand during the movement satisfies a distance parameter.

5. A method according to claim 1, wherein:

determining whether the modifier for the first mid-air gesture is present includes determining whether the human target is holding a modifier object.

6. A method according to claim 5, wherein:

the first user-interface action is an operating system user-interface control; and

the second user-interface action is a game-play control.

7. A method according to claim 1, wherein:

determining whether the modifier for the first mid-air gesture is present includes determining whether the human target issues a speech modifier.

8. A method according to claim 1, wherein the plurality of images is a plurality of depth images.

9. One or more processor readable storage devices having processor readable code embodied on the one or more processor readable storage devices, the processor readable code for programming one or more processors to perform a method comprising:

receiving depth information of a capture area including a human target;

creating a skeletal model for mapping movement of the human target over time;

detecting a gesture modifier from the human target that indicates a gesture period by determining that the human target has raised a hand to a position satisfying a gesture modifier filter parameter;

determining a start of the gesture period based on a time at which the human target raises the hand;

in response to detecting the gesture modifier, determining from the skeletal mapping whether movement of the human target during the gesture period satisfies one or more filters for a first mid-air gesture;

if the movement of the human target during the gesture period satisfies the one or more filters, performing a first user-interface action corresponding to the first mid-air gesture;

determining that the human target has lowered their hand from the position satisfying the gesture modifier filter parameter; and

determining an end of the gesture period based on a time at which the human target lowers the hand.

10. One or more processor readable storage devices according to claim 9, the method further comprising:

prior to detecting the gesture modifier, determining whether movement of the human target satisfies a first set of viable gestures;

in response to detecting the gesture modifier;

determining that the gesture modifier is associated with a second set of viable gestures, the second set of viable gestures including the first mid-air gesture and being different from the first set of viable gestures, and

only determining from the skeleton mapping whether the movement of the human target satisfies the second set of viable gestures.

11. One or more processor readable storage devices according to claim 9, wherein movement of the human target outside of the gesture period that satisfies the one or more filters will not trigger performance of the first user-interface action corresponding to the first mid-air gesture.

12. One or more processor readable storage devices according to claim 9, wherein movement of the human target outside of the gesture period that satisfies the one or more filters will trigger performance of the first user-interface action corresponding to the first mid-air gesture.

13. One or more processor readable storage devices according to claim 9, wherein:

determining whether the modifier for the first mid-air gesture is present includes determining whether the human target is holding a modifier object.

14. One or more processor readable storage devices according to claim 9, wherein:

determining whether the modifier for the first mid-air gesture is present includes determining whether the human target issues a speech modifier.

15. A method of operating a user-interface using mid-air motion of a human target, comprising:

tracking movement of a human target from a plurality of depth images using skeletal mapping of the human target in a known three-dimensional coordinate system;

determining from the skeletal mapping whether movement including a first hand of the human target satisfies one or more filters for a first mid-air gesture;

determining from the skeletal mapping whether a second hand of the human target satisfies one or more filters for a modifier of the first mid-air gesture, including determining whether the second hand of the human target satisfies a position parameter, the one or more filters for the modifier of the first mid-air gesture define a plurality of ranges for a position of the second hand;

if the movement including the first hand of the human target satisfies the one or more filters for the first mid-air gesture and the second hand of the human target does not satisfy the one or more filters for the modifier of the first mid-air gesture, performing a first user-interface action, the first user-interface action includes navigating through a list of menu items by a first amount; and

if the movement including the first hand of the human target satisfies the one or more filters for the first mid-air gesture and the second hand of the human target satisfies the one or more filters for the modifier of the first mid-air gesture, performing a second user-interface action, the second user-interface action includes navigating through the list of menu items by a second amount when the position of the second hand is in a first range and navigating through the list of menu items by a third amount when the position of the second hand is in a second range.

16. A method of operating a user-interface using mid-air motion of a human target, comprising:

tracking movement of a human target from a plurality of depth images using skeletal mapping of the human target in a known three-dimensional coordinate system;

determining from the skeletal mapping whether movement including a first hand of the human target satisfies one or more filters for a first mid-air gesture, the first mid-air gesture is a horizontal fling gesture;

determining from the skeletal mapping whether a second hand of the human target satisfies one or more filters for a modifier of the first mid-air gesture;

if the movement including the first hand of the human target satisfies the one or more filters for the first mid-air gesture and the second hand of the human target does not satisfy the one or more filters for the modifier of the first mid-air gesture, performing a first user-interface action, the first user-interface action includes horizontally scrolling a list of a menu items; and

if the movement including the first hand of the human target satisfies the one or more filters for the first mid-air gesture and the second hand of the human target satisfies the one or more filters for the modifier of the first mid-air gesture, performing a second user-interface action, the second user-interface action includes vertically scrolling a list of menu items.