Gestures And Gesture Recognition For Manipulating A User-Interface

• US 20110289455A1
• Filed: 05/18/2010
• Published: 11/24/2011
• Est. Priority Date: 05/18/2010
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mattingly, Andy, Klein, Christian, Vassigh, Ali, Poulos, Adam, Dayal, Arjun, Reville, Brendan
• Application Number:
  US12/782,377
• Publication Number:
  US 20110289455A1
• Time in Patent Office: Days
• US Class Current: 715/830
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

Symbolic gestures and associated recognition technology are provided for controlling a system user-interface, such as that provided by the operating system of a general computing system or multimedia console. The symbolic gesture movements in mid-air are performed by a user with or without the aid of an input device. A capture device is provided to generate depth images for three-dimensional representation of a capture area including a human target. The human target is tracked using skeletal mapping to capture the mid-air motion of the user. The skeletal mapping data is used to identify movements corresponding to pre-defined gestures using gesture filters that set forth parameters for determining when a target'"'"'s movement indicates a viable gesture. When a gesture is detected, one or more pre-defined user-interface control actions are performed.

Claims

1. A method of operating a user-interface using mid-air motion of a human target, comprising:

receiving a plurality of images from a capture device, the plurality of images including the human target;

tracking movement of the human target from the plurality of images using skeletal mapping of the human target;

determining from the skeletal mapping whether the movement of the human target satisfies one or more filters for a first mid-air gesture, the one or more filters specifying that the first mid-air gesture be performed by a particular hand or by both hands; and

if the movement of the human target satisfies the one or more filters, performing at least one user-interface action corresponding to the mid-air gesture.

2. A method according to claim 1, further comprising:

providing at least one gesture filter corresponding to each of a plurality of mid-air gestures, including providing the one or more filters for the first mid-air gesture;

determining a context of the user-interface;

determining a set of viable mid-air gestures corresponding to the context of the user-interface, the set including the first mid-air gesture and less than all of the plurality of mid-air gestures; and

in response to determining the context of the user-interface, only determining from the skeleton mapping whether the movement of the human target satisfies the at least one gesture filter corresponding to each of the viable mid-air gestures in the set.

3. A method according to claim 1, wherein:

the first mid-air gesture is a horizontal fling gesture;

determining whether the movement of the human target satisfies the one or more filters for the horizontal fling gesture includes;

determining whether a position of a hand of the human target satisfies a starting position parameter,

determining whether a direction of movement of the hand from the starting position satisfies a directional parameter, and

determining whether a distance traveled by the hand during the movement satisfies a distance parameter,

determining whether the movement of the hand satisfying the distance parameter occurs within a time parameter.

4. A method according to claim 3, wherein the at least one user-interface action corresponding to the horizontal fling gesture includes a horizontal scrolling action of menu items of the user-interface, the method further comprising:

horizontally scrolling the menu items of the user-interface by a first amount when the distance traveled by the hand is a first distance; and

horizontally scrolling the menu items of the user-interface by a second amount when the distance traveled by the hand is a second distance, the first amount being less than the second amount and the first distance being less than the second distance.

5. A method according to claim 3, wherein the at least one user-interface action corresponding to the horizontal fling gesture includes a horizontal scrolling action of menu items of the user-interface, the method further comprising:

determining a velocity of the hand during the movement of the hand;

horizontally scrolling the menu items of the user-interface by a first amount when the velocity is a first velocity; and

horizontally scrolling the menu items of the user-interface by a second amount when the velocity is a second velocity, the first amount being less than the second amount and the first velocity being less than the second velocity.

6. A method according to claim 3, wherein the horizontal fling gesture is a right-handed horizontal fling gesture and the hand is a right hand of the human target, the method further comprising:

filtering to remove skeletal mapping information of a left hand of the human target when determining whether the movement of the human target satisfies the one or more filters for the right-handed horizontal fling gesture.

7. A method according to claim 1, wherein the first mid-air gesture is a vertical fling gesture and the at least one user-interface action corresponding to the vertical fling gesture includes a vertical scrolling action of menu items of the user-interface, the method further comprising:

determining a velocity of the hand of the human target in performing the vertical fling gesture;

vertically scrolling the menu items of the user-interface by a first amount when the velocity of the hand is a first velocity; and

vertically scrolling the menu items of the user-interface by a second amount when the velocity of the hand is a second velocity, the first amount being less than the second amount and the first velocity being less than the second velocity.

8. A method according to claim 1, wherein:

the first mid-air gesture is a press gesture;

the at least one user-interface action corresponding to the press gesture includes a selection of a menu item of the user-interface;

determining whether the movement of the human target satisfies the one or more filters for the press gesture includes;

determining whether a position of a hand of the human target satisfies a starting position parameter,

determining whether a direction of movement of the hand from the starting position satisfies a directional parameter, the directional parameter corresponding to movement of the hand of the human target away from the human target'"'"'s body and towards the capture device,

determining an ending position of the hand of the human target,

determining whether a distance traveled by the hand during the movement satisfies a distance parameter,

determining whether the movement of the hand satisfying the distance parameter satisfies a time parameter; and

performing selection of a menu item includes selecting a first menu item corresponding to the ending position of the hand of the human target.

9. A method according to claim 1, wherein:

the first mid-air gesture is a back gesture;

the at least one user-interface action corresponding to the press gesture includes navigating backwards within the user-interface;

determining whether the movement of the human target satisfies the one or more filters for the back gesture includes;

determining whether a position of a hand of the human target satisfies a starting position parameter,

determining whether a direction of movement of the hand from the starting position satisfies a directional parameter, the directional parameter corresponding to movement of the hand of the human target away from the capture device and towards the human target'"'"'s body.

10. A method according to claim 1, wherein:

the mid-air gesture is a two-handed press gesture;

the at least one user-interface action corresponding to the two-handed press gesture includes navigating backwards within the user-interface;

determining whether the movement of the human target satisfies the one or more filters for the two-handed press gesture includes;

determining whether a position of a right hand of the human target satisfies a first starting position parameter,

determining whether a position of a left hand of the human target satisfies a second starting position parameter,

determining whether a direction of movement of the right hand from its starting position satisfies a first directional parameter, the first directional parameter corresponding to movement of the right hand away from the human target'"'"'s body and towards the capture device,

determining whether a direction of movement of the left hand from its starting position satisfies a second directional parameter, the second directional parameter corresponding to movement of the left hand away from the human target'"'"'s body and towards the capture device,

determining whether the movement of the left hand and the movement of the right hand satisfy a coordination parameter.

11. A method according to claim 1, wherein:

the mid-air gesture is a two-handed compression gesture;

the at least one user-interface action corresponding to the two-handed compression includes navigating backwards within the user-interface;

determining whether the movement of the human target satisfies the one or more filters for the two-handed press gesture includes;

determining whether a position of a right hand of the human target satisfies a first starting position parameter,

determining whether a position of a left hand of the human target satisfies a second starting position parameter,

determining whether a direction of movement of the right hand from its starting position satisfies a first directional parameter, the first directional parameter corresponding to movement of the right hand toward a left side of the human target'"'"'s body,

determining whether a direction of movement of the left hand from its starting position satisfies a second directional parameter, the second directional parameter corresponding to movement of the left hand toward a right side of the human target'"'"'s body,

determining whether the movement of the left hand and the movement of the right hand satisfy a coordination parameter.

12. A method according to claim 1, wherein the plurality of images is a plurality of depth images.

13. A system for tracking user movement to control a user-interface, comprising:

an operating system providing the user-interface;

a tracking system in communication with an image capture device to receive depth information of a capture area including a human target and to create a skeletal model mapping movement of the human target over time;

a gestures library storing a plurality of gesture filters, each gesture filter containing information for at least one gesture, wherein one or more of the plurality of gesture filters specify that a corresponding gesture be performed by a particular hand or both hands; and

a gesture recognition engine in communication with the gestures library for receiving the skeletal model and determining whether the movement of the human target satisfies one or more of the plurality of gesture filters, the gesture recognition engine providing an indication to the operating system when one or more of the plurality of gesture filters are satisfied by the movement of the human target.

14. A system according to claim 13, wherein:

the gesture recognition engine determines a context of the user-interface and in response, accesses a subset of the plurality of gesture filters corresponding to the determined context, the subset including less than all of the plurality of gesture filters, the gesture recognition engine only determining whether the movement of the human target satisfies one or more of the subset of the plurality of gesture filters.

15. A system according to claim 13, further comprising:

at least one first processor executing the operating system, gestures library and gesture recognition engine;

the image capture device;

at least one second processor receiving the depth information from the image capture device and executing the tracking system, the depth information including a plurality of depth images.

16. A system according to claim 13, wherein:

the plurality of gesture filters includes a vertical fling gesture filter, a horizontal fling gesture filter, a press gesture filter, a back gesture filter, a two-handed press gesture filter and a two-handed compression gesture filter.

17. One or more processor readable storage devices having processor readable code embodied on the one or more processor readable storage devices, the processor readable code for programming one or more processors to perform a method comprising:

providing at least one gesture filter corresponding to each of a plurality of mid-air gestures for controlling an operating system user-interface, the plurality of mid-air gestures including at least two of a horizontal fling gesture, a vertical fling gesture, a one-handed press gesture, a back gesture, a two-handed press gesture and a two-handed compression gesture;

tracking movement of a human target from a plurality of depth images using skeletal mapping of the human target in a known three-dimensional coordinate system;

determining from the skeletal mapping whether the movement of the human target satisfies the at least one gesture filter for each of the plurality of mid-air gestures; and

controlling the operating system user-interface in response to determining that the movement of the human target satisfies one or more of the gesture filters.

18. One or more processor readable storage devices according to claim 17, wherein the horizontal fling gesture is a right-handed horizontal fling gesture, the method further comprising:

filtering to remove skeletal mapping information of a left hand of the human target when determining whether the movement of the human target satisfies the at least one gesture filter for the right-handed horizontal fling gesture.

19. One or more processor readable storage devices according to claim 17, wherein the method further comprises:

generating a handle associated with an area of the operating system user-interface; and

detecting engagement by the human target with the handle;

wherein determining whether the movement of the human target satisfies the at least one gesture filter only determines whether movement of the human target while engaged with the handle satisfies the at least one gesture filter.

20. One or more processor readable storage devices according to claim 17, wherein:

determining whether the movement of the human target satisfies the vertical fling gesture includes determining a velocity of a hand of the human target when performing the vertical fling gesture;

controlling the operating system user-interface in response to determining that the movement of the human target satisfies the vertical fling gesture includes vertical scrolling a list of menu items provided by the user-interface by a first amount when the velocity of the hand of the human target is equal to a first velocity and vertically scrolling the list of menu items provided by the user-interface by a second amount when the velocity of the hand of the human target is equal to a second velocity, the first amount being less than the second amount and the first velocity being less than the second velocity.