PHYSICAL INTERACTION ZONE FOR GESTURE-BASED USER INTERFACES

• US 20110193939A1
• Filed: 02/09/2010
• Published: 08/11/2011
• Est. Priority Date: 02/09/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Pennington, Ernest L., Klein, Christian, Vassigh, Ali M.
• Granted Patent: US 8,659,658 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/8011 Ball
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

In a motion capture system having a depth camera, a physical interaction zone of a user is defined based on a size of the user and other factors. The zone is a volume in which the user performs hand gestures to provide inputs to an application. The shape and location of the zone can be customized for the user. The zone is anchored to the user so that the gestures can be performed from any location in the field of view. Also, the zone is kept between the user and the depth camera even as the user rotates his or her body so that the user is not facing the camera. A display provides feedback based on a mapping from a coordinate system of the zone to a coordinate system of the display. The user can move a cursor on the display or control an avatar.

Claims

1. A processor-implemented method for tracking user movement in a motion capture system, comprising the processor-implemented steps of:

tracking a body in a field of view of the motion capture system, including determining a model of the body;

determining reference points of the model;

determining a size and position of a zone based on the reference points, the zone is a 3-D volume in the field of view and has a coordinate system which is defined relative to at least one of the reference points;

tracking movement of a hand of the body in the zone relative to the coordinate system of the zone; and

based on the tracking, translating the movement of the hand in the zone to a corresponding action on a display.

2. The processor-implemented method of claim 1, wherein the zone is offset from a center of the body, and is curved according to a natural biomechanical range of movement of the body.

3. The processor-implemented method of claim 1, wherein the tracking the movement of the hand includes determining that the hand has entered the zone, the method further comprising:

initiating a user interface on the display in response to the determining that the hand has entered the zone.

4. The processor-implemented method of claim 3, wherein the tracking the movement of the hand includes determining that the hand has exited the zone, the method further comprising:

terminating the user interface on the display in response to the determining that the hand has exited the zone.

5. The processor-implemented method of claim 1, wherein:

determining a size and position of another zone based on the reference points, the another zone is a 3-D volume in the field of view and has a coordinate system which is defined relative to at least one of the reference points;

tracking movement of another hand of the body in the another zone relative to the coordinate system of the another zone; and

based on the tracking movement of the another hand, translating the movement of the another hand in the another zone to a corresponding action on the display.

6. The processor-implemented method of claim 1, wherein:

the corresponding action on the display is a movement of a cursor; and

the tracking the movement of the hand includes determining that the hand moves from a first position to a second position within a specified time period, in response to which the cursor moves from a non-edge position on the display to an edge position on the display, wherein the second position is at a margin of the zone.

7. The processor-implemented method of claim 1, further comprising:

determining a size of the body based on the reference points, where the size of the zone is based on the size of the body.

8. The processor-implemented method of claim 1, wherein the zone includes at least first and second subsets, and the method further comprises:

detecting that the hand is in the first or second subset, wherein the corresponding action on the display is responsive to the detecting that the hand is in the first or second subset.

9. The processor-implemented method of claim 1, wherein:

the zone is anchored to the body as the body walks in the field of view.

10. The processor-implemented method of claim 1, wherein:

the reference points identify shoulders of the body, and the position of the zone is defined relative to whichever shoulder is closest to a depth camera of the motion capture system, so that the zone remains between the body and the depth camera.

11. The processor-implemented method of claim 1, wherein:

the reference points identify a shoulder line and head of the body.

12. The processor-implemented method of claim 1, wherein:

the corresponding action on the display is a scrolling of a menu.

13. The processor-implemented method of claim 1, wherein:

the corresponding action on the display is 3-D movement of an avatar.

14. The processor-implemented method of claim 1, further comprising:

storing data to a profile of the body based on the size of the zone.

15. A motion capture system, comprising:

a depth camera system having a field of view;

a display; and

one or more processors in communication with the depth camera system and the display, the processor executes instructions to track user movement and to provide a signal to the display to display images;

wherein the depth camera system and the one or more processors;

to track the body in the field of view, determine a model of the body;

determine reference points of the model;

determine a size and position of a zone based on the reference points, the zone is a 3-D volume in the field of view and has a coordinate system which is defined relative to at least one of the reference points and is anchored to the model of the body;

track movement of a hand of the body in the zone relative to the coordinate system of the zone;

translate the movement of the hand in the zone from the coordinate system of the zone to a coordinate system of the display; and

update the display based on the translated movement of the hand.

16. The motion capture system of claim 15, wherein:

the reference points identify a portion of the hand of the body; and

the movement of the hand of the body is translated to the corresponding action on the display based on movement of the portion of the hand.

17. The motion capture system of claim 15, wherein:

the display is rectangular;

the zone is curved according to a natural biomechanical range of movement of the body;

and the display is updated based on a mapping between points in the display and corresponding points in the zone.

18. Tangible computer readable storage having computer readable software embodied thereon for programming at least one processor to perform a method in a motion capture system, the method comprising:

tracking a body in a field of view of the motion capture system, including determining a model of the body;

determining reference points of the model;

determining a size and position of a first zone based on the reference points, the first zone is a 3-D volume in the field of view and has a coordinate system which is defined relative to at least one of the reference points;

determining a size and position of a second zone based on the reference points, the second zone is a 3-D volume in the field of view and has a coordinate system which is defined relative to at least one of the reference points, the second zone is smaller than the first zone and overlaps, at least in part, with the first zone;

tracking movement of a hand of the body in the first zone relative to the coordinate system of the first zone;

tracking movement of the hand of the body in the second zone relative to the coordinate system of the second zone; and

selecting one of the tracking movement of the body in the first zone and the tracking movement of the body in the second zone, for use in updating the display.

19. The tangible computer readable storage of claim 18, wherein:

the tracking movement of the hand of the body in the second zone is selected when the hand of the body is confined to the second zone in a period of time; and

the tracking movement of the hand of the body in the first zone is selected when the hand of the body is not confined to the second zone in the period of time.

20. The tangible computer readable storage of claim 18, wherein the method performed further comprises:

switching between the selecting the tracking movement of the body in the first zone for use in the updating the display, and the selecting the tracking movement of the body in the second zone for use in the updating the display.