Physical interaction zone for gesture-based user interfaces

• US 8,659,658 B2
• Filed: 02/09/2010
• Issued: 02/25/2014
• Est. Priority Date: 02/09/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Vassigh, Ali M., Klein, Christian, Pennington, Ernest L.
• Primary Examiner(s): Avellino, Joseph
• Assistant Examiner(s): Afolabi, Mark O
• Application Number:
  US12/703,143
• Publication Number:
  US 20110193939A1
• Time in Patent Office: 1,477 Days
• Field of Search: 348120-170, 345473-485, 706250-169
• US Class Current: 348/143
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/8011 Ball
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

In a motion capture system having a depth camera, a physical interaction zone of a user is defined based on a size of the user and other factors. The zone is a volume in which the user performs hand gestures to provide inputs to an application. The shape and location of the zone can be customized for the user. The zone is anchored to the user so that the gestures can be performed from any location in the field of view. Also, the zone is kept between the user and the depth camera even as the user rotates his or her body so that the user is not facing the camera. A display provides feedback based on a mapping from a coordinate system of the zone to a coordinate system of the display. The user can move a cursor on the display or control an avatar.

Claims

1. A processor-implemented method for tracking user movement in a motion capture system, comprising the processor-implemented steps of:

tracking a body of a user in a field of view of the motion capture system, the tracking comprises determining a model of the body;

determining reference points of the model, the reference points identify shoulders of the body and include a reference point which represents one of the shoulders;

determining a size of the body based on the reference points;

determining a natural biomechanical range of movement of a hand of the body based on the reference points;

determining a size and position of a zone encompassing the natural biomechanical range of movement of the hand based on the reference points, the zone is a 3-D volume in the field of view and has a coordinate system which is defined relative to one of the reference points, the size of the zone is based on the size of the body, and the determining the size and position of the zone comprises determining a distance from the reference point which represents the one of the shoulders to a rear surface of the zone, wherein a gap exists between the reference point which represents the one of the shoulders and the rear surface of the zone, and determining a distance between the rear surface of the zone and a front surface of the zone;

tracking movement of the hand in the zone relative to the coordinate system of the zone; and

based on the tracking of the movement of the hand, translating the movement of the hand in the zone to a corresponding movement of a cursor on a display.

2. The processor-implemented method of claim 1, wherein

the zone is offset from a center of the body, and is curved according to the natural biomechanical range of movement of the hand.

3. The processor-implemented method of claim 1, wherein the tracking the movement of the hand comprises determining that the hand has crossed between subsets zones of the zone, the method further comprising:

providing an input to an application which provides the display indicating that the hand has crossed between subsets zones, in response to the determining that the hand has crossed between the subsets zones.

4. The processor-implemented method of claim 1, wherein:

determining a size and position of another zone based on the reference points, the another zone is a 3-D volume in the field of view and has a coordinate system which is defined relative to one of the reference points;

tracking movement of another hand of the body in the another zone relative to the coordinate system of the another zone; and

based on the tracking movement of the another hand, translating the movement of the another hand in the another zone to a corresponding action on the display.

5. The processor-implemented method of claim 1, wherein:

the zone in which the movement of the hand is tracked comprises a central subzone which is smaller than a reach envelop of the body, and a lateral subzone which is lateral of the central subzone; and

the tracking the movement of the hand comprises determining that the hand moves from a position in the central subzone, past a boundary between the central subzone and the lateral subzone, to a position in the lateral subzone —

within a specified time period, in response to which the cursor moves from a non-edge position on the display to an edge position on the display.

6. The processor-implemented method of claim 1, wherein the zone comprises first and second subsets, and the method further comprises:

detecting that the hand is in the first or second subset, wherein the corresponding action on the display is responsive to the detecting that the hand is in the first or second subset.

7. The processor-implemented method of claim 1, wherein:

the zone is anchored to the body as the body walks in the field of view.

8. The processor-implemented method of claim 1, wherein:

the position of the zone is defined relative to whichever shoulder is closest to a depth camera of the motion capture system, so that the zone remains between the body and the depth camera.

9. The processor-implemented method of claim 1, wherein:

the determining the size and position of the zone comprises setting a width of the zone based on a distance from the reference point which represents the one of the shoulders to a side surface of the zone and a distance from the reference point which represents the one of the shoulders to another side surface of the zone, and setting a height of the zone based on a distance from the reference point which represents the one of the shoulders to a top surface of the zone and a distance from the reference point which represents the one of the shoulders to a bottom surface of the zone.

14. The processor-implemented method of claim 1, wherein:

the translating of the movement of the hand in the zone to the corresponding movement of the cursor on the display is based on a nonlinear mapping between points in the zone and corresponding points in the display, where an upper left point, an upper right point, a lower left point and a lower right point of the zone are mapped to an upper left point, an upper right point, a lower left point and a lower right point, respectively, of the display.

15. The processor-implemented method of claim 1, wherein:

the determining the natural biomechanical range of movement of the hand based on the reference points comprises determining a natural biomechanical range of movement of an arm of the body based on the reference points.

16. The processor-implemented method of claim 1, wherein:

the reference points identify another point which represents another of the shoulders;

a length of a shoulder line of the body is determined as a distance between the point which represents one of the shoulders and the point which represents the another of the shoulders; and

the size of the zone is a function of the length of the shoulder line.

17. The processor-implemented method of claim 1, wherein:

the reference points include a point which represents an elbow and a point which represents a hand of the body;

an arm length is determined based on a sum of a distance between the point which represents the one of the shoulders and the point which represents the elbow and a distance between the point which represents the elbow and the point which represents the hand; and

a width of the zone is a function of the arm length.

18. The processor-implemented method of claim 1, wherein:

the reference points identify a torso of the body; and

a height of the zone is a function of a height of the torso.

10. A motion capture system, comprising:

a depth camera system having a field of view;

a display; and

one or more processors in communication with the depth camera system and the display, the processor executes instructions to track user movement and to provide a signal to the display to display images;

wherein the depth camera system and the one or more processors;

to track a body of a user in the field of view, determine a model of the body;

determine reference points of the model, the reference points comprise a reference point which identifies a shoulder of the body, a reference point which identifies an elbow of the body, a reference point which identifies a hand of the body;

determine a length of an arm of the body based on a sum of a distance between the reference point which identifies the shoulder of the body and the reference point which identifies the elbow of the body and a distance between the reference point which identifies the elbow of the body and the reference point which identifies the hand;

determine a position of a zone encompassing a comfortable reach of the hand based on the reference point which identifies the shoulder of the body, the zone is a 3-D volume in the field of view and has a coordinate system which is defined relative to one of the reference points and is anchored to the model of the body;

determine a width of the zone based on a specified percentage of the length of the arm;

track movement of the hand in the zone relative to the coordinate system of the zone;

determine a portion of the zone which the hand moves across in the movement;

translate the movement of the hand in the zone from the coordinate system of the zone to a coordinate system of the display; and

update the display based on the translated movement of the hand.

11. The motion capture system of claim 10, wherein:

the display is rectangular;

the zone is curved according to a natural biomechanical range of movement of the hand; and

the display is updated based on a mapping between points in the display and corresponding points in the zone, the mapping maps a curved motion of the hand in the zone to a horizontal motion of a cursor on the display.

19. The motion capture system of claim 10, wherein:

the reference points comprise a reference point which identifies a chin of the body and a reference point which identifies a waist of the body; and

the one or more processors;

determine a distance between the reference point which identifies the chin of the body and the reference point which identifies the waist of the body; and

determine a height of the zone based on the distance between the reference point which identifies the chin of the body and the reference point which identifies the waist of the body.

12. Tangible computer readable storage device having computer readable software embodied thereon for programming a processor to perform a method in a motion capture system, the method comprising:

tracking a body in a field of view of the motion capture system, the tracking comprises determining a model of the body;

determining reference points of the model;

determining a size and position of a first zone based on the reference points, the first zone is a 3-D volume in the field of view and has a coordinate system which is defined relative to one of the reference points, and the size of the first zone is based on a range of movement of a hand of the body when the hand pivots from an elbow;

determining a size and position of a second zone based on the reference points, the second zone is a 3-D volume in the field of view and has a coordinate system which is defined relative to one of the reference points, the second zone is smaller than the first zone and overlaps, at least in part, with the first zone, and the size of the second zone is based on a range of movement of the hand when the hand pivots from a wrist;

tracking movement of the hand in the first zone relative to the coordinate system of the first zone and translatin˜

the movement of the hand in the first zone to a corresponding movement of a cursor on a display based on a mapping between the first zone and the display;

based on the tracking of the movement of the hand in the first zone, storing a record of a position of the hand for a period of time;

based on the record, determining that hand of the body is confined to the second zone over the period of time; and

based on the determining that hand of the body is confined to the second zone over the period of time, switching from tracking movement of the hand in the first zone relative to the coordinate system of the first zone and the translating the movement of the hand in the first zone to the corresponding movement of the cursor on the display based on the mapping between the first zone and the display, to tracking movement of the hand in the second zone relative to the coordinate system of the second zone and translating the movement of the hand in the second zone to a corresponding movement of the cursor on the display based on a mapping between the second zone and the display.

13. The tangible computer readable storage device of claim 12, wherein:

the tracking movement of the hand in the second zone relative to the coordinate system of the second zone and translating the movement of the hand in the second zone based on the mapping between the second zone and the display, allows the body to more easily move the cursor on the display compared to the tracking the movement of the hand in the first zone relative to the coordinate system of the first zone and the translating the movement of the hand in the first zone based on the mapping between the first zone and the display.

20. The tangible computer readable storage device of claim 12, wherein:

the translating of the movement of the hand in the first zone relative to the coordinate system of the first zone to the corresponding movement of the cursor on the display is based on a nonlinear mapping between points in the first zone and corresponding points in the display.