HAND-LOCATION POST-PROCESS REFINEMENT IN A TRACKING SYSTEM

• US 20110262002A1
• Filed: 04/26/2010
• Published: 10/27/2011
• Est. Priority Date: 04/26/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lee, Johnny Chung
• Granted Patent: US 8,351,651 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A tracking system having a depth camera tracks a user'"'"'s body in a physical space and derives a model of the body, including an initial estimate of a hand position. Temporal smoothing is performed when the initial estimate moves by less than a threshold level from frame to frame, while little or no smoothing is performed when the movement is more than the threshold. The smoothed estimate is used to define a local volume for searching for a hand extremity to define a new hand position. Another process generates stabilized upper body points that can be used as reliable reference positions, such as by detecting and accounting for occlusions. The upper body points and a prior estimated hand position are used to define an arm vector. A search is made along the vector to detect a hand extremity to define a new hand position.

Claims

1. A processor-implemented method for tracking user movement in a motion capture system, comprising the processor-implemented steps of:

tracking a user'"'"'s hand in a field of view of the motion capture system over time, including obtaining a 3-D depth image of the hand at different points in time; and

for a point in time;

obtaining an initial estimate of a location of the hand in the field of view based on the tracking;

determining a difference of the initial estimate relative to a corresponding estimate of a prior point in time;

determining if the difference is less than a threshold;

if the difference is less than the threshold, providing a current estimate of the location by changing the initial estimate by an amount which is less than the difference; and

if the difference is not less than the threshold, providing a current estimate of the location substantially as the initial estimate;

based on the current estimate, defining a volume in the field of view;

searching the 3-D depth image in the volume to determine a new estimate of a location of the hand in the field of view; and

providing a control input to an application which represents the hand in the field of view based, at least in part, on the new estimate of the location, or a value derived from the new estimate of the location.

2. The processor-implemented method of claim 1, wherein:

the amount is set so that, when the difference is less than the threshold, the current estimate trails the initial estimate more when the difference is smaller, and less when the difference is greater.

3. The processor-implemented method of claim 1, wherein:

the amount is set so that, when the difference is less than the threshold, the current estimate approaches the initial estimate as the difference approaches the threshold.

4. The processor-implemented method of claim 1, wherein

the amount is a nonlinear function of the difference.

5. The processor-implemented method of claim 1, wherein:

the volume in the field of view is centered at the current estimate.

6. The processor-implemented method of claim 1, further comprising, for the point in time:

determining a difference of the new estimate relative to a corresponding estimate of a prior point in time;

determining if the difference of the new estimate is less than the threshold;

if the difference of the new estimate is less than the threshold, providing a new current estimate of the location by changing the new estimate by an amount which is less than the difference of the new estimate; and

if the difference of the new estimate is not less than the threshold, providing a new current estimate of the location substantially as the new estimate, the control input is provided to the application based at least in part on the new current estimate, or a value derived from the new current estimate.

7. The processor-implemented method of claim 1, wherein:

the searching includes identifying locations of the hand in the volume and determining an average of the locations of the hand.

8. A processor-implemented method for tracking user movement in a motion capture system, comprising the processor-implemented steps of:

tracking a body in a field of view of the motion capture system, including obtaining a 3-D depth image and determining a 3-D skeletal model of the body;

for one point in time, identifying a location of a hand of the 3-D skeletal model in the field of view; and

for a next point in time;

identifying a reference point of the 3-D skeletal model;

defining at least one vector from the reference point in the next point in time to the location of the hand in the one point in time;

traversing the at least one vector to look for a most probable location of the hand in the next point in time, including scoring candidate locations which are part of the 3-D skeletal model based on their distance along the at least one vector and their distance perpendicularly from the at least one vector;

based on the most probable location of the hand, defining a volume in the field of view;

searching the 3-D depth image in the volume to determine a location of the hand in the field of view; and

providing a control input to an application which represents the hand in the field of view based at least in part on the determined location of the hand, or a value derived from the determined location of the hand.

9. The processor-implemented method of claim 8, wherein:

the scoring candidate locations includes providing a score indicating a more probable location in proportion to the distance along the at least one vector being greater, and in proportion to the distance perpendicularly from the at least one vector being lower.

10. The processor-implemented method of claim 8, wherein:

the searching includes identifying locations of the hand in the volume and determining an average of the locations of the hand.

11. The processor-implemented method of claim 8, wherein:

the volume in the field of view is centered at the estimate of the location of the hand in the field of view.

12. The processor-implemented method of claim 8, wherein:

the reference point of the 3-D skeletal model is a shoulder.

13. The processor-implemented method of claim 8, wherein:

the reference point of the 3-D skeletal model undergoes temporal smoothing before the at least one vector is defined.

14. The processor-implemented method of claim 8, wherein:

the reference point of the 3-D skeletal model is identified based on a set of points which are identified from the 3-D skeletal model, the set of points identify at least both shoulders of the 3-D skeletal model.

15. The processor-implemented method of claim 8, wherein:

the reference point of the 3-D skeletal model is identified based on a set of points which are identified from the 3-D skeletal model; and

the method further comprises determining when at least one point of the set of points is potentially occluded, and, responsive to determining that the at least one point of the set of points is potentially occluded, determining a location of the at least one point based on at least one other point of the 3-D skeletal model.

16. The processor-implemented method of claim 15, wherein:

the determining when the at least one point of the set of points is potentially occluded comprises measuring a proximity of at least one arm joint to at least one upper body joint in the 3-D skeletal model in a projected camera image.

17. Tangible computer readable storage having computer readable software embodied thereon for programming at least one processor to perform a method in a motion capture system, the method comprising:

tracking a user'"'"'s hand in a field of view of the motion capture system over time, including obtaining a 3-D depth image of the hand at different points in time; and

for a point in time;

obtaining an initial estimate of a location of the hand in the field of view based on the tracking;

determining a difference of the initial estimate relative to a corresponding estimate of a prior point in time;

determining if the difference is less than a threshold;

if the difference is less than the threshold, providing a current estimate of the location which trails the initial estimate;

if the difference is not less than the threshold, providing a current estimate of the location by one of;

(a) setting the current estimate as substantially the initial estimate, and (b) providing the current estimate so that it trails the initial estimate less than when the difference is less than the threshold; and

providing a control input to an application which represents the hand in the field of view based, at least in part, on the current estimate of the location, or a value derived from the current estimate of the location.

18. The tangible computer readable storage of claim 17, wherein:

when the difference is less than the threshold, the current estimate trails the initial estimate more when the difference is smaller, and less when the difference is greater.

19. The tangible computer readable storage of claim 17, wherein:

when the difference is less than the threshold, the current estimate approaches the initial estimate as the difference approaches the threshold.

20. The tangible computer readable storage of claim 17, wherein the method performed further comprises:

based on the current estimate, defining a volume in the field of view; and

searching the 3-D depth image in the volume to determine a new estimate of a location of the hand in the field of view, the searching includes identifying locations of the hand in the volume.