Hand-location post-process refinement in a tracking system

• US 8,452,051 B1
• Filed: 12/18/2012
• Issued: 05/28/2013
• Est. Priority Date: 04/26/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lee, Johnny Chung
• Primary Examiner(s): Werner, Brian P
• Application Number:
  US13/718,494
• Publication Number:
  US 20130120244A1
• Time in Patent Office: 161 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A tracking system having a depth camera tracks a user'"'"'s body in a physical space and derives a model of the body, including an initial estimate of a hand position. Temporal smoothing is performed in which some latency is imposed when the initial estimate moves by less than a threshold level from frame to frame, while little or no latency is imposed when the movement is more than the threshold. The smoothed estimate is used to define a local volume for searching for a hand extremity to define a new hand position. Another process generates stabilized upper body points that can be used as reliable reference positions, such as by detecting and accounting for occlusions. The upper body points and a prior estimated hand position are used to define an arm vector. A search is made along the vector to detect a hand extremity to define a new hand position.

Claims

1. Tangible computer readable storage device having computer readable software embodied thereon for programming a processor to perform a method for tracking user movement in a motion capture system, the method comprising:

tracking a body in a field of view of the motion capture system, including obtaining a 3-D depth image and determining a 3-D skeletal model of the body;

for one point in time, identifying a location of a hand of the 3-D skeletal model in the field of view; and

for a next point in time;

identifying a reference point of the 3-D skeletal model;

defining at least one vector from the reference point in the next point in time to the location of the hand in the one point in time;

traversing the at least one vector to look for a most probable location of the hand in the next point in time, including scoring candidate locations which are part of the 3-D skeletal model based on their distance along the at least one vector and their distance perpendicularly from the at least one vector;

based on the most probable location of the hand, defining a volume in the field of view;

searching the 3-D depth image in the volume to determine a location of the hand in the field of view; and

providing to an application a control input which represents the hand in the field of view based on the determined location of the hand, or a value derived from the determined location of the hand.

2. The tangible computer readable storage device of claim 1, wherein:

the scoring candidate locations includes providing a score indicating a more probable location in proportion to the distance along the at least one vector being greater, and in proportion to the distance perpendicularly from the at least one vector being lower.

3. The tangible computer readable storage device of claim 1, wherein:

the searching includes identifying locations of edges of the hand in the volume and determining an average of the locations of the edges.

4. The tangible computer readable storage device of claim 1, wherein:

the volume in the field of view is centered at an estimate of the location of the hand in the field of view.

5. The tangible computer readable storage device of claim 1, wherein:

the reference point of the 3-D skeletal model is a shoulder.

6. The tangible computer readable storage device of claim 1, wherein:

the reference point of the 3-D skeletal model undergoes temporal smoothing before the at least one vector is defined.

7. The tangible computer readable storage device of claim 1, wherein:

the reference point of the 3-D skeletal model is identified based on a set of points which are identified from the 3-D skeletal model, the set of points identify at least both shoulders of the 3-D skeletal model.

8. The tangible computer readable storage device of claim 1, wherein:

the reference point of the 3-D skeletal model is identified based on a set of points which are identified from the 3-D skeletal model; and

the method further comprises determining when at least one point of the set of points is potentially occluded, and, responsive to determining that the at least one point of the set of points is potentially occluded, determining a location of the at least one point based on another point of the 3-D skeletal model.

9. The tangible computer readable storage device of claim 8, wherein:

the determining when the at least one point of the set of points is potentially occluded comprises measuring a proximity of an arm joint to an upper body joint in the 3-D skeletal model in a projected camera image.

10. A processor-implemented method for tracking user movement in a motion capture system, comprising the processor-implemented steps of:

tracking a body in a field of view of the motion capture system, including obtaining a 3-D depth image and determining a 3-D skeletal model of the body;

for one point in time, identifying a location of a hand of the 3-D skeletal model in the field of view; and

for a next point in time;

identifying a reference point of the 3-D skeletal model;

defining at least one vector from the reference point in the next point in time to the location of the hand in the one point in time;

traversing the at least one vector to look for a most probable location of the hand in the next point in time, including scoring candidate locations which are part of the 3-D skeletal model based on their distance along the at least one vector and their distance perpendicularly from the at least one vector;

based on the most probable location of the hand, defining a volume in the field of view;

searching the 3-D depth image in the volume to determine a location of the hand in the field of view; and

providing to an application a control input which represents the hand in the field of view based on the determined location of the hand, or a value derived from the determined location of the hand.

11. The processor-implemented method of claim 10, wherein:

the scoring candidate locations includes providing a score indicating a more probable location in proportion to the distance along the at least one vector being greater, and in proportion to the distance perpendicularly from the at least one vector being lower.

12. The processor-implemented method of claim 10, wherein:

the reference point of the 3-D skeletal model undergoes temporal smoothing before the at least one vector is defined.

13. The processor-implemented method of claim 10, wherein:

the reference point of the 3-D skeletal model is identified based on a set of points which are identified from the 3-D skeletal model, the set of points identify at least both shoulders of the 3-D skeletal model.

14. The processor-implemented method of claim 10, wherein:

the reference point of the 3-D skeletal model is identified based on a set of points which are identified from the 3-D skeletal model; and

the method further comprises determining when at least one point of the set of points is potentially occluded, and, responsive to determining that the at least one point of the set of points is potentially occluded, determining a location of the at least one point based on another point of the 3-D skeletal model.

15. The processor-implemented method of claim 14, wherein:

the determining when the at least one point of the set of points is potentially occluded comprises measuring a proximity of an arm joint to an upper body joint in the 3-D skeletal model in a projected camera image.

16. A motion capture system, comprising:

a depth camera system having a field of view; and

a processor in communication with the depth camera system, the processor receives images of the field of view and executes instructions to track user movement in the field of view, the instructions cause the processor to track a body in a field of view, including obtaining a 3-D depth image and determining a 3-D skeletal model of the body, for one point in time, identify a location of a hand of the 3-D skeletal model in the field of view, and for a next point in time;

identify a reference point of the 3-D skeletal model, define at least one vector from the reference point in the next point in time to the location of the hand in the one point in time, traverse the at least one vector to look for a most probable location of the hand in the next point in time, including providing a score for candidate locations which are part of the 3-D skeletal model based on their distance along the at least one vector and their distance perpendicularly from the at least one vector, based on the most probable location of the hand, define a volume in the field of view, search the 3-D depth image in the volume to determine a location of the hand in the field of view, and provide to an application a control input which represents the hand in the field of view based on the determined location of the hand, or a value derived from the determined location of the hand.

17. The motion capture system of claim 16, wherein:

the providing a score for the candidate locations includes providing a score indicating a more probable location in proportion to the distance along the at least one vector being greater, and in proportion to the distance perpendicularly from the at least one vector being lower.

18. The motion capture system of claim 16, wherein:

the search includes identifying locations of edges of the hand in the volume and determining an average of the locations of the edges.

19. The motion capture system of claim 16, wherein:

the volume in the field of view is centered at an estimate of the location of the hand in the field of view.

20. The motion capture system of claim 16, wherein:

the reference point of the 3-D skeletal model is a shoulder.