Fully automatic dynamic articulated model calibration

• US 8,610,723 B2
• Filed: 06/29/2011
• Issued: 12/17/2013
• Est. Priority Date: 06/22/2011
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lee, Kyungsuk David, Balan, Alexandru
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Du, Haixia
• Application Number:
  US13/172,255
• Publication Number:
  US 20120327089A1
• Time in Patent Office: 902 Days
• Field of Search: None
• US Class Current: 345/473
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects

Abstract

A depth sensor obtains images of articulated portions of a user'"'"'s body such as the hand. A predefined model of the articulated body portions is provided. The model is matched to corresponding depth pixels which are obtained from the depth sensor, to provide an initial match. The initial match is then refined using distance constraints, collision constraints, angle constraints and a pixel comparison using a rasterized model. Distance constraints include constraints on distances between the articulated portions of the hand. Collision constraints can be enforced when the model meets specified conditions, such as when at least two adjacent finger segments of the model are determined to be in a specified relative position, e.g., parallel. The rasterized model includes depth pixels of the model which are compared to identify overlapping pixels. Dimension of the articulated portions of the model are individually adjusted.

Claims

1. A computer readable storage device comprising computer readable software embodied thereon for programming at least one processor to perform a method for modeling a pose of articulated body portions of an object, the method comprising:

obtaining depth pixels of the articulated body portions in one or more frames;

processing the depth pixels of the one or more frames to identify the articulated body portions of the object;

accessing a model, the model comprising articulated body portions corresponding to the articulated body portions of the object;

matching the articulated body portions of the model to the identified articulated body portions of the object to provide an initial match;

evaluating an extent to which distance constraints are violated in the initial match by the articulated body portions of the model, the distance constraints comprise constraints on distances between joints of the articulated body portions of the model; and

adjusting the model in an attempt to satisfy the distance constraints, including adjusting a length of at least one of the articulated body portions of the model based on the extent to which the distance constraints are violated, wherein;

the distance constraints constrain at least two of the articulated body portions of the model to be arranged end to end;

the evaluating comprises evaluating an extent to which the at least two of the articulated body portions of the model are arranged at least one of;

further than end to end in the initial match, or closer than end to end in the initial match; and

the adjusting a length comprises at least one of;

(i) lengthening the at least two of the articulated body portions of the model according to the extent to which the at least two of the articulated body portions of the model are arranged further than end to end in the initial match, or (ii) shortening the at least two of the articulated body portions of the model according to the extent to which the at least two of the articulated body portions of the model are arranged closer than end to end in the initial match.

2. The computer readable storage device of claim 1, wherein:

the articulated body portions comprise a palm and fingers of a hand, including finger segments; and

the at least two of the articulated body portions of the model comprise finger segments of one of the fingers.

3. The computer readable storage device of claim 1, wherein:

the articulated body portions comprise a palm and fingers of a hand;

the distance constraints constrain the fingers of the model to extend from the palm of the model at predefined joints of the palm;

the evaluating comprises evaluating an extent to which the fingers of the model do not extend from the predefined joints of the palm; and

the adjusting the model comprises moving the fingers of the model to extend from the predefined joints of the palm according to the extent to which the fingers of the model do not extend from the predefined joints of the palm.

4. The computer readable storage device of claim 1, wherein:

the articulated body portions comprise a palm and fingers of a hand; and

the evaluating and adjusting a length are performed for each of the fingers individually.

5. The computer readable storage device of claim 1, wherein the method performed further comprises:

rasterizing the model to provide depth pixels of the model;

comparing the depth pixels of the model to the depth pixels of the one or more frames to identify, from among the depth pixels of the one or more frames, non-overlapping depth pixels of the one or more frames which are not overlapping in at least one comparison plane with the depth pixels of the model; and

adjusting the model based on the comparing to cause the model to more closely match the non-overlapping depth pixels of the one or more frames, by adjusting a dimension of the at least one of the articulated body portions of the model.

6. The computer readable storage device of claim 5, wherein:

the adjusting the model based on the comparing comprises increasing a width of the at least one of the articulated body portions of the model according to a number of non-overlapping depth pixels of the one or more frames which are adjacent to and lateral of the at least one of the articulated body portions of the model.

7. The computer readable storage device of claim 1, wherein the articulated body portions comprise a palm and fingers of a hand, and the method performed further comprises:

evaluating an extent to which collision constraints are violated in the model by at least two adjacent fingers of the model; and

adjusting the model in an attempt to satisfy the collision constraints, including adjusting dimensions of at least one of the two adjacent fingers of the model based on the extent to which the collision constraints are violated by the at least one of the two adjacent fingers of the model.

8. The computer readable storage device of claim 7, wherein:

the adjusting the model in the attempt to satisfy the collision constraints comprises decreasing a width of the at least one of the two adjacent fingers of the model based on the extent to which the collision constraints are violated by the at least one of the two adjacent fingers of the model.

9. The computer readable storage device of claim 1, wherein the method performed further comprises:

rasterizing the model to provide depth pixels of the model;

comparing the depth pixels of the model to the depth pixels of the one or more frames to identify, from among the depth pixels of the model, non-overlapping depth pixels of the model which are not overlapping in at least one comparison plane with the depth pixels of the one or more frames; and

adjusting the model based on the comparing to cause the model to more closely match the one or more frames, by adjusting a dimension of at least one of the articulated body portions of the model.

10. The computer readable storage device of claim 9, wherein:

the adjusting the model based on the comparing comprises decreasing a width of the at least one of the articulated body portions of the model according to a number of non-overlapping depth pixels of the model which are adjacent to and lateral of the at least one of the articulated body portions of the model.

11. A processor-implemented method for modeling a pose of a hand of a user, comprising:

obtaining depth pixels of the hand in one or more frames;

processing the depth pixels of the one or more frames to identify articulated portions of the hand;

accessing a model of the articulated portions of the hand, the articulated portions of the hand of the model comprising a palm and fingers, including finger segments;

matching the articulated portions of the hand of the model to the identified articulated portions of the hand of the depth pixels of the one or more frames, to provide an initial match;

evaluating an extent to which distance constraints are violated in the initial match by at least one of the fingers, the distance constraints comprise constraints on distances between finger segments of the at least one of the fingers;

rasterizing the model to provide depth pixels of the model;

comparing the depth pixels of the at least one of the fingers to the depth pixels of the one or more frames to identify, from among the depth pixels of the one or more frames, non-overlapping depth pixels of the one or more frames which are not overlapping in at least one comparison plane with the depth pixels of the at least one of the fingers of the model; and

adjusting the model;

(a) in an attempt to satisfy the distance constraints, including adjusting a length of at least one finger segments of at least one of the fingers of the model based on the extent to which the distance constraints are violated by the at least one of the fingers, and (b) based on the comparing, to cause the model to more closely match the non-overlapping depth pixels of the one or more frames, by increasing a width of the at least one of the finger segments of the at least one of the fingers of the model.

12. The processor-implemented method of claim 11, further comprising:

adjusting the model based on the comparing to cause the model to more closely match the non-overlapping depth pixels of the one or more frames, by increasing a length of at least one of the finger segments of the at least one of the fingers of the model.

13. The processor-implemented method of claim 11, further comprising:

comparing the depth pixels of at least one of the articulated portions of the hand, other than the at least one of the fingers, to the depth pixels of the one or more frames to identify, from among the depth pixels of the one or more frames, non-overlapping depth pixels of the at least one of the articulated portions of the hand of the model which are not overlapping in the at least one comparison plane with the depth pixels of the one or more frames; and

adjusting the model based on the comparing of the depth pixels of the at least one of the articulated portions of the hand, other than the at least one of the fingers, to the depth pixels of the one or more frames, to cause the model to more closely match the depth pixels of the one or more frames, by decreasing a width of the at least one of the articulated portions of the hand of the model.

14. The processor-implemented method of claim 11, further comprising:

the increasing the width of the at least one of the finger segments of the at least one of the fingers of the model is based on a number of the non-overlapping depth pixels of the one or more frames which are adjacent to and lateral of the at least one of the finger segments of the at least one of the fingers of the model.

15. A processor-implemented method for modeling a pose of a hand of a user, comprising:

obtaining depth pixels of the hand in one or more frames;

processing the depth pixels of the one or more frames to identify articulated portions of the hand;

accessing a model of the articulated portions of the hand, the articulated portions of the hand of the model comprising a palm and fingers, including finger segments;

matching the articulated portions of the hand of the model to the identified articulated portions of the hand of the depth pixels of the one or more frames, to provide an initial match;

evaluate an extent to which distance constraints are violated in the initial match by at least one of the fingers, the distance constraints comprise constraints on distances between finger segments of the at least one of the fingers;

deciding to enforce collision constraints;

in response to the deciding to enforce collision constraints, evaluating an extent to which the collisions constraints are violated in the initial match by at least two adjacent finger segments of the model; and

adjusting the model;

(a) in an attempt to satisfy the collision constraints, based on the extent to which the collision constraints are violated by the at least two adjacent finger segments of the model, by decreasing widths of the at least two adjacent finger segments of the model and (b) in an attempt to satisfy the distance constraints, based on the extent to which the distance constraints are violated by the at least one of the fingers.

16. The processor-implemented method of claim 15, wherein:

the deciding to enforce collision constraints comprises determining that the at least two adjacent finger segments of the model are in a specified relative position.

17. The processor-implemented method of claim 16, wherein:

the at least two adjacent finger segments of the model are in the specified relative position when respective longitudinal axes of the at least two adjacent finger segments of the model are substantially parallel, within a specified angular threshold.

18. The processor-implemented method of claim 17, wherein:

the extent to which the collision constraints are violated by the at least two adjacent finger segments of the model is based on a distance between the respective longitudinal axes.