Probabilistic And Constraint Based Articulated Model Fitting

• US 20130329011A1
• Filed: 11/28/2012
• Published: 12/12/2013
• Est. Priority Date: 06/29/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lee, Kyungsuk David, Balan, Alexandru
• Granted Patent: US 9,344,707 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

A depth sensor obtains images of articulated portions of a user'"'"'s body such as the hand. A predefined model of the articulated body portions is provided. Representative attract points of the model are matched to centroids of the depth sensor data, and a rigid transform of the model is performed, in an initial, relatively coarse matching process. This matching process is then refined in a non-rigid transform of the model, using attract point-to-centroid matching. In a further refinement, an iterative process rasterizes the model to provide depth pixels of the model, and compares the depth pixels of the model to the depth pixels of the depth sensor. The refinement is guided by whether the depth pixels of the model are overlapping or non-overlapping with the depth pixels of the depth sensor. Collision, distance and angle constraints are also imposed on the model.

Claims

1. A computer readable storage device having computer readable software embodied thereon for programming at least one processor to perform a method for modeling a pose of articulated body portions of an object, the method comprising:

obtaining depth pixels of the articulated body portions of the object in one or more frames;

processing the depth pixels of the one or more frames to identify centroids of the articulated body portions of the object;

accessing a model, the model includes articulated body portions which correspond to the articulated body portions of the object, and which each have at least one representative attract point;

matching the representative attract points to the centroids and performing a rigid transform of the model, based on a comparison between the representative attract points and the centroids;

refining the match by performing a non-rigid transform of the model based on a comparison between the representative attract points and the centroids, resulting in changes to relative orientations and locations of the articulated body portions of the model; and

further refining the match by rasterizing the model to provide depth pixels of the model, and by performing a non-rigid transform of the model based on a comparison between the depth pixels of the one or more frames and the depth pixels of the model.

2. The computer readable storage device of claim 1, wherein:

the performing a rigid transform comprises performing translating, rotating and scaling of the model.

3. The computer readable storage device of claim 1, wherein:

the rigid transform is computed in a closed form.

4. The computer readable storage device of claim 1, wherein:

in each of the multiple iterations;

the comparison identifies, from among the depth pixels of the one or more frames, non-overlapping depth pixels of the one or more frames which are not overlapping in at least one comparison plane with the depth pixels of the model;

each of the articulated body portions of the model has at least one joint;

at least one of the articulated body portions of the model is adjacent to the non-overlapping depth pixels of the one or more frames;

each of the multiple iterations provides adjustment vectors for the joints based on the comparison to move at least one of the joints of the at least one of the articulated body portions of the model to more closely match the non-overlapping depth pixels of the one or more frames; and

the multiple iterations are performed until a stop criterion is met, the stop criterion is based on at least one of magnitudes of the adjustment vectors, number of iterations, and computation time.

5. The computer readable storage device of claim 4, wherein:

the adjustment vectors are based on collision constraints for collisions between the articulated body portions of the model, distance constraints for distances between joints of adjacent portions of the articulated body portions of the model, and angle constraints for angles between the adjacent portions of the articulated body portions of the model.

6. The computer readable storage device of claim 4, wherein:

in each of the multiple iterations;

the comparison identifies, from among the depth pixels of the one or more frames, overlapping depth pixels of the one or more frames which are overlapping in the at least one comparison plane with depth pixels of the at least one of the articulated body portions of the object; and

the adjustment vectors move at least one of the joints of the at least one of the articulated body portions of the model to more closely match the overlapping depth pixels of the one or more frames.

7. The computer readable storage device of claim 6, wherein:

in each of the multiple iterations;

the comparison identifies, from among the depth pixels of the model, non-overlapping depth pixels of the model which are not overlapping in the at least one comparison plane with the depth pixels of the one or more frames; and

the adjustment vectors move at least one of the joints of the at least one of the articulated body portions of the model in a direction which tends to equalize a number of the non-overlapping depth pixels of the model around a perimeter of the at least one of the articulated body portions of the model.

8. The computer readable storage device of claim 6, wherein the method performed further comprises:

for the non-overlapping depth pixels of the one or more frames, for the overlapping depth pixels of the one or more frames, and for the at least one of the articulated body portions of the model;

determining a probability that the depth pixel is part of the at least one of the articulated body portions of the model, the adjustment vectors are based on a highest scoring joint of the model as determined by probabilities and distances.

9. The computer readable storage device of claim 1, wherein:

the articulated body portions comprise a palm, finger segments and thumb segments, each having at least one representative attract point.

10. A processor-implemented method for modeling a pose of an object, comprising the processor-implemented steps of:

obtaining depth pixels of the object in one or more frames;

processing the depth pixels of the one or more frames to identify centroids of articulated body portions of the object;

accessing a model, the model includes articulated body portions which correspond to the articulated body portions of the object and which each have at least one representative attract point;

matching the representative attract points to the centroids;

rigidly transforming respective positions of the representative attract points in 3D space based on the matching, to provide transformed representative attract points;

determining which of the transformed representative attract points are further than a threshold distance from a corresponding one of the centroids;

adjusting the transformed representative attract points which are further than a respective threshold distance from the corresponding one of the centroids, to be closer than the respective threshold distance from the corresponding one of the centroids;

rasterizing the model after the rigidly transforming and adjusting, to provide depth pixels of the model;

comparing the depth pixels of the model to the depth pixels of the one or more frames in at least one comparison plane; and

adjusting the model based on the comparing.

11. The processor-implemented method of claim 10, wherein the processing comprises, for the depth pixels of the one or more frames, and for the articulated body portions of the model:

determining a probability that the depth pixel of the one or more frames is part of the articulated body portion, and determining a centroid of the articulated body portion based on positions in 3D space of depth pixels of the one or more frames which have a relatively high probability of being part of the articulated body portion.

12. The processor-implemented method of claim 10, wherein:

at least one of the representative attract points of at least one of the articulated body portions of the model is at a location at which a corresponding one of the centroids is expected.

13. The processor-implemented method of claim 10, wherein:

the matching the representative attract points to the centroids is based on distances between the representative attract points and the centroids, and confidence measures between the centroids and the representative attract points.

14. The processor-implemented method of claim 13, wherein:

the matching the representative attract points to the centroids is based on ordering of the centroids along an axis of importance;

the articulated body portions of the model comprise fingers; and

the axis of importance indicates an expected order of the fingers.

15. The processor-implemented method of claim 10, wherein:

the comparing comprises identifying, from among the depth pixels of the one or more frames, non-overlapping depth pixels of the one or more frames which are not overlapping in the at least one comparison plane with the depth pixels of the model; and

the adjusting comprises moving at least one of the articulated body portions of the model which is adjacent to the non-overlapping depth pixels of the one or more frames, in a direction toward the non-overlapping depth pixels of the one or more frames.

16. The processor-implemented method of claim 15, wherein:

the processing comprises, for the non-overlapping depth pixels of the one or more frames, and for the articulated body portions of the model;

determining a probability that the non-overlapping depth pixel of the one or more frames is part of the articulated body portion;

the adjusting comprises determining a score, for each of the transformed representative attract points, based on;

(a) the probabilities; and

(b) distances between;

(i) the non-overlapping depth pixels of the one or more frames and (ii) the transformed representative attract points; and

the method performed further comprises, based on the scores;

identifying one or more of the articulated body portions to move.

17. The processor-implemented method of claim 10, wherein:

the depth pixels of the model include depth pixels of at least one of the articulated body portions of the model;

the comparing comprises identifying, from among the depth pixels of the one or more frames, overlapping depth pixels of the one or more frames which are overlapping in the at least one comparison plane with the depth pixels of the at least one of the articulated body portions of the model; and

the adjusting comprises moving the at least one of the articulated body portions of the model, in a direction to more closely match the overlapping depth pixels of the one or more frames.

18. The processor-implemented method of claim 10, wherein:

the comparing comprises identifying, from among the depth pixels of the model, non-overlapping pixels of the model which are not overlapping in the at least one comparison plane with the depth pixels of the one or more frames; and

the adjusting comprises moving at least one of the articulated body portions of the model which includes the non-overlapping depth pixels of the model, in a direction which tends to equalize a number of the non-overlapping depth pixels of the model around a perimeter of the at least one of the articulated body portions of the model.

19. An imaging system, comprising:

a depth sensor having a field of view, the depth sensor obtains depth pixels of an object in a field of view of the depth sensor in one or more frames;

a display; and

one or more processors in communication with the depth sensor and the display, the processor executes instructions to model a pose of the object and to provide a signal to the display to display images, the one or more processors;

process the depth pixels of the depth sensor to identify centroids of articulated portions of the object;

access a model of the articulated portions of the object, each articulated portion of the object having zero or more representative attract points;

use a rigid transform without changing relative orientations of the articulated portions of the object in the model, to match the model to the depth pixels of the depth sensor, by matching the representative attract points to the centroids;

provide changes to the relative orientations and to locations of the articulated portions of the object in the model, to refine the match of the model to the depth pixels of the depth sensor, by refining the match of the representative attract points to the centroids;

perform multiple iterations, each of the multiple iterations further refines the match of the model to the depth pixels of the depth sensor, by rasterizing the model to provide depth pixels of the model, and adjusting the relative orientations and locations of the articulated portions of the object in the model based on a comparison between the depth pixels of the depth sensor and the depth pixels of the model, the pose of the object is defined by the model after the multiple iterations;

provide a control input to an application based on the pose; and

update the display based on the control input.

20. The imaging system of claim 19, wherein:

in each of the multiple iterations;

the comparison between the depth pixels of the depth sensor and the depth pixels of the model identifies, from among the depth pixels of the depth sensor, non-overlapping depth pixels of the depth sensor which are not overlapping in at least one comparison plane with the depth pixels of the model;

each of the articulated portions of the model has at least one joint;

at least one of the articulated portions of the model is adjacent to the non-overlapping depth pixels of the depth sensor; and

each of the multiple iterations provides adjustment vectors for the joints, the adjustment vectors are based on the comparison to move at least one of the joints of the at least one of articulated portions of the model to more closely match the non-overlapping depth pixels of the depth sensor.