Human Body Pose Estimation

• US 20110210915A1
• Filed: 03/03/2011
• Published: 09/01/2011
• Est. Priority Date: 05/01/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kim, David, Fitzgibbon, Andrew William, Girshick, Ross Brook, Cook, Matthew Darius, Molyneaux, David Geoffrey, Kohli, Pushmeet, Criminisi, Antonio, Shotton, Jamie Daniel Joseph, Izadi, Shahram, Hilliges, Otmar
• Granted Patent: US 8,638,985 B2
• Time in Patent Office: Days
• US Class Current: 345/157
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

Techniques for human body pose estimation are disclosed herein. Images such as depth images, silhouette images, or volumetric images may be generated and pixels or voxels of the images may be identified. The techniques may process the pixels or voxels to determine a probability that each pixel or voxel is associated with a segment of a body captured in the image or to determine a three-dimensional representation for each pixel or voxel that is associated with a location on a canonical body. These probabilities or three-dimensional representations may then be utilized along with the images to construct a posed model of the body captured in the image.

Claims

1. A method for determining a position of a body using a set of one or more segments, each segment being associated with a portion of the body, the method comprising:

identifying a set of one or more pixels in an image that are associated with at least part of the body; and

for each pixel in the set of pixels, assigning to the pixel a probability that the pixel is within each segment in the set of segments.

2. The method of claim 1, wherein the image is a silhouette image that distinguishes between pixels belonging to the at least part of the body and pixels belonging to a background.

3. The method of claim 2, further comprising generating the silhouette image from a two-dimensional image or a three-dimensional image using a background subtraction algorithm, a GrabCut algorithm, or geodesic segmentation.

4. The method of claim 1, further comprising determining a depth of the pixels in the set of pixels.

5. The method of claim 4, wherein the depth is an average measured depth of the at least part of the body or a default depth.

6. The method of claim 1, wherein the probability is a first probability, and wherein assigning to the pixel the first probability that the pixel is within each segment comprises:

determining at least one of depth or second probability at the pixel;

selecting at least one other pixel each at some angle and distance from the pixel;

determining at least one of depth or a second probability at each of the at least one other pixels; and

performing the assigning of the first probability based at least in part on a relationship between the depth or the second probability at the pixel and the depth or the second probability at each of the at least one other pixels.

7. The method of claim 6, further comprising determining the distance and the angle of each other pixel from the pixel based on one or more decision trees, each tree being generated based on a different set of body pose data.

8. The method of claim 1, further comprising:

using the assigned first probabilities to determine a centroid pixel within each of the segments; and

determining a location of one or more joints based on the centroid pixels.

9. A system for determining a position of a body using a set of one or more segments, each segment being associated with a portion of the body, the system comprising:

a plurality of cameras for generating a plurality of images, at least one of the cameras being placed at a different angle with respect to another of the cameras;

a processor that executes instructions to perform operations comprising;

generating a volumetric image from the plurality of images;

identifying a set of one or more voxels in a volumetric image that are associated with at least part of the body; and

for each voxel in the set of voxels, assigning to the voxel a probability that the voxel is within each segment in the set of segments.

10. The system of claim 9, wherein the probability is a first probability, and wherein assigning to the voxel the first probability that the voxel is within each segment comprises:

determining at least one of an occupancy or a second probability at the voxel;

selecting at least one other voxel each at some angle and distance from the voxel;

determining at least one of an occupancy or the second probability at each of the at least one other voxels; and

performing the assigning of the first probability based at least in part on a relationship between the occupancy or the second probability at the voxel and the occupancy or the second probability at each of the at least one other voxels.

11. The system of claim 10, wherein the distance and the angle of each other voxel from the voxel is determined based on one or more decision trees, each tree being generated based on a different set of body pose data.

12. The system of claim 11, wherein sets of the body pose data comprise volumetric representations of body poses.

13. The system of claim 9, wherein the operations further comprise:

using the assigned probabilities to determine a centroid voxel within each of the segments; and

determining a location of one or more joints based on the centroid voxels.

14. One or more computer storage devices having recorded thereon computer executable instructions for associating locations on a body captured in an image with locations on a model body having a canonical pose, the instructions when executed causing a computing device to perform operations comprising:

identifying a set of one or more pixels or voxels in the image that includes at least part of the body; and

for each pixel or voxel in the set of pixels or voxels, determining a three-dimensional representation that is associated with a location on the model body.

15. The one or more computer storage devices of claim 14, wherein the image is a depth image, a silhouette image, or a volumetric image.

16. The one or more computer storage devices of claim 14, wherein the operations further comprise utilizing the determined three-dimensional representations and the image to ascertain a pose of the body captured in the image.

17. The one or more computer storage devices of claim 14, wherein the three-dimensional representation comprises a three-dimensional coordinate and a variance.

18. The one or more computer storage devices of claim 14, wherein the three-dimensional representation for a specific location on or in the body captured in the image is the same regardless of where within the image the specific location is found and regardless of a shape, a size, or a pose of the body.

19. The one or more computer storage devices of claim 14, wherein the determining comprises performing feature tests at nodes of one or more decision trees, each tree being generated based on a different set of body pose data that describes body poses using three-dimensional representations associated with locations on the model body.

20. The one or more computer storage devices of claim 19, wherein the operations further comprise generating the one or more decision trees, the generating including selecting a feature for each node of each decision tree based on an minimization function that minimizes spacial variance among examples that have a same result with respect to a binary test associated with the feature.

21. The one or more computer storage devices of claim 14, wherein the operations further comprise fitting a model person onto the pixels or voxels based on the three-dimensional representations determined for those pixels or voxels to estimate a user pose captured in the image.