Human body pose estimation

• US 10,210,382 B2
• Filed: 12/22/2015
• Issued: 02/19/2019
• Est. Priority Date: 05/01/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Shotton, Jamie Daniel Joseph, Fitzgibbon, Andrew William
• Primary Examiner(s): Lu, Tom Y
• Application Number:
  US14/978,123
• Publication Number:
  US 20160171295A1
• Time in Patent Office: 1,155 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Techniques for human body pose estimation are disclosed herein. Depth map images from a depth camera may be processed to calculate a probability that each pixel of the depth map is associated with one or more segments or body parts of a body. Body parts may then be constructed of the pixels and processed to define joints or nodes of those body parts. The nodes or joints may be provided to a system which may construct a model of the body from the various nodes or joints.

Claims

1. A method of generating a three dimensional model of at least part of a body, comprising:

receiving, at a computing system, a depth image comprising pixels having coordinates with xyz values;

determining, at the computing system, using the coordinates of the pixels that a first pixel in the depth image corresponds to the at least part of the body;

determining, at the computing system, using the coordinates of the pixels that a second pixel in the depth image corresponds to the at least part of the body based on a position of the second pixel relative to the first pixel;

selecting a third pixel among a plurality of pixels in the depth image for determination of whether the third pixel corresponds to the at least part of the body based on determining that the second pixel in the depth image corresponds to the at least part of the body, and based on a position of the third pixel relative to the second pixel; and

generating the three dimensional model using the first pixel, the second pixel and the third pixel.

2. The method of claim 1,

wherein determining that the first pixel in the depth image corresponds to the at least part of the body comprises determining a first probability that the first pixel in the depth image corresponds to the at least part of the body, and

wherein determining that the second pixel in the depth image corresponds to the at least part of the body comprises:

determining that the second pixel in the depth image corresponds to the at least part of the body based on the position of the second pixel relative to the first pixel, and the first probability.

3. The method of claim 2, further comprising:

determining that the third pixel in the depth image has a zero probability of corresponding to the at least part of the body based on the third pixel having a depth value associated with a background.

4. The method of claim 1,

wherein determining that the second pixel in the depth image corresponds to the at least part of the body based on the position of the second pixel relative to the first pixel comprises determining that the second pixel in the depth image corresponds to the at least part of the body using a decision tree.

5. The method of claim 1,

wherein determining that the second pixel in the depth image corresponds to the at least part of the body comprises determining that the second pixel in the depth image corresponds to the at least part of the body based on an angle and a distance of the second pixel relative to the first pixel.

6. The method of claim 5, wherein determining that the second pixel in the depth image corresponds to the at least part of the body comprises:

determining that the second pixel in the depth image corresponds to the at least part of the body based on the angle and the distance of the second pixel relative to the first pixel, and a depth value of the first pixel.

7. A system for generating a three dimensional model of at least part of a body, comprising:

computing memory bearing instructions that cause the system to perform operations comprising;

receive a depth image comprising pixels having coordinates with xyz values;

determine using the coordinates of the pixels that a first pixel in the depth image corresponds to the at least part of the body;

determine using the coordinates of the pixels that a second pixel in the depth image corresponds to the at least part of the body based on a position of the second pixel relative to the first pixel;

select a third pixel among a plurality of pixels in the depth image for determination of whether the third pixel corresponds to the at least part of the body based on determining that the second pixel in the depth image corresponds to the at least part of the body, and based on a position of the third pixel relative to the second pixel; and

generate the three dimensional model using the first pixel, the second pixel and the third pixel.

8. The system of claim 7,

wherein the memory further bears instructions that cause the system at least to:

select the second pixel among the plurality of pixels in the depth image for determination of whether the second pixel corresponds to the at least part of the body based on the position of the second pixel relative to the first pixel and a threshold amount.

9. The system of claim 8,

wherein the threshold amount would increase where a depth value associated with the first pixel were to decrease.

10. The system of claim 7,

wherein the memory further bears instructions that cause the system at least to:

determine a centroid pixel of the at least part of the body based on the first pixel and the second pixel.

11. The system of claim 10,

wherein the memory further bears instructions that cause the system at least to:

determine a location of a joint of the at least part of the body based on the centroid pixel.

12. The system of claim 7,

wherein the instructions that cause the system at least to determine that the first pixel in the depth image corresponds to the at least part of the body further cause the system at least to determine a first probability that the first pixel in the depth image corresponds to the at least part of the body, and

wherein the instructions that cause the system at least to determine that the second pixel in the depth image corresponds to the at least part of the body further cause the system at least to:

determine that the second pixel in the depth image corresponds to the at least part of the body based on the position of the second pixel relative to the first pixel, and the first probability.

13. The system of claim 12,

wherein the memory further bears instructions that cause the system at least to:

determine that the third pixel in the depth image has a zero probability of corresponding to the at least part of the body based on the third pixel having a depth value associated with a background.

14. A computer-readable storage device that is not a propagating signal comprising computer-executable instructions that cause a computing system to perform operations comprising:

receiving a depth image comprising pixels having xyz coordinates;

determining using the coordinates of the pixels that a first pixel in the depth image corresponds to at least part of a body;

determining using the coordinates of the pixels that a second pixel in the depth image corresponds to the at least part of the body based on a position of the second pixel relative to the first pixel;

selecting a third pixel among a plurality of pixels in the depth image for determination of whether the third pixel corresponds to the at least part of the body based on determining that the second pixel in the depth image corresponds to the at least part of the body, and based on a position of the third pixel relative to the second pixel; and

generating a three dimensional model of the at least part of the body using the first pixel, the second pixel and the third pixel.

15. The computer-readable storage device of claim 14,

wherein determining that the first pixel in the depth image corresponds to the at least part of the body comprises determining a first probability that the first pixel in the depth image corresponds to the at least part of the body, and

wherein determining that the second pixel in the depth image corresponds to the at least part of the body comprises:

determining that the second pixel in the depth image corresponds to the at least part of the body based on the position of the second pixel relative to the first pixel, and the first probability.

16. The computer-readable storage device of claim 15, further comprising computer-executable instructions that cause the computing system to perform operations comprising:

determining that the third pixel in the depth image has a zero probability of corresponding to the at least part of the body based on the third pixel having a depth value associated with a background.

17. The computer-readable storage device of claim 14, further comprising computer-executable instructions that, upon execution on the computing system, cause the computing system to perform operations comprising:

selecting the second pixel among the plurality of pixels in the depth image for determination of whether the second pixel corresponds to the at least part of the body based on the position of the second pixel relative to the first pixel and a threshold amount.

18. The computer-readable storage device of claim 17, wherein the threshold amount would increase where if a depth value associated with the first pixel were to decrease.