POSE TRACKING PIPELINE

• US 20100197390A1
• Filed: 10/21/2009
• Published: 08/05/2010
• Est. Priority Date: 01/30/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lee, Johnny, Al-Ghosien, Momim M., Bronder, Matt, Shotton, Jamie Daniel Joseph, Finocchio, Mark, Peeper, Craig, Geiss, Ryan M., Leyvand, Tommer, Williams, Oliver, Craig, Robert Matthew
• Granted Patent: US 8,295,546 B2
• Time in Patent Office: Days
• US Class Current: 463/30
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/1006 having additional degrees of freedom
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A method of tracking a target includes receiving from a source an observed depth image of a scene including the target. Each pixel of the observed depth image is labeled as either a foreground pixel belonging to the target or a background pixel not belonging to the target. Each foreground pixel is labeled with body part information indicating a likelihood that that foreground pixel belongs to one or more body parts of the target. The target is modeled with a skeleton including a plurality of skeletal points, each skeletal point including a three dimensional position derived from body part information of one or more foreground pixels.

Claims

1. A method of tracking a target, the method comprising:

receiving from a source one or more observed depth images of a scene including the target;

labeling each pixel of the one or more observed depth images as either a foreground pixel belonging to the target, a foreground pixel belonging to another target, or a background pixel not belonging to a target;

labeling each foreground pixel with body part information indicating a likelihood that that foreground pixel belongs to one or more body parts of the target; and

modeling the target with a skeleton including a plurality of skeletal points, each skeletal point including a three dimensional position derived from body part information of one or more foreground pixels from one or more observed depth images.

2. The method of claim 1, where the skeleton is one of a plurality of plausible skeletons derived from body part information of one or more foreground pixels from one or more observed depth images.

3. The method of claim 2, where modeling the target with the skeleton includes finding a plurality of candidate locations for each skeletal bone defined by adjacent skeletal points, and assembling the plurality of plausible skeletons to include skeletal bones at different combinations of the plurality of candidate locations.

4. The method of claim 3, where finding the plurality of candidate locations for each skeletal bone includes finding a corresponding plurality of centroids for a body part corresponding to that skeletal bone, each centroid defining a location of a clump of neighboring foreground pixels individually having body part probability distributions indicating that that body part is probable for that clump of neighboring foreground pixels, each centroid further defining a single probability representing all individual body part probability distributions within the clump.

5. The method of claim 3, further comprising deriving a single skeleton from the plurality of plausible skeletons.

6. The method of claim 5, where deriving a single skeleton from the plurality of plausible skeletons includes selecting one of the plurality of plausible skeletons as the single skeleton.

7. The method of claim 5, where deriving a single skeleton from the plurality of plausible skeletons includes combining parts of two or more of the plurality of plausible skeletons into the single skeleton.

8. The method of claim 5, where deriving a single skeleton from the plurality of plausible skeletons includes scoring two or more plausible skeletons against one another based on one or more of weight, observed motion over time, anticipated bone length, and foreground/background crossing.

9. The method of claim 5, where deriving a single skeleton from the plurality of plausible skeletons includes applying one or more constraints to one or more of the plurality of plausible skeletons.

10. The method of claim 1, where labeling each foreground pixel of an observed depth image with body part information includes applying to the observed depth image one or more decision trees trained from a collection of known poses to find a body part probability distribution indicating a likelihood that the foreground pixel belongs to each of one or more body parts.

11. The method of claim 1, where labeling each foreground pixel of an observed depth image with body part information includes applying to the observed depth image one or more decision trees trained from a collection of known poses and where a terminal node of each decision tree yields a best-guess of a body part for the foreground pixel and a confidence that the best-guess is correct.

12. The method of claim 1, where labeling each pixel as either a foreground pixel or a background pixel includes finding a floor in the observed depth image and labeling floor pixels as background pixels.

13. The method of claim 1, where labeling each pixel as either a foreground pixel or a background pixel includes creating a depth history image from two or more frames of depth information, and where a pixel is labeled as a foreground pixel if a depth value for that pixel is substantially smaller in the observed depth image than in the depth history image.

14. A method of tracking a target, the method comprising:

receiving from a depth camera one or more observed depth images of a scene including the target;

labeling each pixel of the one or more observed depth images as either a foreground pixel belonging to the target, a foreground pixel belonging to another target, or a background pixel not belonging to a target;

labeling each foreground pixel of the target with a body part probability distribution indicating a likelihood that that foreground pixel belongs to each of one or more body parts of the target; and

independently extracting a plurality of candidate locations for each of a plurality of skeletal bones based on the body part probability distribution of each foreground pixel of the target; and

assembling a plurality of plausible skeletons that include skeletal bones at different combinations of the plurality of candidate locations, each plausible skeleton including a plurality of skeletal points, each skeletal point defining a three dimensional position.

15. The method of claim 14, further comprising deriving a single skeleton from the plurality of plausible skeletons.

16. The method of claim 15, where deriving a single skeleton from the plurality of plausible skeletons includes selecting one of the plurality of plausible skeletons as the single skeleton.

17. The method of claim 14, where deriving a single skeleton from the plurality of plausible skeletons includes combining parts of two or more of the plurality of plausible skeletons into the single skeleton.

18. A computing system, comprising:

a source configured to capture depth information;

a logic subsystem in operative communication with the source; and

a data-holding subsystem holding instructions executable by the logic subsystem to;

receive from the source one or more observed depth images of a scene including the target;

label each pixel of the one or more observed depth images as either a foreground pixel belonging to the target, a foreground pixel belonging to another target, or a background pixel not belonging to a target;

label each foreground pixel of the target with body part information indicating a likelihood that that foreground pixel belongs to one or more body parts of the target; and

model the target with a skeleton including a plurality of skeletal points, each skeletal point including a three dimensional position derived from body part information of one or more foreground pixels from one or more observed depth images.

19. The computing system of claim 18, where the skeleton is one of a plurality of plausible skeletons derived from body part information of one or more foreground pixels from one or more observed depth images.

20. The computing system of claim 19, where modeling the target with the skeleton includes finding a plurality of candidate locations for each skeletal bone defined by adjacent skeletal points, and assembling the plurality of plausible skeletons to include skeletal bones at different combinations of the plurality of candidate locations.