Pose tracking pipeline

• US 8,295,546 B2
• Filed: 10/21/2009
• Issued: 10/23/2012
• Est. Priority Date: 01/30/2009
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Craig, Robert Matthew, Leyvand, Tommer, Peeper, Craig, Al-Ghosien, Momin M., Bronder, Matt, Williams, Oliver, Geiss, Ryan M., Shotton, Jamie Daniel Joseph, Lee, Johnny, Finocchio, Mark
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US12/603,437
• Publication Number:
  US 20100197390A1
• Time in Patent Office: 1,098 Days
• Field of Search: 382/100, 382/103, 382106-107, 382/123, 382/155, 382/162, 382/168, 382/180, 382/181, 382189-199, 382/219, 382/232, 382/254, 382/274, 382/276, 382/291, 382/312, 382/305, 382/154, 375/240.08
• US Class Current: 382/103
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/1006 having additional degrees of freedom
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A method of tracking a target includes receiving from a source an observed depth image of a scene including the target. Each pixel of the observed depth image is labeled as either a foreground pixel belonging to the target or a background pixel not belonging to the target. Each foreground pixel is labeled with body part information indicating a likelihood that that foreground pixel belongs to one or more body parts of the target. The target is modeled with a skeleton including a plurality of skeletal points, each skeletal point including a three dimensional position derived from body part information of one or more foreground pixels.

Claims

1. A method of tracking a target, the method comprising:

receiving from a source one or more observed depth images of a scene including the target;

labeling each pixel of the one or more observed depth images as either a foreground pixel belonging to the target, a foreground pixel belonging to another target, or a background pixel not belonging to a target;

labeling each foreground pixel with body part information indicating a likelihood that that foreground pixel belongs to one or more body parts of the target; and

modeling the target with a virtual skeleton including a plurality of virtual skeletal points, each virtual skeletal point including a three dimensional position derived from body part information of one or more foreground pixels from the one or more observed depth images.

2. The method of claim 1, where the virtual skeleton is one of a plurality of plausible virtual skeletons derived from body part information of one or more foreground pixels from the one or more observed depth images.

3. The method of claim 2, where modeling the target with the virtual skeleton includes finding a plurality of candidate locations for each virtual skeletal bone defined by adjacent virtual skeletal points, and assembling the plurality of plausible virtual skeletons to include virtual skeletal bones at different combinations of the plurality of candidate locations.

4. The method of claim 3, where finding the plurality of candidate locations for each virtual skeletal bone includes finding a corresponding plurality of centroids for a body part corresponding to that virtual skeletal bone, each centroid defining a location of a clump of neighboring foreground pixels individually having body part probability distributions indicating that that body part is probable for that clump of neighboring foreground pixels, each centroid further defining a single probability representing all individual body part probability distributions within the clump.

5. The method of claim 3, further comprising deriving a single virtual skeleton from the plurality of plausible virtual skeletons.

6. The method of claim 5, where deriving a single virtual skeleton from the plurality of plausible virtual skeletons includes selecting one of the plurality of plausible virtual skeletons as the single virtual skeleton.

7. The method of claim 5, where deriving a single virtual skeleton from the plurality of plausible virtual skeletons includes combining parts of two or more of the plurality of plausible virtual skeletons into the single virtual skeleton.

8. The method of claim 5, where deriving a single virtual skeleton from the plurality of plausible virtual skeletons includes scoring two or more plausible virtual skeletons against one another based on one or more of weight, observed motion over time, anticipated bone length, and foreground/background crossing.

9. The method of claim 5, where deriving a single virtual skeleton from the plurality of plausible virtual skeletons includes applying one or more constraints to one or more of the plurality of plausible virtual skeletons.

10. The method of claim 1, where labeling each foreground pixel of an observed depth image with body part information includes applying to the observed depth image one or more decision trees trained from a collection of known poses to find a body part probability distribution indicating a likelihood that the foreground pixel belongs to each of one or more body parts.

11. The method of claim 1, where labeling each foreground pixel of an observed depth image with body part information includes applying to the observed depth image one or more decision trees trained from a collection of known poses and where a terminal node of each decision tree yields a best-guess of a body part for the foreground pixel and a confidence that the best-guess is correct.

12. The method of claim 1, where labeling each pixel as either a foreground pixel or a background pixel includes finding a floor in the observed depth image and labeling floor pixels as background pixels.

13. The method of claim 1, where labeling each pixel as either a foreground pixel or a background pixel includes creating a depth history image from two or more frames of depth information, and where a pixel is labeled as a foreground pixel if a depth value for that pixel is substantially smaller in the observed depth image than in the depth history image.

14. A method of tracking a target, the method comprising:

receiving from a depth camera one or more observed depth images of a scene including the target;

labeling each pixel of the one or more observed depth images as either a foreground pixel belonging to the target, a foreground pixel belonging to another target, or a background pixel not belonging to a target;

labeling each foreground pixel of the target with a body part probability distribution indicating a likelihood that that foreground pixel belongs to each of one or more body parts of the target; and

independently extracting a plurality of candidate locations for each of a plurality of virtual skeletal bones based on the body part probability distribution of each foreground pixel of the target; and

assembling a plurality of plausible virtual skeletons that include virtual skeletal bones at different combinations of the plurality of candidate locations, each plausible virtual skeleton including a plurality of virtual skeletal points, each virtual skeletal point defining a three dimensional position.

15. The method of claim 14, further comprising deriving a single virtual skeleton from the plurality of plausible virtual skeletons.

16. The method of claim 15, where deriving a single virtual skeleton from the plurality of plausible virtual skeletons includes selecting one of the plurality of plausible virtual skeletons as the single virtual skeleton.

17. The method of claim 14, where deriving a single virtual skeleton from the plurality of plausible virtual skeletons includes combining parts of two or more of the plurality of plausible virtual skeletons into the single virtual skeleton.

18. A computing system, comprising:

a source configured to capture depth information;

a logic subsystem in operative communication with the source; and

a data-holding subsystem holding instructions executable by the logic subsystem to;

receive from the source one or more observed depth images of a scene including a target;

label each pixel of the one or more observed depth images as either a foreground pixel belonging to the target, a foreground pixel belonging to another target, or a background pixel not belonging to a target;

label each foreground pixel of the target with body part information indicating a likelihood that that foreground pixel belongs to one or more body parts of the target; and

model the target with a virtual skeleton including a plurality of virtual skeletal points, each virtual skeletal point including a three dimensional position derived from body part information of one or more foreground pixels from the one or more observed depth images.

19. The computing system of claim 18, where the virtual skeleton is one of a plurality of plausible virtual skeletons derived from body part information of one or more foreground pixels from the one or more observed depth images.

20. The computing system of claim 19, where modeling the target with the virtual skeleton includes finding a plurality of candidate locations for each virtual skeletal bone defined by adjacent virtual skeletal points, and assembling the plurality of plausible virtual skeletons to include virtual skeletal bones at different combinations of the plurality of candidate locations.