Pose tracking pipeline

• US 8,565,485 B2
• Filed: 09/13/2012
• Issued: 10/22/2013
• Est. Priority Date: 01/30/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Craig, Robert Matthew, Leyvand, Tommer, Peeper, Craig, Al-Ghosien, Momin M., Bronder, Matt, Williams, Oliver, Geiss, Ryan M., Shotton, Jamie Daniel Joseph, Lee, Johnny, Finocchio, Mark
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US13/615,269
• Publication Number:
  US 20130028476A1
• Time in Patent Office: 404 Days
• Field of Search: 382/100, 382/103, 382106-107, 382/123, 382/155, 382/162, 382/180, 382/181, 382189-199, 382/219, 382/232, 382/254, 382/274, 382/276, 382/291, 382/312, 382/305, 382/154, 375/240.08, 375/240.01
• US Class Current: 382/103
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/1006 having additional degrees of freedom
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A method of tracking a target includes receiving from a source a depth image of a scene including the human subject. The depth image includes a depth for each of a plurality of pixels. The method further includes identifying pixels of the depth image that belong to the human subject and deriving from the identified pixels of the depth image one or more machine readable data structures representing the human subject as a body model including a plurality of shapes.

Claims

1. A method of modeling a human subject, the method comprising:

receiving, from a single depth camera viewing a scene from a single perspective, a depth map of the scene including the human subject, the depth map including a depth for each of a plurality of depth-map pixels;

for each of a plurality of depth-map pixels imaging the human subject, associating with that depth-map pixel a body-part likelihood that that depth-map pixel images a body part of the human subject;

modeling the human subject with a plurality of plausible virtual skeletons each including a plurality of virtual joints, each virtual joint defined with a three-dimensional position derived from one or more depth-map pixels associated with a body-part likelihood corresponding to that virtual joint; and

deriving a single virtual skeleton from the plurality of plausible virtual skeletons.

2. The method of claim 1, where modeling the human subject with the virtual skeleton includes finding a plurality of candidate locations for each virtual bone defined by adjacent virtual joints, and assembling the plurality of plausible virtual skeletons to include virtual bones at different combinations of the plurality of candidate locations.

3. The method of claim 2, where finding the plurality of candidate locations for each virtual bone includes finding a corresponding plurality of centroids for a body part corresponding to that virtual bone, each centroid defining a location of a clump of neighboring depth-map pixels individually having body-part likelihoods indicating that that body part is probable for that clump of neighboring depth-map pixels, each centroid further defining a single probability representing all individual body-part likelihoods within the clump.

4. The method of claim 1, where deriving a single virtual skeleton from the plurality of plausible virtual skeletons includes selecting one of the plurality of plausible virtual skeletons as the single virtual skeleton.

5. The method of claim 1, where deriving a single virtual skeleton from the plurality of plausible virtual skeletons includes combining parts of two or more of the plurality of plausible virtual skeletons into the single virtual skeleton.

6. The method of claim 1, where deriving a single virtual skeleton from the plurality of plausible virtual skeletons includes applying one or more constraints to one or more of the plurality of plausible virtual skeletons.

7. The method of claim 1, where associating the body-part likelihood with the depth-map pixel includes applying to the depth map one or more machine learning algorithms trained from a collection of known poses to find a body-part likelihood that the depth-map pixel images each of one or more body parts.

8. A method of modeling a human subject, the method comprising:

receiving, from a single depth camera viewing a scene from a single perspective, a depth map of a scene including the human subject;

modeling the human subject with a virtual skeleton including a plurality of virtual joints, each virtual joint defined with a three-dimensional position derived from a depth of one or more pixels of the depth map, the three-dimensional position of each joint defined in a same coordinate system; and

outputting the three-dimensional position for each virtual joint in the same coordinate system.

9. The method of claim 8, where the three-dimensional position for each virtual joint is output via an application programming interface.

10. A method of modeling a human subject, the method comprising:

receiving, from a single depth camera viewing a scene from a single perspective, a depth map of the scene including the human subject, the depth map including a depth for each of a plurality of depth-map pixels;

for each of a plurality of depth-map pixels imaging the human subject, associating with that depth-map pixel a body-part likelihood that that depth-map pixel images a body part of the human subject;

modeling the human subject with a plurality of plausible virtual skeletons each including a plurality of virtual bones virtually extending between two virtual joints, each virtual joint defined with a three-dimensional position derived from one or more depth-map pixels associated with a body-part likelihood corresponding to one or more virtual bones extending from that virtual joint; and

deriving a single virtual skeleton from the plurality of plausible virtual skeletons.

11. The method of claim 10, where modeling the human subject with the virtual skeleton includes finding a plurality of candidate locations for each virtual bone, and assembling the plurality of plausible virtual skeletons to include virtual bones at different combinations of the plurality of candidate locations.

12. The method of claim 11, where finding the plurality of candidate locations for each virtual bone includes finding a corresponding plurality of centroids for a body part corresponding to that virtual bone, each centroid defining a location of a clump of neighboring depth-map pixels individually having body-part likelihoods indicating that that body part is probable for that clump of neighboring depth-map pixels, each centroid further defining a single probability representing all individual body-part likelihoods within the clump.

13. The method of claim 10, where deriving a single virtual skeleton from the plurality of plausible virtual skeletons includes selecting one of the plurality of plausible virtual skeletons as the single virtual skeleton.

14. The method of claim 10, where deriving a single virtual skeleton from the plurality of plausible virtual skeletons includes combining parts of two or more of the plurality of plausible virtual skeletons into the single virtual skeleton.

15. The method of claim 10, where deriving a single virtual skeleton from the plurality of plausible virtual skeletons includes applying one or more constraints to one or more of the plurality of plausible virtual skeletons.

16. The method of claim 10, where associating the body-part likelihood with the depth-map pixel includes applying to the depth map one or more machine learning algorithms trained from a collection of known poses to find a body-part likelihood that the depth-map pixel images each of one or more body parts.