Pose tracking pipeline

• US 9,465,980 B2
• Filed: 09/05/2014
• Issued: 10/11/2016
• Est. Priority Date: 01/30/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Craig, Robert Matthew, Leyvand, Tommer, Peeper, Craig, Al-Ghosien, Momin M., Bronder, Matt, Williams, Oliver, Geiss, Ryan M., Shotton, Jamie Daniel Joseph, Lee, Johnny, Finocchio, Mark
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US14/478,778
• Publication Number:
  US 20150145860A1
• Time in Patent Office: 767 Days
• Field of Search: 382/103, 382106-107, 382/123, 382/155, 382/162, 382/168, 382180-181, 382189-199, 382/219, 382/232, 382/254, 382/274, 382/276, 382/291, 382/312, 382/154, 382/305, 70/47, 375/240.08, 375/240.12, 345/158, 345/419, 348/155, 348/143, 348/169
• US Class Current: 1/1
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/1006 having additional degrees of freedom
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A method of tracking a subject includes receiving from a source a depth image of a scene including the subject. The depth image includes a depth for each of a plurality of pixels. The method further includes identifying pixels of the depth image that image the subject and deriving from the identified pixels of the depth image one or more machine readable data structures representing the subject as a model including a plurality of shapes.

Claims

1. A method performed using a system, comprising:

based on at least one image of at least part of a scene obtained from an image capture device, the at least part of the scene including one or more parts of a human body in a first position, separating image data included in the at least one image and corresponding to the one or more parts of the human body from image data included in the at least one image and corresponding to a background of the at least part of the scene;

developing a first model corresponding to the one or more parts of the human body based at least in part on the image data corresponding to the one or more parts of the human body, the first model including one or more skeletal points;

developing a second model based at least in part on data associated with the one or more skeletal points and including shapes corresponding to the one or more parts of the human body;

displaying the second model; and

controlling operation of a program executing on the system based at least in part on using the data associated with the one or more skeletal points of the first model to detect movement of the one or more parts of the human body, the based at least in part on using the data associated with the one or more skeletal points of the first model to detect movement of the one or more parts of the human body performed by the system being based at least in part on the at least one image of the at least part of the scene and an additional one or more images.

2. The method of claim 1, wherein the shapes include a cylinder or a sphere.

3. The method of claim 1, wherein the one or more skeletal points include a roll joint, each roll joint utilized to track an axial roll angle.

4. The method of claim 1, wherein the program is a gaming program and the system is a gaming system.

5. The method of claim 1, wherein the image capture device includes a depth camera, and the at least one image includes at least one depth image.

6. The method of claim 5, further comprising:

downsampling the at least one depth image to a downsampled at least one depth image having a lower processing resolution than the at least one depth image.

7. The method of claim 5, wherein separating image data includes distinguishing one or more parts of the human body that are to be tracked from background elements in the at least one depth image.

8. The method of claim 7, wherein the at least one depth image includes two or more depth frames, and the method further comprises:

deriving a depth history image from the two or more depth frames, wherein a depth value for each pixel of the depth history image is set to a deepest depth value the pixel experiences during the two or more depth frames; and

wherein distinguishing one or more parts of the human body includes identifying moving objects in a foreground of the scene from non-moving background elements using the depth history image.

9. The method of claim 8, wherein identifying moving objects includes determining whether an object is behind a reference plate that is a history of minimum or maximum depth values observed for the non-moving background elements.

10. The method of claim 1, wherein the one or more parts of the human body belong to a player of an electronic game holding an object, such that motions of the player and the object are utilized to adjust and/or control parameters of the electronic game.

11. The method of claim 1, wherein controlling operation of the program includes one or more of ending a game, pausing the game, selecting a level of the game, viewing scores associated with the game, or saving the game.

12. A system comprising:

a capture device configured to obtain one or more images of at least part of a scene;

a logic device;

a data-holding device configured to hold instructions executable by the logic device to;

obtain via the capture device at least one image of the at least part of the scene, the scene including one or more parts of a human body in a first position;

separate image data included in the at least one image and corresponding to the one or more parts of the human body from image data included in the at least one image and corresponding to a background of the at least part of the scene;

develop a first model corresponding to the one or more parts of the human body using the image data corresponding to the one or more parts of the human body, the first model including one or more skeletal points;

develop a second model based at least in part on data associated with the one or more skeletal points and including shapes corresponding to the one or more parts of the human body;

display the second model; and

control operation of a program executing on the system based at least in part on using the data associated with the one or more skeletal points of the first model to detect movement of the one or more parts of the human body, the based at least in part on using the data associated with the one or more skeletal points of the first model to detect movement of the one or more parts of the human body performed by the system being based at least in part on the at least one image of the at least part of the scene and an additional one or more images.

13. A method performed using a system, comprising:

based at least in part on a first set of images of at least part of a scene including one or more parts of a human body in a first position and obtained from a plurality of image capture devices, with each of the plurality of image capture devices located in a different position relative to the other of the plurality of image capture devices, developing a first model corresponding to the one or more parts of the human body in the first position, the first model including one or more skeletal points;

developing a first display model based at least in part on data associated with the one or more skeletal points of the first model and including shapes corresponding to the one or more parts of the human body;

displaying the first display model;

developing a second model based at least in part on a second set of images of at least part of the scene including the one or more parts of the human body in a second position and obtained from the plurality of image capture devices, the second model including the one or more skeletal points

developing a second display model based at least in part on data associated with the one or more skeletal points of the second model and including shapes corresponding to the one or more parts of the human body;

displaying the second display model;

determining differences between the first model and the second model; and

controlling at least a portion of operation of a program executing on the system based at least in part on the differences between the first model and the second model.

14. The method of claim 13, wherein the shapes include a cylinder or a sphere.

15. The method of claim 13, wherein the one or more body parts include a human finger.

16. The method of claim 13, wherein the one or more skeletal points include one or more joints, the one or more joints include roll joints, each roll joint utilized to track axial roll angles.

17. The method of claim 16, wherein the first model and the second model further include one or more model parts corresponding to an object held by a player of a gaming program, and wherein motions of the player and the object are utilized to adjust and/or control parameters of the gaming program.

18. The method of claim 16, wherein controlling operation of the program includes one or more of ending a game, pausing the game, selecting a level of the game, viewing scores associated with the game, or saving the game.

19. The method of claim 13, wherein the plurality of image capture devices includes a plurality of depth cameras, wherein the first set of images includes a first set of depth images, and wherein the second set of images includes a second set of depth images.

20. The method of claim 19, wherein forming the first model includes separating image data included in the first set of depth images and corresponding to the one or more parts of the human body from image data included in the first set of images and corresponding to a background of the scene, and wherein forming the second model includes separating image data included in the second set of images and corresponding to the one or more parts of the human body from image data included in the second set of image depth images and corresponding to the background of the scene.

21. The method of claim 13, wherein the second model is formed based at least in part on the first model and changes in position between the one or more parts of the human body in the first position in the first set of images and the one or more parts of the human body in the second position in the second set of images.

22. A system comprising:

a capture device configured to obtain one or more images of at least part of a scene;

a logic device;

a data-holding device configured to hold instructions executable by the logic device to;

obtain via the capture device a first set of images of the at least part of the scene, the scene including one or more parts of a human body in a first position;

develop a first model corresponding to the one or more parts of the human body in the first position based at least in part on the first set of images, the first model including one or more skeletal points; and

develop a first display model based at least in part on data associated with the one or more skeletal points of the first model and including shapes corresponding to the one or more parts of the human body;

display the first display model;

develop a second model based at least in part on a second set of images of at least part of the scene including the one or more parts of the human body in a second position, the second model including the one or more skeletal points;

develop a second display model based at least in part on data associated with the one or more skeletal points of the second model and including shapes corresponding to the one or more parts of the human body;

display the second display model;

determine differences between the first model and the second model; and

control operation of a program executing on the system based at least in part on differences between the first model and the second model.