Visual target tracking

• US 9,842,405 B2
• Filed: 10/23/2013
• Issued: 12/12/2017
• Est. Priority Date: 01/30/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geiss, Ryan M.
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): CHEN, XUEMEI G
• Application Number:
  US14/061,298
• Publication Number:
  US 20140051515A1
• Time in Patent Office: 1,511 Days
• Field of Search: 345424, 363 31
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: A63F 2300/6615 using models with different levels of detail [LOD]
• CPC Class Codes: A63F 2300/8029 Fighting without shooting
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models

Abstract

A method of tracking a target includes classifying a pixel having a pixel address with one or more pixel cases. The pixel is classified based on one or more observed or synthesized values. An example of an observed value for a pixel address includes an observed depth value obtained from a depth camera. Examples of synthesized values for a pixel address include a synthesized depth value calculated by rasterizing a model of the target; one or more body-part indices estimating a body part corresponding to that pixel address; and one or more player indices estimating a target corresponding to that pixel address. One or more force vectors are calculated for the pixel based on the pixel case, and the force vector is mapped to one or more force-receiving locations of the model representing the target to adjust the model representing the target into an adjusted pose.

Claims

1. A method of tracking a human subject, the method comprising:

obtaining an observed depth image including the human subject from a depth camera;

generating a machine-readable model representing the human subject from the observed depth image;

deriving a synthesized depth image from the machine-readable model representing the human subject, the synthesized depth image including a two-dimensional matrix of synthesized pixels, each synthesized pixel in the two-dimensional matrix of synthesized pixels including a depth value and having a same pixel address as a corresponding pixel of the observed depth image;

determining, for each synthesized pixel, a depth difference between a depth of that synthesized pixel and a depth of a corresponding pixel from a subsequent observed depth image; and

adjusting the machine-readable model representing the human subject into an adjusted pose based on the depth difference and different pixel cases for different ones of a plurality of synthesized pixels.

2. The method of claim 1, where the machine-readable model comprises one or both of polygonal meshes and a set of mathematical primitives, where the synthesized depth image is derived, at least in part, by rasterizing the machine-readable model representing the human subject, and wherein the depth value of each synthesized pixel represents a depth value at a corresponding location of the machine-readable model.

3. The method of claim 1, further comprising classifying each synthesized pixel of the plurality of synthesized pixels with a pixel case.

4. The method of claim 3, where each synthesized pixel is classified with the pixel case based on at least a body-part index of the synthesized pixel.

5. The method of claim 4, where each synthesized pixel is classified with the pixel case, at least in part, based on a difference between an original body-part index of the synthesized pixel and a snap body-part index of the synthesized pixel or a grow body-part index of the synthesized pixel.

6. The method of claim 3, where each synthesized pixel is classified with the pixel case based on at least a player index of the synthesized pixel.

7. The method of claim 6, where the synthesized pixel is classified with the pixel case, at least in part, based on a difference between an original player index of the synthesized pixel and a snap player index of the synthesized pixel or a grow player index of the synthesized pixel.

8. The method of claim 3, where each synthesized pixel is classified with the pixel case based on at least the depth difference for that synthesized pixel.

9. The method of claim 3, where each synthesized pixel has a pixel address, and where the synthesized pixel at the pixel address is classified with the pixel case based on one or more of an original player index, a snap player index, a grow player index, an original body-part index, a snap body-part index, a grow body-part index, an original synthesized depth value, a snap depth value, and an observed depth value for that pixel address.

10. The method of claim 3, where the pixel case is selected from a set of defined pixel cases including a refine-z pixel case, a magnetism pixel case, a push pixel case, a pull pixel case, a self-occluding push pixel case, and a self-occluding pull pixel case.

11. The method of claim 1, wherein adjusting the machine-readable model includes computing a force vector for each synthesized pixel of the plurality of synthesized pixels based on a pixel case for that synthesized pixel, and mapping the force vector to one or more force-receiving locations of the machine-readable model.

12. The method of claim 11, where the force vector mapped to the force-receiving location is one of a plurality of force vectors mapped to that force-receiving location, each such force vector corresponding to a synthesized pixel and classified pixel case, and each such force vector contributing to adjustment of the machine-readable model representing the human subject into the adjusted pose.

13. A data-holding device storing instructions executable by a logic subsystem, the instructions comprising:

instructions to obtain an observed depth image including a human subject from a depth camera;

instructions to generate a machine-readable model representing the human subject from the observed depth image;

instructions to obtain a synthesized depth image of the machine-readable model representing the human subject, the synthesized depth image including a two-dimensional matrix of synthesized pixels, each synthesized pixel in the two-dimensional matrix of synthesized pixels including a synthesized depth value and having a pixel address corresponding to a pixel address of a respective pixel of the observed depth image;

instructions to determine, for each synthesized pixel, a depth difference between a depth of that synthesized pixel and a depth of a corresponding pixel from a subsequent observed depth image; and

instructions to adjust the machine-readable model representing the human subject into an adjusted pose based on the depth difference and different pixel cases for different ones of a plurality of synthesized pixels.

14. The data-holding device of claim 13, where the synthesized depth image is obtained, at least in part, by rasterizing the machine-readable model representing the human subject, and wherein the different pixel cases for different ones of the two-dimensional matrix of synthesized pixels are based at least in part on a comparison between respective synthesized depth values of the synthesized depth image and observed depth values of the subsequent observed depth image.

15. The data-holding device of claim 13, where the machine-readable model is adjusted by computing a force vector for each synthesized pixel of the plurality of synthesized pixels based on a pixel case for that synthesized pixel, and mapping a force vector to one or more force-receiving locations of the machine-readable model.

16. The data-holding device of claim 15, where the force vector mapped to the force-receiving location is one of a plurality of force vectors mapped to that force-receiving location, each such force vector corresponding to a synthesized pixel and classified pixel case, and each such force vector contributing to adjustment of the machine-readable model representing the human subject into the adjusted pose.

17. A method of tracking a human subject, the method comprising:

obtaining an observed depth image including the human subject from a depth camera;

generating a machine-readable model representing the human subject from the observed depth image;

deriving a synthesized depth image from the machine-readable model representing the human subject, the synthesized depth image including a two-dimensional matrix of synthesized pixels, each synthesized pixel in the two-dimensional matrix of synthesized pixels including a depth value and having a same pixel address as a corresponding pixel of the observed depth image;

determining, for each synthesized pixel, a depth difference between a depth of that synthesized pixel and a depth of a corresponding pixel from a subsequent observed depth image; and

adjusting the machine-readable model representing the human subject into an adjusted pose based on the depth difference.

18. The method of claim 17, wherein the machine-readable model includes a plurality of portions, each portion associated with a part index corresponding to a part of the human subject, and wherein adjusting the machine-readable model into the adjusted pose based on the depth difference is further based on, for each portion of the machine-readable model, a respective part index associated with that portion of the machine-readable model.

19. The method of claim 18, where the part index is a body-part index corresponding to a part of the human subject, and wherein at least one portion of the machine-readable model having a first body-part index is adjusted differently than another portion of the machine-readable model having a second body-part index because of a difference between the first body-part index and the second body-part index.

20. The method of claim 18, where each portion has an original body-part index, a snap body-part index, and a grow body-part index;

and where each portion of the machine-readable model is classified with a pixel case based on, for each of the one or more synthesized pixels, the original body-part index, the snap body-part index, the grow body-part index, and the depth difference at that pixel address, and where the pixel case is selected from a set of defined pixel cases including a refine-z pixel case, a magnetism pixel case, a push pixel case, a pull pixel case, a null pixel case, a self-occluding push pixel case, and a self-occluding pull pixel case.