Visual target tracking using model fitting and exemplar

• US 7,974,443 B2
• Filed: 11/23/2010
• Issued: 07/05/2011
• Est. Priority Date: 01/30/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Marais, Charles Claudius, Kipman, Alex, Mathe, Zsolt, Lee, Johnny Chung, Finocchio, Mark, Geiss, Ryan M.
• Primary Examiner(s): Mehta; Bhavesh M
• Assistant Examiner(s): MOTSINGER, SEAN T
• Application Number:
  US12/953,069
• Publication Number:
  US 20110058709A1
• Time in Patent Office: 224 Days
• Field of Search: 382/103, 382/107, 382190-208, 463 36- 39, 345419-427
• US Class Current: 382/103
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A method of tracking a target includes receiving an observed depth image of the target from a source and analyzing the observed depth image with a prior-trained collection of known poses to find an exemplar pose that represents an observed pose of the target. The method further includes rasterizing a model of the target into a synthesized depth image having a rasterized pose and adjusting the rasterized pose of the model into a model-fitting pose based, at least in part, on differences between the observed depth image and the synthesized depth image. Either the exemplar pose or the model-fitting pose is then selected to represent the target.

Claims

1. A method of tracking a human target, the method comprising:

receiving an observed depth image of a scene from a source;

body scanning the scene to identify a human target in the scene;

removing non-target background information from the observed depth image;

applying to the observed depth image one or more decision trees trained from a collection of known poses to find an exemplar pose that represents an observed pose of the human target;

rasterizing a body model of the human target into a synthesized depth image having a rasterized pose;

adjusting the rasterized pose of the body model into a model-fitting pose based, at least in part, on differences between the observed depth image and the synthesized depth image;

comparing a confidence in the exemplar pose to a confidence in the model-fitting pose;

selecting the exemplar pose if the confidence in the exemplar pose is higher than or equal to the confidence in the model-fitting pose; and

selecting the model-fitting pose if the confidence in the model-fitting pose is higher than the confidence in the exemplar pose.

2. The method of claim 1, where a terminal node of a decision tree yields a best-guess of a body part for a pixel and a confidence that the best-guess is correct.

3. The method of claim 1, where removing non-target background information from the observed depth image includes removing depth image information outside of a sphere surrounding the target.

4. The method of claim 1, further comprising analyzing the observed depth image with a hand-identifying algorithm configured to identify hands on the human target;

and

increasing relative confidence of the exemplar pose if the exemplar pose more closely places hands in a same location as the hand-identifying algorithm; and

increasing relative confidence of the model-fitting pose if the model-fitting pose more closely places hands in a same location as the hand-identifying algorithm.

5. A method of tracking a target, the method comprising:

receiving an observed depth image of a target from a source;

analyzing the observed depth image with a prior-trained collection of known poses to find an exemplar pose that represents an observed pose of the target;

rasterizing a model of the target into a synthesized depth image having a rasterized pose;

adjusting the rasterized pose of the model into a model-fitting pose based, at least in part, on differences between the observed depth image and the synthesized depth image; and

selecting the exemplar pose or the model-fitting pose.

6. The method of claim 5, further comprising analyzing the observed depth image with a hand-identifying algorithm configured to identify hands on the target;

and

biasing selection of the exemplar pose or the model-fitting pose toward a pose that more closely places hands in a same location as the hand-identifying algorithm.

7. The method of claim 5, further comprising body scanning a scene of the observed depth image to identify the target.

8. The method of claim 5, further comprising removing non-target background information from the observed depth image.

9. The method of claim 8, where removing non-target background information from the observed depth image includes removing depth image information outside of a three-dimensional buffer surrounding the target.

10. The method of claim 8, where removing non-target background information from the observed depth image includes removing depth image information outside of a sphere surrounding the target.

11. The method of claim 5, where the source includes a depth camera.

12. The method of claim 5, where the source includes stereo cameras.

13. The method of claim 5, where analyzing the observed depth image with a prior-trained collection of known poses includes applying to the observed depth image one or more decision trees trained from the prior-trained collection of known poses.

14. The method of claim 13, where a terminal node of a decision tree yields a best-guess of a body part for a pixel and a confidence that the best-guess is correct.

15. The method of claim 14, further comprising locating each joint position of the exemplar pose based, at least in part, on the best-guess of the body part for each pixel.

16. The method of claim 15, further comprising assigning a confidence to each joint position based, at least in part, on individual confidences for each pixel.

17. The method of claim 5, further comprising assessing a confidence of the model-fitting pose based on a comparison of the model-fitting pose and the observed pose.

18. The method of claim 5, where selecting the exemplar pose or the model-fitting pose includes selecting the exemplar pose if a confidence in the exemplar pose is higher than or equal to a confidence in the model-fitting pose, and selecting the model-fitting pose if the confidence in the model-fitting pose is higher than the confidence in the exemplar pose.

19. The method of claim 5, where adjusting the rasterized pose of the model into the model-fitting pose includes applying one or more forces to force-receiving locations of the model and allowing the model to move responsive to such forces.

20. A computing system, comprising:

a source configured to capture depth information;

a logic subsystem operatively connected to the source; and

a data-holding subsystem holding instructions executable by the logic subsystem to;

receive an observed depth image of a target from a source;

analyze the observed depth image with a prior-trained collection of known poses to find an exemplar pose that represents an observed pose of the target;

rasterize a model of the target into a synthesized depth image having a rasterized pose;

adjust the rasterized pose of the model into a model-fitting pose based, at least in part, on differences between the observed depth image and the synthesized depth image; and

select the exemplar pose or the model-fitting pose.