DETECTION OF BODY AND PROPS

• US 20110085705A1
• Filed: 12/20/2010
• Published: 04/14/2011
• Est. Priority Date: 05/01/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Criminisi, Antonio, Molyneaux, David, Hilliges, Otmar, Cook, Mat, Shotton, Jamie, Winn, John, Izadi, Shahram
• Granted Patent: US 8,660,303 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A system and method for detecting and tracking targets including body parts and props is described. In one aspect, the disclosed technology acquires one or more depth images, generates one or more classification maps associated with one or more body parts and one or more props, tracks the one or more body parts using a skeletal tracking system, tracks the one or more props using a prop tracking system, and reports metrics regarding the one or more body parts and the one or more props. In some embodiments, feedback may occur between the skeletal tracking system and the prop tracking system.

Claims

1. A method for detecting one or more targets, comprising:

acquiring one or more depth images from one or more depth sensing image sensors, a first depth image of the one or more depth images includes a plurality of pixels;

generating a classification map associated with the first depth image, the step of generating includes assigning to one or more pixels of the plurality of pixels a probability that the one or more pixels is associated with a particular target of the one or more targets, the one or more targets include a first target representing at least a portion of a first body part and a second target representing at least a portion of a first prop;

fitting at least one computer model of one or more computer models onto at least a portion of the classification map, the one or more computer models include at least a first model of the first target and at least a second model of the second target;

performing skeletal tracking on the first target;

performing prop tracking on the second target; and

reporting a first position of the first target and a second position of the second target.

2. The method of claim 1, further comprising:

acquiring one or more color images from one or more color sensing image sensors, the step of performing prop tracking includes using color information from the one or more color images to help track the second target.

3. The method of claim 1, wherein:

the step of generating is performed using a machine learning technique, the machine learning technique uses a training set of segmented images, the training set of segmented images includes one or more retrofitted images.

4. The method of claim 1, wherein:

the step of performing skeletal tracking includes receiving position information regarding the second target, the position information is taken into consideration when determining the first position of the first target.

5. The method of claim 1, wherein:

the step of performing prop tracking includes receiving position information regarding the first target, the position information is taken into consideration when determining the second position of the second target.

6. The method of claim 1, further comprising:

switching a game mode based on the step of reporting.

7. The method of claim 1, further comprising:

receiving orientation information from the first prop, the step of performing prop tracking uses the orientation information to help track the first prop.

8. The method of claim 1, wherein:

the second model includes one or more negative spaces.

9. An electronic device for detecting one or more targets, comprising:

a depth sensing camera, the depth sensing camera captures a first depth image; and

one or more processors, the one or more processors in communication with the depth sensing camera, the one or more processors generate one or more classification maps associated with one or more body parts and one or more props, the one or more processors track the one or more body parts using skeletal tracking, the one or more processors track the one or more props using prop tracking, the one or more processors report metrics regarding the one or more body parts and the one or more props.

10. The electronic device of claim 9, wherein:

the one or more classification maps are generated using a machine learning technique, the machine learning technique uses a training set of segmented images, the training set of segmented images includes one or more retrofitted images.

11. The electronic device of claim 9, wherein:

prop tracking includes using skeletal tracking information when scoring prop hypotheses.

12. The electronic device of claim 9, wherein:

skeletal tracking includes using prop tracking information when scoring skeletal hypotheses.

13. The electronic device of claim 9, further comprising:

a color sensing camera, the color sensing camera captures a first color image, the one or more processors receive at least a portion of the first color image, the one or more processors use color information from the at least a portion of the first color image to distinguish targets that are similar in size and shape.

14. The electronic device of claim 9, wherein:

at least one prop of the one or more props is an active prop; and

prop tracking is performed in parallel with skeletal tracking.

15. One or more storage devices containing processor readable code for programming one or more processors to perform a method comprising the steps of:

acquiring one or more depth images from one or more depth sensing image sensors, a first depth image of the one or more depth images includes a plurality of pixels;

generating a classification map associated with the first depth image, the step of generating includes assigning to one or more pixels of the plurality of pixels a probability that the one or more pixels is associated with a particular target of one or more targets, the one or more targets include a first target representing at least a portion of a first body part and a second target representing at least a portion of a first prop;

performing skeletal tracking on the first target, the step of performing skeletal tracking includes receiving position information regarding the second target, the position information is taken into consideration when determining a first position of the first target;

performing prop tracking on the second target, the step of performing prop tracking includes receiving position information regarding the first target, the position information is taken into consideration when determining a second position of the second target; and

reporting the first position of the first target and the second position of the second target.

16. The one or more storage devices of claim 15, wherein:

the step of generating is performed using a machine learning technique, the machine learning technique uses a training set of segmented images, the training set of segmented images includes one or more retrofitted images.

17. The one or more storage devices of claim 16, wherein:

at least one of the one or more retrofitted images includes a plurality of props, the plurality of props includes the first prop.

18. The one or more storage devices of claim 17, wherein:

the first prop is an active prop.

19. The one or more storage devices of claim 18, further comprising:

fitting at least one computer model of one or more computer models onto at least a portion of the classification map, the one or more computer models include at least a first model of the first target and at least a second model of the second target, the second model includes one or more negative spaces.

20. The one or more storage devices of claim 19, wherein:

the step of reporting causes the one or more processors to select a new game mode.