Detection of body and props

• US 8,660,303 B2
• Filed: 12/20/2010
• Issued: 02/25/2014
• Est. Priority Date: 05/01/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Izadi, Shahram, Shotton, Jamie, Winn, John, Criminisi, Antonio, Hilliges, Otmar, Cook, Mat, Molyneaux, David
• Primary Examiner(s): Akhavannik, Hadi
• Assistant Examiner(s): RASHIDIAN, MOHAMMAD M
• Application Number:
  US12/972,837
• Publication Number:
  US 20110085705A1
• Time in Patent Office: 1,163 Days
• Field of Search: 382/100, 382103-106, 382/159, 382170-173, 382/181, 382/190, 382/195, 382/203, 382209-210, 382/226, 382/228, 382/241, 382/266, 382276-278, 382/285, 382/293, 345/156, 345/158, 345473-474, 345/143, 345418-419, 345/629, 463/32, 463/36, 463/31, 463/37, 463/42, 463/5, 463/8, 704/231, 704/E15.005, 704/E15.004, 356/614, 356/616, 701/24, 715/700, 715/702, 715/863, 348/135, 348/169, 348/154, 348/19, 348/E7.085
• US Class Current: 382/103
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A system and method for detecting and tracking targets including body parts and props is described. In one aspect, the disclosed technology acquires one or more depth images, generates one or more classification maps associated with one or more body parts and one or more props, tracks the one or more body parts using a skeletal tracking system, tracks the one or more props using a prop tracking system, and reports metrics regarding the one or more body parts and the one or more props. In some embodiments, feedback may occur between the skeletal tracking system and the prop tracking system.

Claims

1. A method for detecting one or more targets, comprising:

acquiring one or more depth images from one or more depth sensing image sensors, a first depth image of the one or more depth images includes a plurality of pixels;

generating a classification map associated with the first depth image, the generating includes assigning to one or more pixels of the plurality of pixels a probability that the one or more pixels is associated with a particular target of the one or more targets, the one or more targets include a first target representing at least a portion of a first body part and a second target representing at least a portion of a first prop;

fitting at least one computer model of one or more computer models onto at least a portion of the classification map, the one or more computer models include at least a first model of the first target and at least a second model of the second target;

performing skeletal tracking on the first target;

performing prop tracking on the second target; and

reporting a first position of the first target and a second position of the second target, the performing skeletal tracking includes receiving position information regarding the second target, the position information is taken into consideration when determining the first position of the first target.

2. The method of claim 1, wherein:

the first prop is an inanimate object; and

the performing skeletal tracking includes receiving prop tracking information associated with the first prop, the prop tracking information includes orientation information associated with the second target and the position information associated with the second target, the performing skeletal tracking includes determining the first position of the first target based on the prop tracking information.

3. The method of claim 1, wherein:

the generating a classification map is performed using a machine learning technique, the machine learning technique uses a training set of segmented images, the training set of segmented images includes one or more retrofitted images.

4. The method of claim 2, wherein:

the performing skeletal tracking includes scoring generated skeletal hypotheses based on the prop tracking information.

5. The method of claim 1, wherein:

the performing prop tracking includes receiving additional position information regarding the first target, the additional position information is taken into consideration when determining the second position of the second target.

6. The method of claim 1, further comprising:

switching a game mode based on the reporting, the first prop comprises a gaming prop.

7. The method of claim 1, further comprising:

receiving orientation information from the first prop, the performing prop tracking uses the orientation information to help track the first prop.

8. The method of claim 1, wherein:

the second model includes one or more negative spaces.

9. An electronic device for detecting one or more targets, comprising:

a depth sensing camera, the depth sensing camera captures a first depth image; and

one or more processors, the one or more processors in communication with the depth sensing camera, the one or more processors generate one or more classification maps associated with one or more body parts and one or more props, the one or more processors track the one or more body parts using skeletal tracking, the one or more processors track the one or more props using prop tracking, the one or more processors report metrics regarding the one or more body parts and the one or more props, the metrics include a first position of a first body part of the one or more body parts and a second position of a second prop of the one or more props, the skeletal tracking includes receiving position information associated with the second prop, the position information is taken into consideration when determining the first position of the first body part.

10. The electronic device of claim 9, wherein:

the one or more classification maps are generated using a machine learning technique, the machine learning technique uses a training set of segmented images, the training set of segmented images includes one or more retrofitted images.

11. The electronic device of claim 9, wherein:

the prop tracking includes using skeletal tracking information when scoring prop hypotheses.

12. The electronic device of claim 9, wherein:

the skeletal tracking includes using prop tracking information when scoring skeletal hypotheses, the prop tracking information includes orientation information associated with the second prop and the position information associated with the second prop, the second prop is an inanimate object.

13. The electronic device of claim 9, further comprising:

a color sensing camera, the color sensing camera captures a first color image, the one or more processors receive at least a portion of the first color image, the one or more processors use color information from the at least a portion of the first color image to distinguish targets of the one or more targets that are similar in size and shape.

14. The electronic device of claim 9, wherein:

at least one prop of the one or more props is an active prop; and

the prop tracking is performed in parallel with the skeletal tracking.

15. One or more storage devices containing processor readable code for programming one or more processors to perform a method comprising the steps of:

acquiring one or more depth images from one or more depth sensing image sensors, a first depth image of the one or more depth images includes a plurality of pixels;

generating a classification map associated with the first depth image, the generating includes assigning to one or more pixels of the plurality of pixels a probability that the one or more pixels is associated with a particular target of one or more targets, the one or more targets include a first target representing at least a portion of a first body part and a second target representing at least a portion of a first prop, the first prop is an inanimate object;

performing skeletal tracking on the first target, the performing skeletal tracking includes receiving position information regarding the second target, the position information is taken into consideration when determining a first position of the first target;

performing prop tracking on the second target, the performing prop tracking includes receiving additional position information regarding the first target, the additional position information is taken into consideration when determining a second position of the second target; and

reporting the first position of the first target and the second position of the second target.

16. The one or more storage devices of claim 15, wherein:

the generating a classification map is performed using a machine learning technique, the machine learning technique uses a training set of segmented images, the training set of segmented images includes one or more retrofitted images.

17. The one or more storage devices of claim 16, wherein:

at least one of the one or more retrofitted images includes a plurality of props, the plurality of props includes the first prop.

18. The one or more storage devices of claim 17, wherein:

the first prop is an active prop.

19. The one or more storage devices of claim 18, further comprising:

fitting at least one computer model of one or more computer models onto at least a portion of the classification map, the one or more computer models include at least a first model of the first target and at least a second model of the second target, the second model includes one or more negative spaces.

20. The one or more storage devices of claim 19, wherein:

the reporting causes the one or more processors to select a new game mode, the first prop comprises a gaming prop.