Device for identifying and tracking multiple humans over time

• US 8,744,121 B2
• Filed: 05/29/2009
• Issued: 06/03/2014
• Est. Priority Date: 05/29/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Polzin, R. Stephen, Kipman, Alex A., Finocchio, Mark J., Geiss, Ryan Michael, Perez, Kathryn Stone, Tsunoda, Kudo, Bennett, Darren Alexander
• Primary Examiner(s): Akhavannik, Hadi
• Assistant Examiner(s): RASHIDIAN, MOHAMMAD M
• Application Number:
  US12/475,308
• Publication Number:
  US 20100303289A1
• Time in Patent Office: 1,831 Days
• Field of Search: 382/100, 382/103, 382/104, 382/106, 382/293, 382/159, 382170-173, 382/181, 382/190, 382/195, 382/203, 382209-210, 382/226, 382/228, 382/241, 382/266, 382/276, 382/277, 382/278, 382/285, 348/135, 348/169, 348/154, 348/19, 348/E7.085, 348/208.14, 345/156, 345/158, 345/143, 345418-419, 345473-474, 345/629, 463/32, 463/36, 463/31, 463/37, 463/42, 463/5, 463/8, 704/231, 704/E15.005, 704/E15.004, 356/614, 356/616, 701/24, 715/700, 715/702, 715/863
• US Class Current: 382/103
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/5375 for graphically or textually suggesting an action, e.g. by displaying an arrow indicating a turn in a driving game
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1043 being characterized by constructional details
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A system recognizes human beings in their natural environment, without special sensing devices attached to the subjects, uniquely identifies them and tracks them in three dimensional space. The resulting representation is presented directly to applications as a multi-point skeletal model delivered in real-time. The device efficiently tracks humans and their natural movements by understanding the natural mechanics and capabilities of the human muscular-skeletal system. The device also uniquely recognizes individuals in order to allow multiple people to interact with the system via natural movements of their limbs and body as well as voice commands/responses.

Claims

1. A system for analyzing at least one user in a capture area, comprising:

a processor; and

a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause the system to at least;

receive depth data for the capture area;

receive color data for the capture area;

determine a location of the user in the capture area, including a location of a first body part and a second body part of the user, based on the depth data; and

determine an orientation of the first body part of the user relative to the second body part of the user at the location in the capture area based on the color data.

2. The system of claim 1 wherein the orientation of the user comprises the user facing toward a capture device that captures the color data, or the user facing away from the capture device.

3. The system of claim 1 wherein the instructions that, when executed on the processor, cause the system to at least determine the orientation of the user at the location in the capture area based on the color data, further cause the system to at least:

determine that the orientation of the user comprises the user facing toward a capture device in response to determining that the user'"'"'s face is present in the color data.

4. The system of claim 1, wherein the memory further bears instructions that, when executed on the processor, cause the system to at least:

generate a skeletal model of the user based at least in part on the depth data.

5. The system of claim 4, wherein the a memory further bears instructions that, when executed on the processor, cause the system to at least:

track movement of the user over time based on the skeletal model.

6. The system of claim 5, wherein when the memory further bears instructions that, when executed on the processor, cause the system to at least:

determine that the system is unable to track movement of the user from the depth data without the color data; and

track movement of the user based on the depth data supplemented with the color data.

7. The system of claim 4, wherein the memory further bears instructions that, when executed on the processor, cause the system to at least:

track the user over time based on known mechanics of the human muscular-skeletal system.

8. The system of claim 1 wherein the instructions that, when executed on the processor, cause the system to at least determine the orientation of the user at the location in the capture area based on the color data, further cause the system to at least:

determine that the orientation of the user comprises the user facing away from a capture device in response to determining that a threshold amount of the user'"'"'s hair color or less than a threshold amount of the user'"'"'s facial features is present in the color data.

9. The system of claim 1, wherein the color data comprises black and white or grayscale colors.

10. A method for tracking a human user in a capture area, comprising:

receiving depth data for the capture area;

receiving color data for the capture area;

determining a location of the user in the capture area, including a location of a first body part and a second body part of the user, based on the depth data; and

determining an orientation of the first body part of the user relative to the second body part of the user at the location in the capture area based on the color data.

11. The method of claim 10, wherein the orientation of the user comprises the user facing toward a capture device that captures the color data, or the user facing away from the capture device.

12. The method of claim 10, further comprising receiving sound data from at least one microphone, and recognizing and tracking comprises recognizing and tracking movement of the user in the capture area over time based on a combination of at least two of the depth data, the color data and the sound data.

13. The method of claim 10, further comprising generating a skeletal model of the user from the depth data, and using the skeletal model to track movement of the user over time based on the known mechanics of the human muscular-skeletal system.

14. The method of claim 10, further comprising:

determining that the system is unable to track movement of the user from the depth data without the color data; and

tracking movement of the user based on the depth data supplemented with the color data.

15. The method of claim 14, further comprising generating a motion capture file of the movements of the user in real-time based on the tracked-movements of the user.

16. The method of claim 10, further comprising using sound information provided by a microphone to distinguish between two users detected in the capture area based on recognition of each user'"'"'s voice.

18. The method of claim 10, further comprising:

determining a constraint of where the first body part may be located based on a rotational orientation of the first body part relative to the second body part and the location of the second body part;

determining that the location of the first body part violates the constraint; and

in response to determining that the location of the first body part violates the constraint, adjusting the position of the first body part to a location that does not violate the constraint.

19. The method of claim 10, further comprising:

determining a rotational orientation of the first body part relative to the second body part based on the color data, the rotational orientation indicating how the first body part is rotated at its location in the capture area.

20. The method of claim 10, wherein the first body part and the second body part are both directly connected at a joint.

17. A computer-readable storage medium on which is stored computer-executable instructions for tracking a human user in a capture area, the instructions, when executed by a processor, causing the processor to perform operations comprising:

receive depth data for the capture area;

receive color data for the capture area;

determining a location of the user in the capture area, including a location of a first body part and a second body part of the user, based on the depth data; and

determining an orientation of the first body part of the user relative to the second body part of the user at the location in the capture area based on the color data.