Device for identifying and tracking multiple humans over time

• US 9,656,162 B2
• Filed: 04/14/2014
• Issued: 05/23/2017
• Est. Priority Date: 05/29/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Finocchio, Mark J., Geiss, Ryan Michael, Tsunoda, Kudo, Polzin, R. Stephen, Kipman, Alex A., Perez, Kathryn Stone, Bennett, Darren Alexander
• Primary Examiner(s): Hu, Kang
• Assistant Examiner(s): Blaise, Malina D
• Application Number:
  US14/252,657
• Publication Number:
  US 20140228123A1
• Time in Patent Office: 1,135 Days
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/5375 for graphically or textually suggesting an action, e.g. by displaying an arrow indicating a turn in a driving game
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1043 being characterized by constructional details
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A system recognizes human beings in their natural environment, without special sensing devices attached to the subjects, uniquely identifies them and tracks them in three dimensional space. The resulting representation is presented directly to applications as a multi-point skeletal model delivered in real-time. The device efficiently tracks humans and their natural movements by understanding the natural mechanics and capabilities of the human muscular-skeletal system. The device also uniquely recognizes individuals in order to allow multiple people to interact with the system via natural movements of their limbs and body as well as voice commands/responses.

Claims

1. A system for analyzing a user in a capture area, comprising:

a processor; and

a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause the system at least to;

receive depth data for the capture area;

receive color data for the capture area;

determine a skeletal model of the user in the capture area based on the depth data, the user having a first body part and a second body part, the skeletal model comprising a representation of the first body part and a representation of the second body part of the user; and

determine an orientation of the first body part of the user relative to the second body part of the user based on the color data.

2. The system of claim 1, wherein the instructions that, when executed on the processor, cause the system at least to determine the orientation of the first body part of the user relative to the second body part of the user, further cause the system at least to:

determine an axial roll angle of the first body part of the user relative to the second body part of the user; and

constrain a range of motion of the representation of the first body part relative to the representation of the second body part in the skeletal model based on the axial roll angle.

3. The system of claim 2, wherein the memory further bears instructions that, when executed on the processor, cause the system at least to:

receive second color data for the capture area;

determine a second orientation of the first body part of the user relative to the second body part of the user based on the second color data;

determine that the second orientation of the first body part of the user relative to the second body part of the user violates the constrained range of motion; and

modify an orientation of the representation of the first body part of the user relative to the representation of the second body part of the user in the skeletal model based on the constrained range of motion.

4. The system of claim 1, wherein the instructions that, when executed on the processor, cause the system at least to determine the orientation of the first body part of the user relative to the second body part of the user based on the color data, further cause the system at least to:

determine that the orientation of the user comprises the user facing toward a capture device in response to determining that a face of the user is present in the color data.

5. The system of claim 1, wherein the memory further bears instructions that, when executed on the processor, cause the system at least to:

determine a location of the user in the capture area based in on the depth data.

6. The system of claim 5, wherein the memory further bears instructions that, when executed on the processor, cause the system at least to:

track the user over time based on known mechanics of a human muscular-skeletal system.

7. The system of claim 1, wherein the memory further bears instructions that, when executed on the processor, cause the system at least to:

track movement of the user over time based on the skeletal model.

8. The system of claim 7, wherein when the memory further bears instructions that, when executed on the processor, cause the system at least to:

determine that the system is unable to track movement of the user from the depth data without the color data; and

track movement of the user based on the depth data supplemented with the color data.

9. The system of claim 1, wherein the instructions that, when executed on the processor, cause the system at least to determine the orientation of the first body part of the user relative to the second body part of the user based on the color data, further cause the system at least to:

determine that the orientation of the user comprises the user facing away from a capture device in response to determining that a threshold amount of hair color of the user or less than a threshold amount of facial features of the user is present in the color data.

10. The system of claim 1, wherein the color data comprises black and white or grayscale colors.

11. A method for tracking a human user in a capture area, comprising:

receiving depth data for the capture area;

receiving color data for the capture area;

determining a skeletal model of the user in the capture area based on the depth data, the skeletal model comprising a first body part and a second body part of the user; and

determining an orientation of the first body part of the user relative to the second body part of the user based on the color data.

12. The method of claim 11, wherein determining the orientation of the first body part of the user relative to the second body part of the user further comprises:

determining an axial roll angle of the first body part of the user relative to the second body part of the user; and

constraining a range of motion of a representation of the first body part relative to a representation of the second body part in the skeletal model based on the axial roll angle.

13. The method of claim 12, further comprising:

receiving second color data for the capture area;

determining a second orientation of the first body part of the user relative to the second body part of the user based on the second color data;

determining that the second orientation of the first body part of the user relative to the second body part of the user violates the constrained range of motion; and

modifying an orientation of the representation of the first body part of the user relative to the representation of the second body part of the user in the skeletal model based on the constrained range of motion.

14. The method of claim 11, further comprising:

receiving sound data from at least one microphone; and

recognizing and tracking comprises recognizing and tracking movement of the user in the capture area over time based on a combination of at least two of the depth data, the color data and the sound data.

15. The method of claim 11, further comprising:

tracking movement of the user over time based on known mechanics of a human muscular-skeletal system.

16. The method of claim 11, further comprising:

determining, by one or more computers, that the one or more computers are unable to track movement of the user from the depth data without the color data; and

tracking, by the one or more computers, movement of the user based on the depth data supplemented with the color data.

17. The method of claim 11, wherein the first body part and the second body part are both directly connected at a joint.

18. A computer-readable storage device bearing computer-executable instructions that, when executed by a computer, causing the computer to perform operations comprising:

receiving depth data for a user in a capture area;

receiving color data for the capture area;

determining a skeletal model of the user in the capture area based on the depth data, the skeletal model comprising a first body part and a second body part of the user; and

determining an orientation of the first body part of the user relative to the second body part of the user based on the color data.

19. The computer-readable storage device of claim 18, further bearing computer-executable instructions that, when executed by the computer, causing the computer to perform operations comprising:

determining a constraint of where a representation of the first body part may be located based on a rotational orientation of the first body part relative to the second body part and the location of the second body part;

determining that the location of the first body part violates the constraint; and

in response to determining that the location of the first body part violates the constraint, adjusting a position of a representation of the first body part in the skeletal model to a location that does not violate the constraint.

20. The computer-readable storage device of claim 18, further bearing computer-executable instructions that, when executed by the computer, causing the computer to perform operations comprising:

determining a rotational orientation of the first body part relative to the second body part based on the color data, the rotational orientation indicating how the first body part is rotated at its location in the capture area.