Human tracking system

• US 8,564,534 B2
• Filed: 10/07/2009
• Issued: 10/22/2013
• Est. Priority Date: 10/07/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Leyvand, Tommer, Lee, Johnny, Peeper, Craig, Stachniak, Szymon, Liu, Shao
• Primary Examiner(s): SHENG, TOM V
• Application Number:
  US12/575,388
• Publication Number:
  US 20110080336A1
• Time in Patent Office: 1,476 Days
• Field of Search: 345/156, 715/700, 715863-866, 382/103
• US Class Current: 345/156
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/34 Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

An image such as a depth image of a scene may be received, observed, or captured by a device. A grid of voxels may then be generated based on the depth image such that the depth image may be downsampled. A background included in the grid of voxels may also be removed to isolate one or more voxels associated with a foreground object such as a human target. A location or position of one or more extremities of the isolated human target may be determined and a model may be adjusted based on the location or position of the one or more extremities.

Claims

1. A method for tracking a user, the method comprising:

receiving a depth image that includes a target;

scanning a first orientation to a first side of the target;

scanning the first orientation to a second side of the target;

identifying a first point of reference on the target from the scan of the first orientation;

scanning a second orientation to a third side of the target, based on the first point of reference;

scanning the second orientation to a fourth side of the target;

identifying a second point of reference on the target from the scan of the second orientation;

generating a bounding box based on the reference point of the scan of the first orientation and the reference point of the scan of the second orientation; and

using the identified bounding box to adjust a model of the target.

2. The method of claim 1, wherein the model comprises a skeletal model having joints and bones.

3. The method of claim 1, wherein using the identified bounding box to adjust the model of the target comprises:

assigning a location or position of an extremity of the model to fit within the bounding box.

4. The method of claim 3, further comprising:

determining dimensions of the extremity one or more extremities, and adjusting the joints and the bones a joint or a bone of the model based on the determined dimensions.

5. The method of claim 1, wherein the one or more extremities using the identified bounding box to adjust the model of the target comprises:

adjusting a position of a head, a centroid, a shoulder, a hip, a leg, an arm, a hand, an elbow, a knee, or a foot of the model.

6. The method of claim 1, further comprising:

processing the adjusted model.

7. The method of claim 1, wherein the first orientation comprises one of an orientation of the target'"'"'s hips and an orientation of the target'"'"'s shoulders, and wherein the second orientation comprises the other of the orientation of the target'"'"'s hips and the orientation of the target'"'"'s shoulders.

8. The method of claim 1, wherein generating the bounding box based on the reference point of the scan of the first orientation and the reference point of the scan of the second orientation comprises:

generating the bounding box, the bounding box encompassing the reference point of the scan of the first orientation and the reference point of the scan of the second orientation.

9. The method of claim 1, wherein identifying the first point of reference on the target from the scan of the first orientation comprises:

identifying a location of a hip or a shoulder of the target.

10. The method of claim 1, wherein scanning the first orientation to the first side of the target comprises:

scanning the first orientation from a centroid of the target to the first side of the target.

11. A computer-readable storage medium having stored thereon computer executable instructions for tracking a user, the computer executable instructions comprising instructions for:

receiving a depth image that includes a target;

scanning a first orientation to a first side of the target;

scanning the first orientation to a second side of the target;

identifying a first point of reference on the target from the scan of the first orientation;

scanning a second orientation to a third side of the target, based on the first point of reference;

scanning the second orientation to a fourth side of the target;

identifying a second point of reference on the target from the scan of the second orientation;

generating a bounding box based on the reference point of the scan of the first orientation and the reference point of the scan of the second orientation; and

using the identified bounding box to adjust a model of the target.

12. The computer-readable storage medium of claim 11, wherein the model comprises a skeletal model having joints and bones.

13. The computer-readable storage medium of claim 11, wherein using the identified bounding box to adjust the model of the target comprises:

assigning a location or position of an extremity of the model to fit within the bounding box.

14. The computer-readable storage medium of claim 11, further comprising:

instructions for determining dimensions of the extremity, and adjusting a joint or a bone of the model based on the determined dimensions.

15. The computer-readable storage medium of claim 11, wherein using the identified bounding box to adjust the model of the target comprises:

adjusting a position of a head, a centroid, a shoulder, a hip, a leg, an arm, a hand, an elbow, a knee, or a foot of the model.

16. The computer-readable storage medium of claim 11, further comprising:

instructions for processing the adjusted model.

17. A system for tracking a model associated with a user, the system comprising:

a processor; and

a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause the processor to at least;

receive a depth image that includes a target;

scan a first orientation to a first side of the target;

scan the first orientation to a second side of the target;

identify a first point of reference on the target from the scan of the first orientation;

scan a second orientation to a third side of the target, based on the first point of reference;

scan the second orientation to a fourth side of the target;

identify a second point of reference on the target from the scan of the second orientation;

generate a bounding box based on the reference point of the scan of the first orientation and the reference point of the scan of the second orientation; and

use the identified bounding box to adjust a model of the target.

18. The system of claim 17, wherein the model comprises a skeletal model having joints and bones.

19. The system of claim 18, wherein the instructions that, when executed on the processor, cause the system to at least use the identified bounding box to adjust a model of the target, further cause the system to at least:

assign a location or position of an extremity of the model to the corresponding joints of the model to fit within the bounding box.

20. The system of claim 17, wherein the instructions that, when executed on the processor, cause the system to at least use the identified bounding box to adjust a model of the target, further cause the system to at least:

adjust a position of a head, a centroid, a shoulder, a hip, a leg, an arm, a hand, an elbow, a knee, or a foot of the model.