Human tracking system

• US 9,522,328 B2
• Filed: 09/04/2014
• Issued: 12/20/2016
• Est. Priority Date: 10/07/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Leyvand, Tommer, Lee, Johnny Chung, Stachniak, Szymon, Peeper, Craig, Liu, Shao
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US14/477,713
• Publication Number:
  US 20140375557A1
• Time in Patent Office: 838 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/34 Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

An image such as a depth image of a scene may be received, observed, or captured by a device. A grid of voxels may then be generated based on the depth image such that the depth image may be downsampled. A background included in the grid of voxels may also be removed to isolate one or more voxels associated with a foreground object such as a human target. A location or position of one or more extremities of the isolated human target may be determined and a model may be adjusted based on the location or position of the one or more extremities.

Claims

1. A method for tracking a person, comprising:

receiving a depth image that includes a target, the target corresponding to a person and comprising a head portion, a torso portion, arms portions, and legs portion;

generating voxels from the depth image, each voxel comprising an x-value, a y-value, and a depth value;

identifying a first plurality of voxels corresponding to the target;

calculating an average x-value, an average y-value, and an average depth value from the plurality of voxels corresponding to the target;

identifying a centroid of the target, the centroid having the average x-value, the average y-value, and the average depth value;

searching from the centroid and toward a head portion of the target for a first voxel that is not associated with the target;

searching from the centroid and away from the head portion of the target for a second voxel that is not associated with the target; and

determining a core volume of the target, the core volume based at least in part on the first voxel and the second voxel and excluding at least part of the legs portion of the target.

2. The method of claim 1, wherein generating voxels from the depth image comprises downsampling the received depth image using information included in the received depth image.

3. The method of claim 1, wherein generating voxels from the depth image comprises generating volume elements representing values of the information included in a sub-sampled grid of the received depth image.

4. The method of claim 1, wherein determining a core volume of the target comprises calculating a difference between the first pixel and the second pixel.

5. The method of claim 4, further comprising:

searching in a first direction away from the centroid in the x-direction for a third voxel that is not associated with the image; and

searching in a second direction away from the centroid in the x-direction for a fourth voxel that is not associated with the image.

6. The method of claim 5, wherein determining a core volume of the target comprises calculating a difference between third voxel and the fourth voxel.

7. The method of claim 6, wherein determining a core volume comprises defining a bounding box, the bounding box defined at least in part by the first voxel, the second voxel, the third voxel, and the fourth voxel.

8. The method of claim 1, wherein determining a core volume comprises determining a volume for a head and a torso.

9. A system for tracking a person, comprising:

a capture device that receives a depth image of a scene;

computing memory communicatively coupled with the capture device, the computing memory having stored thereon processor-executable instructions that cause the system to perform operations comprising;

receiving from the capture device a depth image that includes a target, the target corresponding to a person and comprising a head portion, a torso portion, arms portions, and legs portion;

generating voxels from the depth image, each voxel comprising an x-value, a y-value, and a depth value;

identifying a first plurality of voxels corresponding to the target;

calculating an average x-value, an average y-value, and an average depth value from the plurality of voxels corresponding to the target;

identifying a centroid of the target, the centroid having the average x-value, the average y-value, and the average depth value;

searching from the centroid and toward a head portion of the target for a first voxel that is not associated with the target;

searching from the centroid and away from the head portion of the target for a second voxel that is not associated with the target; and

determining a core volume of the target, the core volume based at least in part on the first voxel and the second voxel and excluding at least part of the legs portion of the target.

10. The system of claim 9, wherein determining a core volume of the target comprises calculating a difference between the first voxel and the second voxel.

11. The system of claim 10, further comprising:

searching in a first direction away from the centroid in the x-direction for a third voxel that is not associated with the image; and

searching in a second direction away from the centroid in the x-direction for a fourth voxel that is not associated with the image.

12. The system of claim 11, wherein determining a core volume of the target comprises calculating a difference between third voxel and the fourth voxel.

13. The system of claim 12, wherein determining a core volume comprises defining a bounding box, the bounding box defined at least in part by the first voxel, the second voxel, the third voxel, and the fourth voxel.

14. A computer-readable storage device bearing computer-readable instructions that cause a computing system to perform operations comprising:

receiving a depth image that includes a target, the target corresponding to a person and comprising a head portion, a torso portion, arms portions, and legs portion;

generating voxels from the depth image, each voxel comprising an x-value, a y-value, and a depth value;

identifying a first plurality of voxels corresponding to the target;

calculating an average x-value, an average y-value, and an average depth value from the plurality of voxels corresponding to the target;

identifying a centroid of the target, the centroid having the average x-value, the average y-value, and the average depth value;

searching from the centroid and toward a head portion of the target for a first voxel that is not associated with the target;

searching from the centroid and away from the head portion of the target for a second voxel that is not associated with the target; and

determining a core volume of the target, the core volume based at least in part on the first voxel and the second voxel and excluding at least part of the legs portion of the target.

15. The computer-readable storage device of claim 14, wherein determining a core volume of the target comprises calculating a difference between the first voxel and the second voxel.

16. The computer-readable storage device of claim 15, further comprising:

searching in a first direction away from the centroid in the x-direction for a third voxel that is not associated with the image; and

searching in a second direction away from the centroid in the x-direction for a fourth voxel that is not associated with the image.

17. The computer-readable storage device of claim 16, wherein determining a core volume of the target comprises calculating a difference between third voxel and the fourth voxel.