Human tracking system

• US 8,970,487 B2
• Filed: 10/21/2013
• Issued: 03/03/2015
• Est. Priority Date: 10/07/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Leyvand, Tommer, Lee, Johnny, Peeper, Craig, Stachniak, Szymon, Liu, Shao
• Primary Examiner(s): SHENG, TOM V
• Application Number:
  US14/059,258
• Publication Number:
  US 20140044309A1
• Time in Patent Office: 498 Days
• Field of Search: 348/77, 382/103, 715/700, 715/863, 715/864, 715/865, 715/866, 345/156
• US Class Current: 345/156
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/34 Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

An image such as a depth image of a scene may be received, observed, or captured by a device. A grid of voxels may then be generated based on the depth image such that the depth image may be downsampled. A background included in the grid of voxels may also be removed to isolate one or more voxels associated with a foreground object such as a human target. A location or position of one or more extremities of the isolated human target may be determined and a model may be adjusted based on the location or position of the one or more extremities.

Claims

1. A computer-implemented method for tracking a physical target comprising:

receiving a depth image that captures the physical target in a physical space;

determining a first point based at least in part on an offset from a known point that corresponds to the physical target;

determining a second point that corresponds to an average location of a limb of the physical target;

determining a direction to search based on a line formed between the first point and the second point; and

determining a third point of the limb based on the direction to search and depth values of the depth image.

2. The method of claim 1, wherein the limb of the physical target is an arm of the physical target, and wherein the third point corresponds to a hand of the physical target.

3. The method of claim 1, wherein the limb of the physical target is an leg of the physical target, and wherein the third point corresponds to a foot of the physical target.

4. The method of claim 1, wherein the direction to search corresponds to moving along the line away from a torso of the physical target.

5. The method of claim 1, wherein determining the third point of the limb based on the direction to search and depth values of the depth image comprises:

determining the third point of the limb based on a pixel of the depth image that has a valid depth value and that is the greatest distance away from the first point among pixels in the direction to search that have valid depth values.

6. The method of claim 1, wherein determining the third point of the limb based on the direction to search and depth values of the depth image comprises:

determining the third point of the limb based on a pixel of the depth image that has a valid x-value and that is the greatest distance away from the first point among pixels in the direction to search that have valid depth values.

7. The method of claim 1, wherein determining the third point of the limb based on the direction to search and depth values of the depth image comprises:

determining the third point of the limb based on a pixel of the depth image that has a valid y-value and that is the greatest distance away from the first point among pixels in the direction to search that have valid depth values.

8. The method of claim 1, wherein determining the first point based at least in part on the offset from the known point that corresponds to the physical target comprises:

determining the offset based on an extending an x-value of the known point in an x-direction away from a torso of the physical target.

9. The method of claim 1, wherein determining the first point based at least in part on the offset from the known point that corresponds to the physical target comprises:

determining the offset based on an extending a z-value of the known point in a z-direction away from a torso of the physical target.

10. The method of claim 1, wherein determining the first point based at least in part on the offset from the known point that corresponds to the physical target comprises:

determining a Y-value of the first point based on a difference between a Y-value of a head or hips of the physical target and a Y-value of the average location of the limb of the physical target.

11. A system, comprising:

a memory bearing instructions that, upon execution by a processor, cause the system at least to;

receive a depth image that captures a physical target in a physical space;

determine a first point based at least in part on an offset from a known point that corresponds to the physical target;

determine a second point that corresponds to an average location of a limb of the physical target;

determine a direction to search based on a line formed between the first point and the second point; and

determine a third point of the limb based on the direction to search and depth values of the depth image.

12. The system of claim 11, wherein the limb of the physical target is an arm of the physical target, and wherein the third point corresponds to a hand of the physical target.

13. The system of claim 11, wherein the limb of the physical target is an leg of the physical target, and wherein the third point corresponds to a foot of the physical target.

14. The system of claim 11, wherein the direction to search corresponds to moving along the line away from a torso of the physical target.

15. The system of claim 11, wherein the instructions that, upon execution by the processor, cause the system at least to determine the first point that corresponds to a point where the limb of the physical target meets the torso of the physical target further cause the system at least to:

modify an X value or a Y value of a head, hip, or shoulder based on the X value or the Y value of the head, hip, or shoulder to determine the first point.

16. A computer-readable storage device bearing computer-executable instructions that, when executed on a computer, cause the computer to perform operations comprising:

receiving a depth image that captures a physical target in a physical space;

determining a first point based at least in part on an offset from a known point that corresponds to the physical target;

determining a second point that corresponds to an average location of a limb of the physical target;

determining a direction to search based on a line formed between the first point and the second point; and

determining a third point of the limb based on the direction to search and depth values of the depth image.

17. The computer-readable storage device of claim 16, wherein determining the third point of the limb based on the direction to search and depth values of the depth image comprises:

determining the third point of the limb based on a pixel of the depth image that has a valid depth value and that is the greatest distance away from the first point among pixels in the direction to search that have valid depth values.

18. The computer-readable storage device of claim 16, wherein determining the third point of the limb based on the direction to search and depth values of the depth image comprises:

determining the third point of the limb based on a pixel of the depth image that has a valid x-value and that is the greatest distance away from the first point among pixels in the direction to search that have valid depth values.

19. The computer-readable storage device of claim 16, wherein determining the third point of the limb based on the direction to search and depth values of the depth image comprises:

determining the third point of the limb based on a pixel of the depth image that has a valid y-value and that is the greatest distance away from the first point among pixels in the direction to search that have valid depth values.

20. The computer-readable storage device of claim 16, wherein determining the first point that corresponds to a point where the limb of the physical target meets the torso of the physical target comprises:

modifying an x-value or a y-value of a head, hip, or shoulder based on the x-value or the y-value of the head, hip, or shoulder to determine the first point.