Methods and systems for determining and tracking extremities of a target

• US 9,659,377 B2
• Filed: 12/15/2014
• Issued: 05/23/2017
• Est. Priority Date: 10/07/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Leyvand, Tommer, Peeper, Craig, Liu, Shao, Stachniak, Szymon Piotr, Lee, Johnny Chung
• Primary Examiner(s): Sheng, Tom
• Application Number:
  US14/570,005
• Publication Number:
  US 20150098619A1
• Time in Patent Office: 890 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2210/12 Bounding box
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

An image such as a depth image of a scene may be received, observed, or captured by a device. A grid of voxels may then be generated based on the depth image such that the depth image may be downsampled. A background included in the grid of voxels may also be removed to isolate one or more voxels associated with a foreground object such as a human target. A location or position of one or more extremities of the isolated human target may then be determined.

Claims

1. A method for determining an orientation of one or more extremities comprising:

receiving a depth image;

generating a grid of voxels based on the depth image;

removing a background included in the grid of voxels to isolate one or more voxels associated with a human target;

defining a head-to-center vector based on locations or positions of a head and a center of the human target;

creating an extremity volume box based on a displacement along the head-to-center vector; and

determining the orientation of the one or more extremities based on a line fit of depth values within the extremity volume box.

2. The method of claim 1, wherein the one or more extremities comprises at least one of the following:

a head, a centroid, a shoulder, a hip, a leg, an arm, a hand, an elbow, a knee, and a foot.

3. The method of claim 1, further comprising estimating a center of the human target, wherein estimating the center of the human target comprises calculating an average position of the voxels in the grid associated with the human target.

4. The method of claim 1, further comprising determining a core volume of the human target.

5. The method of claim 1, further comprising:

creating a torso volume;

identifying voxels outside of the torso volume; and

labeling voxels outside the torso volume as being associated with the one or more extremities.

6. The method of claim 1, further comprising processing the one or more extremities.

7. The method of claim 1, further comprising:

determining a location or position of one or more extremities of the human target; and

refining the location or position of the one or more extremities based on pixels associated with the one or more extremities in the depth image.

8. A system for determining an orientation of one or more extremities comprising:

a capture device, wherein the capture device comprises a camera component to receive a depth image; and

a computing device operable to communicate with the capture device, wherein the computing device comprises a processor operable to perform operations comprising;

generating a grid of voxels based on the depth image;

removing a background included in the grid of voxels to isolate one or more voxels associated with a human target;

defining a head-to-center vector based on locations or positions of a head and a center of the human target;

creating an extremity volume box based on a displacement along the head-to-center vector; and

determining the orientation of the one or more extremities based on a line fit of depth values within the extremity volume box.

9. The system of claim 8, wherein the one or more extremities comprises at least one of the following:

a head, a centroid, a shoulder, a hip, a leg, an arm, a hand, an elbow, a knee, and a foot.

10. The system of claim 8, wherein the operations further comprise estimating a center of the human target, wherein estimating the center of the human target comprises calculating an average position of the voxels in the grid associated with the human target.

11. The system of claim 8, wherein the operations further comprise determining a core volume of the human target.

12. The system of claim 8, wherein the operations further comprise:

creating a torso volume;

identifying voxels outside of the torso volume; and

labeling voxels outside the torso volume as being associated with the one or more extremities.

13. The system of claim 8, wherein the operations further comprise processing the one or more extremities.

14. The system of claim 8, wherein the operations further comprise:

determining a location or position of one or more extremities of the human target; and

refining the location or position of the one or more extremities based on pixels associated with the one or more extremities in the depth image.

15. A computer-readable storage device having stored thereon computer-executable instructions, the computer-executable instructions comprising instructions for:

receiving a depth image;

generating a grid of voxels based on the depth image;

removing a background included in the grid of voxels to isolate one or more voxels associated with a human target;

defining a head-to-center vector based on locations or positions of a head and a center of the human target;

creating an extremity volume box based on a displacement along the head-to-center vector; and

determining an orientation of the one or more extremities based on a line fit of depth values within the extremity volume box.

16. The computer-readable storage device of claim 15, wherein the one or more extremities comprises at least one of the following:

a head, a centroid, a shoulder, a hip, a leg, an arm, a hand, an elbow, a knee, and a foot.

17. The computer-readable storage device of claim 15, wherein the computer-executable instructions further comprise instructions for estimating a center of the human target, wherein estimating the center of the human target comprises calculating an average position of the voxels in the grid associated with the human target.

18. The computer-readable storage device of claim 15, wherein the computer-executable instructions further comprise instructions for determining a core volume of the human target.

19. The computer-readable storage device of claim 15, wherein the computer-executable instructions further comprise instructions for:

creating a torso volume;

identifying voxels outside of the torso volume; and

labeling voxels outside the torso volume as being associated with the one or more extremities.

20. The computer-readable storage device of claim 15, wherein the computer-executable instructions further comprise instructions for processing the one or more extremities.