Systems and methods for tracking a model

• US 9,582,717 B2
• Filed: 10/27/2014
• Issued: 02/28/2017
• Est. Priority Date: 10/07/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lee, Johnny Chung, Leyvand, Tommer, Stachniak, Szymon Piotr, Peeper, Craig C.
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US14/524,277
• Publication Number:
  US 20150146923A1
• Time in Patent Office: 855 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: H04N 13/20 Image signal generators

Abstract

An image such as a depth image of a scene may be received, observed, or captured by a device. A grid of voxels may then be generated based on the depth image such that the depth image may be downsampled. A model may be adjusted based on a location or position of one or more extremities estimated or determined for a human target in the grid of voxels. The model may also be adjusted based on a default location or position of the model in a default pose such as a T-pose, a DaVinci pose, and/or a natural pose.

Claims

1. A method for tracking a user, comprising:

receiving a first depth image, the first depth image being captured by a depth camera;

based at least on identifying an estimated location of an extremity of a user in the first depth image, positioning a part of a model to correspond to the estimated location of the extremity in the first depth image; and

based at least on determining that a location of an extremity has not been estimated, adjusting a part of the model that is located close to a portion of the depth image to correspond to the portion of the depth image.

2. The method of claim 1, wherein adjusting the part of the model that is located close to the portion of the depth image to correspond to the portion of the depth image comprises adjusting the part of the model to a location of at least one voxel in the first depth image.

3. The method of claim 2, wherein adjusting the part of the model to a location of at least one voxel in the first depth image comprises adjusting the part of the model to a location defined by an x-value, y-value, and depth value in the first depth image.

4. The method of claim 1, wherein adjusting the part of the model that is located close to a portion of the depth image to correspond to the portion of the depth image comprises adjusting the part of the model to correspond to a portion of the depth image that represents the user.

5. The method of claim 1, further comprising:

based least on determining that the location of the extremity has not been estimated, positioning the model over the first depth image,

wherein adjusting the part of the model that is located close to the portion of the depth image to correspond to the portion of the depth image comprises adjusting the part of the model to a portion of the depth image that is located closest to the part of the model.

6. The method of claim 1, wherein adjusting the part of the model that is located close to the portion of the depth image to correspond to the portion of the depth image comprises magnetizing the part of the model to a portion of the depth image.

7. The method of claim 6, wherein magnetizing the part of the model to a portion of the depth image comprises magnetizing the part of the model to voxels defining a boundary or surface in the first depth image.

8. The method of claim 1, wherein adjusting the part of the model that is located close to the portion of the depth image comprises adjusting the part of the model to correspond to voxels defining a boundary or surface in the first depth image.

9. The method of claim 1, wherein adjusting the part of the model that is located close to the portion of the depth image to correspond to the portion of the depth image comprises adjusting a joint of a skeletal model.

10. The method of claim 1, wherein identifying an estimated location of an extremity of a user in the first depth image includes determining the location of the extremity of the user.

11. A system, comprising:

at least one processor; and

computing memory communicatively coupled to the at least one processor, the computing memory comprising executable instructions that upon execution cause the system to perform operations comprising;

receiving a first depth image, the first depth image being captured by a depth camera;

identifying an estimated location or position of a first extremity of the user in the first depth image;

positioning a first part of a model to correspond to the estimated location or position of the first extremity in the first depth image; and

based at least on determining that a location or position of a second extremity has not been estimated, adjusting a second part of the model to correspond to a portion of the depth image that is located close to the second part of the model.

12. The system of claim 11, wherein adjusting a second part of the model to correspond to a portion of the depth image that is located close to the second part of the model comprises adjusting the second part of the model to a location of at least one voxel in the first depth image.

13. The system of claim 12, wherein adjusting the second part of the model to a location of at least one voxel in the first depth image comprises adjusting the second part of the model to a location defined by an x-value, y-value, and depth value in the first depth image.

14. The system of claim 12, wherein adjusting a second part of the model to correspond to a portion of the depth image that is located close to the second part of the model comprises adjusting the second part of the model to correspond to a portion of the depth image that represents the user.

15. The system of claim 11, further comprising executable instructions that upon execution cause the system to perform operations comprising:

based at least on determining that the location or the position of the second extremity has not been estimated, positioning the model over the first depth image,

wherein adjusting a second part of the model to correspond to a portion of the depth image that is located close to the second part of the model comprises adjusting the second part of the model to a portion of the depth image that is located closest to the second part of the model.

16. The system of claim 11, wherein adjusting a second part of the model to correspond to a portion of the depth image that is located close to the second part of the model comprises magnetizing the second part of the model to a portion of the depth image.

17. The system of claim 16, wherein magnetizing the second part of the model to a portion of the depth image comprises magnetizing the second part of the model to voxels defining a boundary or surface in the first depth image.

18. The system of claim 11, wherein adjusting a second part of the model to correspond to a portion of the depth image that is located close to the second part of the model comprises adjusting the second part of the model to correspond to voxels defining a boundary or surface in the first depth image.

19. The system of claim 11, wherein a second part of the model to correspond to a portion of the depth image that is located close to the second part of the model comprises adjusting a joint of a skeletal model.

20. The system of claim 11, further comprising executable instructions that upon execution cause the system to perform operations comprising:

downsampling pixels in the depth image before identifying an estimated location or position of a first extremity of the user in the first depth image.

21. The method of claim 11, wherein identifying an estimated location or position of a first extremity of the user in the first depth image comprises determining the location or position of the extremity of the user.

22. A method for tracking a user, comprising:

receiving a first depth image, the first depth image being captured by a depth camera;

based at least on identifying an estimated location of an extremity of a user in the first depth image, positioning a part of a model to correspond to the estimated location of the extremity in the first depth image; and

based at least on determining a failure to estimate a location of an extremity, adjusting a part of the model that is located close to a portion of the depth image to correspond to the portion of the depth image.

23. The method of claim 22, wherein identifying an estimated location of an extremity of a user in the first depth image includes determining the location of the extremity of the user.