Systems and methods for estimating a non-visible or occluded body part

• US 9,182,814 B2
• Filed: 06/26/2009
• Issued: 11/10/2015
• Est. Priority Date: 05/29/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kipman, Alex A., Perez, Kathryn Stone, Finocchio, Mark J., Geiss, Ryan Michael, Tsunoda, Kudo
• Primary Examiner(s): Yentrapati, Avinash
• Application Number:
  US12/492,457
• Publication Number:
  US 20100303302A1
• Time in Patent Office: 2,328 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A depth image of a scene may be received, observed, or captured by a device. The depth image may include a human target that may have, for example, a portion thereof non-visible or occluded. For example, a user may be turned such that a body part may not be visible to the device, may have one or more body parts partially outside a field of view of the device, may have a body part or a portion of a body part behind another body part or object, or the like such that the human target associated with the user may also have a portion body part or a body part non-visible or occluded in the depth image. A position or location of the non-visible or occluded portion or body part of the human target associated with the user may then be estimated.

Claims

1. A method comprising:

receiving a depth image of a scene;

determining that a portion of a human target in the depth image is non-visible;

estimating a position of the non-visible portion of the human target, wherein the estimating comprises;

generating an initial estimation of the position of the non-visible portion of the human target based at least in part on force vectors associated with a visible portion of the human target; and

modifying the initial estimation based at least in part on historical data associated with the non-visible portion, wherein the historical data comprises data associated with a previously received depth image where the non-visible portion is visible, wherein the historical data comprises at least one of an X value, a Y value, or a Z value of the non-visible portion in the previously received depth image where the non-visible portion is visible;

determining a new position of the non-visible portion when a pixel associated with the non-visible portion becomes visible in a subsequent depth image.

2. The method of claim 1, wherein estimating the position of the non-visible portion of the human target further comprises dragging the non-visible portion of the human target behind the visible portion of the human target, wherein the non-visible portion drifts behind the visible portion of the human target.

3. The method of claim 1, wherein estimating the position of the non-visible portion of the human target further comprises generating at least one of the following:

an estimated X value, an estimated Y value, and an estimated Z value for the non-visible portion.

4. The method of claim 3, wherein the estimated X value, the estimated Y value, and the estimated Z value are generated based on the visible portion of the human target.

5. The method of claim 4, wherein the estimated X value, the estimated Y value, and the estimated Z value of the non-visible portion is generated based on at least one of the following:

an X value of the visible portion, a Y value of the visible portion, a Z value of the visible portion, momentum of the visible portion, a velocity of the visible portion, a rotation of the visible portion, or a trajectory of the visible portion.

6. The method of claim 3, further comprising:

adjusting a model of the human target based on the estimated X value, the estimated Y value, and the estimated Z value; and

processing the model of the human target with the estimated X value, the estimated Y value, and the estimated Z value.

7. A computer-readable storage device having stored thereon computer-executable instructions, the computer-executable instructions comprising instructions for:

receiving a depth image of a scene;

determining that a portion of a human target in the depth image is non-visible;

estimating a position of the non-visible portion of the human target, wherein the estimating comprises;

generating an initial estimation of the position of the non-visible portion of the human target based at least in part on force vectors associated with a visible portion of the human target; and

modifying the initial estimation based at least in part on historical data associated with the non-visible portion, wherein the historical data comprises data associated with a previously received depth image where the non-visible portion is visible, wherein the historical data comprises at least one of an X value, a Y value, or a Z value of the non-visible portion in the previously received depth image where the non-visible portion is visible; and

determining a new position of the non-visible portion when a pixel associated with the non-visible portion becomes visible in a subsequent depth image.

8. The computer-readable storage device of claim 7, wherein estimating the position of the non-visible portion of the human target further comprises dragging the non-visible portion of the human target behind the visible portion of the human target, wherein the non-visible portion drifts behind the visible portion of the human target.

9. The computer-readable storage device of claim 7, wherein estimating the position of the non-visible portion of the human target further comprises generating at least one of the following:

an estimated X value, an estimated Y value, and an estimated Z value for the non-visible portion.

10. The computer-readable storage device of claim 9, wherein the estimated X value, the estimated Y value, and the estimated Z value are generated based on the visible portion of the human target.

11. The computer-readable storage device of claim 10, wherein the estimated X value, the estimated Y value, and the estimated Z value of the non-visible portion is generated based on at least one of the following:

an X value of the visible portion, a Y value of the visible portion, a Z value of the visible portion, momentum of the visible portion, a velocity of the visible portion, a rotation of the visible portion, or a trajectory of the visible portion.

12. The computer-readable storage device of claim 9, wherein the computer executable instructions further comprise instructions for:

adjusting a model of the human target based on the estimated X value, the estimated Y value, and the estimated Z value; and

processing the model of the human target with the estimated X value, the estimated Y value, and the estimated Z value.

13. A system comprising:

a capture device, wherein the capture device comprises a camera component operable to receive a depth image of a scene, wherein the depth image comprises a human target associated with a user; and

a computing device operable to communicate with the capture device, wherein the computing device is operable to perform operations comprising;

determining that a portion of the human target in the depth image is non-visible;

estimating a position of the non-visible portion of the human target, wherein the estimating comprises;

generating an initial estimation of the position of the non-visible portion of the human target based at least in part on force vectors associated with a visible portion of the human target; and

modifying the initial estimation based at least in part on historical data associated with the non-visible portion, wherein the historical data comprises data associated with a previously received depth image where the non-visible portion is visible, wherein the historical data comprises at least one of an X value, a Y value, or a Z value of the non-visible portion in the previously received depth image where the non-visible portion is visible; and

determining a new position of the non-visible portion when a pixel associated with the non-visible portion becomes visible in a subsequent depth image.

14. The system of claim 13, wherein estimating the position of the non-visible portion of the human target further comprises dragging the non-visible portion of the human target behind the visible portion of the human target, wherein the non-visible portion drifts behind the visible portion of the human target.

15. The system of claim 13, wherein estimating the position of the non-visible portion of the human target further comprises generating at least one of the following:

an estimated X value, an estimated Y value, and an estimated Z value for the non-visible portion.

16. The system of claim 15, wherein the estimated X value, the estimated Y value, and the estimated Z value are generated based on the visible portion of the human target.

17. The system of claim 16, wherein the estimated X value, the estimated Y value, and the estimated Z value of the non-visible portion is generated based on at least one of the following:

an X value of the visible portion, a Y value of the visible portion, a Z value of the visible portion, momentum of the visible portion, a velocity of the visible portion, a rotation of the visible portion, or a trajectory of the visible portion.

18. The system of claim 15, wherein the operations further comprise:

adjusting a model of the human target based on the estimated X value, the estimated Y value, and the estimated Z value; and

processing the model of the human target with the estimated X value, the estimated Y value, and the estimated Z value.