Classification of posture states

• US 8,488,888 B2
• Filed: 12/28/2010
• Issued: 07/16/2013
• Est. Priority Date: 12/28/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Balan, Alexandru, Siddiqui, Matheen, Geiss, Ryan M., Kipman, Alex Aben-Athar, Williams, Oliver Michael Christian, Shotton, Jamie
• Primary Examiner(s): CHAWAN, SHEELA C
• Application Number:
  US12/979,897
• Publication Number:
  US 20120163723A1
• Time in Patent Office: 931 Days
• Field of Search: 382/100, 382/103, 382/164, 382/171, 382/224, 382/254, 382/258, 382/259, 382/297, 382/312, 382/173, 382/181, 382/276, 382/107, 382190-208, 235419-427, 356/12, 356/21, 356/22, 348/77, 348/169, 345/419, 345/418, 345/427, 463 36- 39
• US Class Current: 382/224
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

Systems and methods for estimating a posture of a body part of a user are disclosed. In one disclosed embodiment, an image is received from a sensor, where the image includes at least a portion of an image of the user including the body part. The skeleton information of the user is estimated from the image, a region of the image corresponding to the body part is identified at least partially based on the skeleton information, and a shape descriptor is extracted for the region and the shape descriptor is classified based on training data to estimate the posture of the body part.

Claims

1. A method for estimating a posture of a body part of a user, comprising:

receiving an image from a sensor, the image including at least a portion of an image of the user including the body part;

estimating skeleton information of the user to obtain a virtual skeleton from the image, the virtual skeleton including a plurality of joints;

identifying a region of the image corresponding to the body part, the identifying being at least partially based on the skeleton information;

extracting a shape descriptor for the region;

classifying the shape descriptor based on training data to estimate the posture of the body part; and

outputting a response based on the estimated posture of the body part.

2. The method of claim 1, further comprising estimating a body size scaling factor based on at least one distance between joints in the skeleton information and normalizing the shape descriptor based on the body size scaling factor.

3. The method of claim 2, wherein identifying a region of the image corresponding to the body part includes using a topographical search based on the body size scaling factor.

4. The method of claim 1, wherein identifying a region of the image corresponding to the body part is at least partially based on a flood fill approach.

5. The method of claim 1, wherein the body part is a hand and estimating the posture of the body part includes estimating whether the hand is open or closed.

6. The method of claim 1, further comprising estimating a plurality of postures of the body part based on a plurality of images from the sensor and performing temporal filtering of the plurality of estimates to estimate an estimated posture of the body part.

7. The method of claim 1, further comprising assigning a confidence interval to the estimated posture of the body part.

8. The method of claim 1, wherein classifying the shape descriptor based on training data to estimate the posture of the body part is based on at least one machine learning technique.

9. The method of claim 1, wherein the body part is a hand and the training data is partitioned based on meta-data, the meta-data including at least one of a hand orientation, a lower arm angle, a lower arm orientation, depth, and a body size of the user.

10. The method of claim 1, wherein the method further comprises, normalizing the shape descriptor to correct for variations in orientation and location of the body part with respect to the sensor and body size between different users.

11. A computer-readable memory including instructions stored thereon that are executable by a computing device to:

receive a depth image from a depth camera, the depth image including at least a portion of an image of a user including a hand;

estimate skeleton information of the user to obtain a virtual skeleton from the depth image, the virtual skeleton including a plurality of joints;

identify a region of the depth image corresponding to the hand, the identifying being at least partially based on the skeleton information;

extract a shape descriptor for the region;

classify the shape descriptor based on training data to estimate the state of the hand, wherein the training data is partitioned based on meta-data, the meta-data including at least one of hand orientation, depth and a body size of the user; and

output a response based on the estimated state of the hand.

12. The computer-readable memory of claim 11 further including instructions executable to:

estimate a body size scaling factor based on at least one distance between joints in the skeleton information,

normalize the shape descriptor based on the body size scaling factor, and

normalize the shape descriptor to correct for variations in orientation and location of the hand with respect to the sensor, and variations in body size between different users.

13. The computer-readable memory of claim 12, wherein identifying a region of the depth image corresponding to the hand includes using a topographical search based on the body size scaling factor.

14. The computer-readable memory of claim 11, wherein identifying a region of the depth image corresponding to the hand is at least partially based on a flood fill approach.

15. The computer-readable memory of claim 11, wherein estimating the state of the hand includes estimating whether the hand is open or closed.

16. The computer-readable memory of claim 11 further including instructions executable to:

estimate a plurality of states of the hand based on a plurality of depth images from the sensor and performing temporal filtering of the plurality of estimates to estimate the state of the hand.

17. The computer-readable memory of claim 11 further including instructions executable to:

assign a confidence interval to the estimated state of the hand.

18. A computing system, comprising:

a depth camera configured to capture depth information useable to construct a depth image;

a logic subsystem operatively connected to the depth camera; and

a data-holding subsystem holding instructions executable by the logic subsystem to;

receive a depth image from the depth camera, the depth image including at least a portion of an image of a user including a hand of the user;

estimate skeleton information of the user to obtain a virtual skeleton from the depth image, the virtual skeleton including a plurality of joints;

identify a region of the depth image corresponding to the hand, the identifying being at least partially based on the skeleton information;

extract a shape descriptor for the region;

classify the shape descriptor based on training data to estimate the state of the hand; and

output a response based on the estimated state of the hand.

19. The system of claim 18, wherein the data-holding subsystem holding instructions is further executable by the logic subsystem to:

partition the training data based on meta-data, the meta-data including at least one of hand orientation, depth and a body size of the user.

20. The system of claim 18, wherein the data-holding subsystem holding instructions is further executable by the logic subsystem to:

estimate a plurality of states of the hand based on a plurality of depth images from the sensor and perform temporal filtering of the plurality of estimates to estimate an estimated state of the hand.