Bi-modal depth-image analysis

• US 9,349,040 B2
• Filed: 11/19/2010
• Issued: 05/24/2016
• Est. Priority Date: 11/19/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Brickhill, David
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US12/950,854
• Publication Number:
  US 20120128201A1
• Time in Patent Office: 2,013 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A depth-image analysis system calculates first mode skeletal data representing a human target in an observed scene if a portion of the human target is observed with a first set of joint positions, and calculates second mode skeletal data representing the human target in the observed scene if the portion of the human target is observed with a second set of joint positions different than the first set of joint positions. The first mode skeletal data and the second mode skeletal data have different skeletal joint constraints.

Claims

1. A depth-image analysis system, comprising:

a logic device in operative communication with one or more depth cameras configured to generate a depth map of an observed scene; and

a data-holding device holding instructions executable by the logic device to;

output first mode skeletal data representing a human target in the observed scene if a portion of the human target is observed with a first set of joint positions; and

output second mode skeletal data representing the human target in the observed scene if the portion of the human target is observed with a second set of joint positions different than the first set of joint positions, the first mode skeletal data having first skeletal joint constraints that allow a left hand virtual skeleton joint to separate from a right hand virtual skeleton joint, and the second mode skeletal data having second skeletal joint constraints that lock together the left hand virtual skeleton joint and the right hand virtual skeleton joint.

2. The depth-image analysis system of claim 1, where the first mode skeletal data is one-hand mode skeletal data identifying each of a plurality of virtual skeleton joints with three-dimensional coordinates, and the first set of joint positions includes a left hand joint and a right hand joint observed in a separated joint position;

and where the second mode skeletal data is two-hand mode skeletal data identifying each of a plurality of virtual skeleton joints with three-dimensional coordinates, and the second set of joint positions includes the left hand joint and the right hand joint observed in an unseparated joint position.

3. The depth-image analysis system of claim 2, where the data-holding device holds instructions executable by the logic device to switch from outputting one-hand mode skeletal data to outputting two-hand mode skeletal data if the left hand joint and the right hand joint are observed within a spatial locking threshold for at least a threshold duration.

4. The depth-image analysis system of claim 3, where the data-holding device holds instructions executable by the logic device to switch from outputting two-hand mode skeletal data to outputting one-hand mode skeletal data if the left hand joint and the right hand joint are observed outside of a spatial unlocking threshold.

5. The depth-image analysis system of claim 4, where the spatial locking threshold is smaller than the spatial unlocking threshold.

6. The depth-image analysis system of claim 2, where the two-hand mode skeletal data includes a stable joint complex including a right elbow joint, a left elbow joint, and a locked hand unit.

7. The depth-image analysis system of claim 6, where an occluded one of the right elbow joint and the left elbow joint is positioned as a reflection of an unoccluded one of the right elbow joint and the left elbow joint across a line extending between the locked hand unit and a sternum point.

8. The depth-image analysis system of claim 6, where one or more joints of the stable joint complex have a reduced degree of freedom.

9. The depth-image analysis system of claim 8, where joints not a part of the stable joint complex have normal degrees of freedom.

10. The depth-image analysis system of claim 6, where the locked hand unit includes a right hand joint and a left hand joint locked together in the stable joint complex.

11. The depth-image analysis system of claim 10, where the locked hand unit is constrained to an average observed position of the right hand joint and the left hand joint.

12. The depth-image analysis system of claim 10, where the locked hand unit is constrained to an observed position of an observed hand joint having a highest confidence.

13. The depth-image analysis system of claim 6, where the two-hand mode skeletal data includes a two-handed prop vector with a fixed orientation relative to the stable joint complex and an origination at the locked hand unit.

14. The depth-image analysis system of claim 13, where the two-handed prop vector is perpendicular to a plane defined by the right elbow joint, the left elbow joint, and the locked hand unit.

21. The depth-image analysis system of claim 10, where a three-dimensional coordinate of the right hand joint and a three-dimensional coordinate of the left hand joint are locked together in the stable joint complex.

15. A method of tracking a human target, the method comprising:

modeling the human target observed within a depth map obtained from one or more depth cameras with a virtual skeleton including a plurality of joints, the plurality of joints including a left hand virtual skeleton joint and a right hand virtual skeleton joint;

constraining the virtual skeleton to a two-hand mode if the left hand virtual skeleton joint and the right hand virtual skeleton joint are observed to move within a spatial locking threshold of one another, where the left hand virtual skeleton joint and the right hand virtual skeleton joint are constrained to remain locked together as a locked hand unit in the two-hand mode; and

switching the virtual skeleton from the two-hand mode to a one-hand mode if the left hand virtual skeleton joint and the right hand virtual skeleton joint are observed to move outside of a spatial unlocking threshold of one another.

16. The method of claim 15, where the spatial locking threshold is smaller than the spatial unlocking threshold.

17. The method of claim 15, where the locked hand unit is constrained to an average observed position of the right hand virtual skeleton joint and the left hand virtual skeleton joint.

18. The method of claim 15, where the locked hand unit is constrained to an observed position of an observed hand virtual skeleton joint having a highest confidence.

19. A method of positioning and aiming a virtual prop, the method comprising:

modeling a human target observed within a depth map obtained from one or more depth cameras with a virtual skeleton including a plurality of joints, the plurality of joints including a left elbow joint, a right elbow joint, a left hand joint, and a right hand joint, the left hand joint and the right hand joint locked together as a locked hand unit; and

positioning the virtual prop at the locked hand unit and aiming the virtual prop with a fixed orientation relative to a plane defined by the left elbow joint, the right elbow joint, and the locked hand unit.

20. The method of claim 19, where the fixed orientation is perpendicular to the plane.