Three-dimensional hand tracking using depth sequences

• US 9,811,721 B2
• Filed: 05/07/2015
• Issued: 11/07/2017
• Est. Priority Date: 08/15/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Tang, Feng, Li, Ang, Shi, Xiaojin
• Primary Examiner(s): Akhavannik, Hadi
• Application Number:
  US14/706,649
• Publication Number:
  US 20160048726A1
• Time in Patent Office: 915 Days
• Field of Search: None
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals

Abstract

In the field of Human-computer interaction (HCI), i.e., the study of the interfaces between people (i.e., users) and computers, understanding the intentions and desires of how the user wishes to interact with the computer is a very important problem. The ability to understand human gestures, and, in particular, hand gestures, as they relate to HCI, is a very important aspect in understanding the intentions and desires of the user in a wide variety of applications. In this disclosure, a novel system and method for three-dimensional hand tracking using depth sequences is described. Some of the major contributions of the hand tracking system described herein include: 1.) a robust hand detector that is invariant to scene background changes; 2.) a bi-directional tracking algorithm that prevents detected hands from always drifting closer to the front of the scene (i.e., forward along the z-axis of the scene); and 3.) various hand verification heuristics.

Claims

1. An apparatus, comprising:

a depth-sensing camera;

a memory having, stored therein, computer program code; and

one or more processing units operatively coupled to the memory and configured to execute instructions in the computer program code that cause the one or more processing units to;

receive a depth map of a scene containing one or more human hands from the depth-sensing camera, the depth map comprising a matrix of pixels, each pixel having a depth value;

extract, from the depth map, features based on the depth values of the pixels in a plurality of patches distributed in respective positions over the one or more human hands, wherein the depth values of the pixels are normalized, such that the features are background-invariant;

match the extracted features to previously-stored features;

estimate a position of at least one of the one or more human hands based, at least in part, on stored information associated with the matched features; and

track the position of the at least one of the one or more human hands,

wherein the instructions to track comprise instructions to track bi-directionally along a z-axis of the scene.

2. The apparatus of claim 1, wherein the features use a constant value for the depth values of background pixels.

3. The apparatus of claim 1, wherein the instructions to match the extracted features to previously-stored features further comprise instructions to use a background-invariant decision forest.

4. The apparatus of claim 1, wherein the instructions to estimate the position of the at least one of the one or more human hands further comprise instructions to disregard pixels that do not exhibit a threshold amount of motion.

5. The apparatus of claim 1, wherein the instructions to estimate the position of the at least one of the one or more human hands further comprise instructions to disregard hands that do not exhibit single-directional connectivity to a human body.

6. The apparatus of claim 1, wherein the instructions to track bi-directionally along a z-axis of the scene further comprise instructions to locate local extrema in the depth map of the scene.

7. The apparatus of claim 1, wherein the instructions to track bi-directionally along a z-axis of the scene further comprise instructions to weight the x-coordinate value and y-value coordinate of the pixels in the plurality of patches.

8. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause one or more processing units to:

receive a depth map of a scene containing one or more human hands from a depth-sensing camera, the depth map comprising a matrix of pixels, each pixel having a depth value;

extract, from the depth map, features based on the depth values of the pixels in a plurality of patches distributed in respective positions over the one or more human hands, wherein the depth values of the pixels are normalized, such that the features are background-invariant;

match the extracted features to previously-stored features;

estimate a position of at least one of the one or more human hands based, at least in part, on stored information associated with the matched features; and

track the position of the at least one of the one or more human hands,

wherein the instructions to track comprise instructions to track bi-directionally along a z-axis of the scene.

9. The non-transitory program storage device of claim 8, wherein the features use a constant value for the depth values of background pixels.

10. The non-transitory program storage device of claim 8, wherein the instructions to match the extracted features to previously-stored features further comprise instructions to use a background-invariant decision forest.

11. The non-transitory program storage device of claim 8, wherein the instructions to estimate the position of the at least one of the one or more human hands further comprise instructions to disregard pixels that do not exhibit a threshold amount of motion.

12. The non-transitory program storage device of claim 8, wherein the instructions to estimate the position of the at least one of the one or more human hands further comprise instructions to disregard hands that do not exhibit single-directional connectivity to a human body.

13. The non-transitory program storage device of claim 8, wherein the instructions to track bi-directionally along a z-axis of the scene further comprise instructions to locate local extrema in the depth map of the scene.

14. The non-transitory program storage device of claim 8, wherein the instructions to track bi-directionally along a z-axis of the scene further comprise instructions to weight the x-coordinate value and y-value coordinate of the pixels in the plurality of patches.

15. A computer-implemented method, comprising:

receiving a depth map of a scene containing one or more human hands from a depth-sensing camera, the depth map comprising a matrix of pixels, each pixel having a depth value;

extracting, from the depth map, features based on the depth values of the pixels in a plurality of patches distributed in respective positions over the one or more human hands, wherein the depth values of the pixels are normalized, such that the descriptors are background-invariant;

matching the extracted features to previously-stored features;

estimating a position of at least one of the one or more human hands based, at least in part, on stored information associated with the matched features; and

tracking the position of the at least one of the one or more human hands,

wherein the instructions to track comprise instructions to track bi-directionally along a z-axis of the scene.

16. The computer-implemented method of claim 15, wherein the features use a constant value for the depth values of background pixels.

17. The computer-implemented method of claim 15, wherein estimating the position of the at least one of the one or more human hands further comprises disregarding background pixels and pixels that do not exhibit a threshold amount of motion.

18. The computer-implemented method of claim 15, wherein estimating the position of the at least one of the one or more human hands further comprises disregarding hands that do not exhibit single-directional connectivity to a human body.

19. The computer-implemented method of claim 15, wherein tracking bi-directionally along a z-axis of the scene further comprises locating local extrema in the depth map of the scene.

20. The computer-implemented method of claim 15, wherein tracking bi-directionally along a z-axis of the scene further comprises weighting the x-coordinate value and y-value coordinate of the pixels in the plurality of patches.