Systems and methods for tracking human hands using parts based template matching using corresponding pixels in bounded regions of a sequence of frames that are a specified distance interval from a reference camera

• US 9,111,135 B2
• Filed: 06/11/2013
• Issued: 08/18/2015
• Est. Priority Date: 06/25/2012
• Status: Active Grant

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Aquifi, Inc.
• Inventors: Hummel, Britta, Murali, Giridhar
• Primary Examiner(s): ANSARI, TAHMINA N
• Application Number:
  US13/915,553
• Publication Number:
  US 20140119599A1
• Time in Patent Office: 798 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Systems and methods for tracking human hands using parts based template matching within bounded regions are described. One embodiment of the invention includes a processor; an image capture system configured to capture multiple images of a scene; and memory containing a plurality of templates that are rotated and scaled versions of a finger template. A hand tracking application configures the processor to: obtain a reference frame of video data and an alternate frame of video data from the image capture system; identify corresponding pixels within the reference and alternate frames of video data; identify at least one bounded region within the reference frame of video data containing pixels having corresponding pixels in the alternate frame of video data satisfying a predetermined criterion; and detect at least one candidate finger within the at least one bounded region in the reference frame of video data.

Claims

1. A system for detecting hand posture using parts based template matching to detect individual human fingers, comprising:

a processor;

an image capture system configured to capture multiple images of a scene, where each image is a frame of video data comprising intensity information for a plurality of pixels and the image capture system comprises a reference camera configured to capture a sequence of frames of video comprising a reference frame and an alternate frame of video data and an alternate view camera configured to capture at least an alternate view frame of video data; and

memory containing;

a hand tracking application; and

a plurality of templates that are rotated and scaled versions of a finger template; and

wherein the hand tracking application configures the processor to;

obtain the reference frame of video data and the alternate frame of video data from the video data captured by the reference camera in the image capture system and the alternate view frame from the video data captured by the alternate view camera;

identify corresponding pixels within the reference and alternate frames of video data as pixels that occupy the same pixel location in each of the reference and alternate frames of video data;

identify at least one bounded region within the reference frame of video data captured by the reference camera containing pixels having corresponding pixels in the alternate frame of video data captured by the reference camera with intensity values that differ by a predetermined amount satisfying a predetermined criterion;

identify corresponding pixels within the reference frame of video data captured by the reference camera and alternate view frame of video data captured by the alternate view camera by performing disparity searches to locate pixels within the alternate view frame of video data captured by the alternate view camera that correspond to pixels within the reference frame of video data captured by the reference camera;

generate a depth map containing distances from the reference camera for pixels in the reference frame of video data using information captured by the reference camera including the disparity between corresponding pixels within the reference frame captured by reference camera and the alternate view frame captured by the alternate view camera;

identify at least one bounded region within the reference frame of video data captured by the reference camera containing pixels having distances from the reference camera that are within a specific range of distances from the reference camera; and

detect at least one candidate finger within one of the at least one bounded regions in the reference frame of video data, where each of the at least one candidate finger is a grouping of pixels identified by searching within each of at least one bounded regions in the reference frame of video data for a grouping of pixels that match one of the plurality of templates.

2. The system of claim 1, wherein the at least one bounded region is a bounding rectangle.

3. The system of claim 1, wherein the at least one bounded region is represented as a mask that indicates the pixels within the reference frame of video data captured by the reference camera that are included in the at least one bounded region.

4. The system of claim 1, wherein the hand tracking application further configures the processor to detect an initialization gesture in a sequence of frames of image data obtained from the image capture system and the at least one bounded region is contained within an interaction zone within the reference frame of video data captured by the reference camera, where the interaction zone is a set of pixels within the reference frame of video data captured by the reference camera defined based upon the location within the reference frame of video data captured by the reference camera of the detected initialization gesture.

5. The system of claim 1, wherein the reference and alternate frames of video data captured by the reference camera are adjacent frames in the sequence of frames of video.

6. The system of claim 1, wherein at least one frame of video data is captured by the image capture system between the times at which the reference frame of video data and the alternate frame of video data are captured by the reference camera of the image capture system.

7. The system of claim 1, wherein the hand tracking application further configures the processor to:

obtain a third frame of video data captured by the reference camera of the image capture system;

identify pixels that occupy the same pixel locations in each of the reference and third frames of video data captured by the reference camera; and

identify at least one bounded region within the reference frame of video data captured by the reference camera containing pixels having corresponding pixels in the alternate frame of video data captured by the reference camera or the third frame of video data captured by the reference camera with intensity values that differ by a predetermined amount.

8. The system of claim 1, wherein the at least one bounded region is a bounding rectangle.

9. The system of claim 1, wherein the at least one bounded region is represented as a mask that indicates the pixels within the reference frame of video data captured by the reference camera that are included in the at least one bounded region.

10. The system of claim 1, wherein:

the memory contains data concerning a finger detected in the alternate frame of video data obtained from the reference camera; and

the at least one bounded region within the reference frame of video data captured by the reference camera contains pixels that are within a specific range of locations within the reference frame of video data captured by the reference camera determined using the data concerning a finger detected in the alternate frame of video data.

11. The system of claim 10, wherein the data concerning a finger detected in the alternate frame of video data obtained from the reference camera comprises at least one piece of data selected from the group consisting of:

the location of the finger detected in the alternate frame of video data captured by the reference camera;

the velocity of the finger detected in the alternate frame of video data captured by the reference camera;

the orientation of the finger detected in the alternate frame of video data captured by the reference camera; and

the distance from the reference camera to the finger detected in the alternate frame of video data captured by the reference camera.

12. The system of claim 1, wherein the depth map contains distances from the reference camera for every pixel in the reference frame of video data captured by the reference camera.

13. The system of claim 1, wherein the depth map contains distances from the reference camera for a number of pixels in the reference frame of video data captured by the reference camera that is less than the total number of pixels in the reference frame of video data captured by the reference camera.

14. The system of claim 13, wherein the depth map contains distances from the reference camera for pixels in the reference frame of video data captured by the reference camera corresponding to pixel locations on a low resolution grid, where the low resolution grid has a resolution that is lower than the resolution of the reference frame of video data.

15. The system of claim 1, wherein the hand tracking application configures the processor to generate a confidence map indicating the reliability of distances contained within the depth map.

16. The system of claim 1, wherein the hand tracking application configures the processor to determine the specific range of distances relative to the distance of the pixel that is closest to the reference camera within the depth map.

17. The system of claim 1, wherein the at least one bounded region comprises a bounded region that encompasses the largest group of pixels within the reference frame of video data captured by the reference camera that satisfy criterion including that they are within the specific range of distances from the reference camera.

18. The system of claim 1, wherein at least one bounded region comprises a bounded region that encompasses the union of all pixels within the reference frame of video data captured by the reference camera that satisfy criterion including that they are within the specific range of distances from the reference camera.