Systems and methods for tracking human hands using parts based template matching within bounded regions

• US 8,830,312 B2
• Filed: 07/15/2013
• Issued: 09/09/2014
• Est. Priority Date: 06/25/2012
• Status: Active Grant

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Aquifi, Inc.
• Inventors: Hummel, Britta, Murali, Giridhar
• Primary Examiner(s): SENFI, BEHROOZ M
• Application Number:
  US13/942,655
• Publication Number:
  US 20130342671A1
• Time in Patent Office: 421 Days
• Field of Search: 348/169, 348/143, 348/77, 348/345, 348/352, 348/349, 382/103, 382/190, 382/254, 382/199, 382/153, 382/165, 382/107, 382/118
• US Class Current: 348/77
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Systems and methods for tracking human hands using parts based template matching within bounded regions are described. One embodiment of the invention includes a processor; an image capture system configured to capture multiple images of a scene; and memory containing a plurality of templates that are rotated and scaled versions of a finger template. A hand tracking application configures the processor to: obtain a reference frame of video data and an alternate frame of video data from the image capture system; identify corresponding pixels within the reference and alternate frames of video data; identify at least one bounded region within the reference frame of video data containing pixels having corresponding pixels in the alternate frame of video data satisfying a predetermined criterion; and detect at least one candidate finger within the at least one bounded region in the reference frame of video data.

Claims

1. A system for detecting hand posture using parts based template matching to detect individual human fingers, comprising:

a processor;

an image capture system configured to capture multiple images of a scene, where;

each image is a frame of video data comprising intensity information for a plurality of pixels; and

the image capture system comprises a reference camera configured to capture a sequence of frames of video comprising a reference and an alternate frame of video data;

memory containing;

a hand tracking application; and

a plurality of templates that are rotated and scaled versions of a finger template; and

wherein the hand tracking application configures the processor to;

obtain a reference frame of video data and an alternate frame of video data from the image capture system;

identify corresponding pixels within the reference and alternate frames of video data as pixels that occupy the same pixel location in each of the reference and alternate frames of video data;

identify at least one bounded region within the reference frame of video data containing pixels having corresponding pixels in the alternate frame of video data having corresponding pixels with intensity values that differ by a predetermined amount; and

detect at least one candidate finger within the at least one bounded region in the reference frame of video data, where each of the at least one candidate finger is a grouping of pixels identified by searching within the at least one bounded region in the reference frame of video data for a grouping of pixels that match one of the plurality of templates.

2. The system of claim 1, wherein the at least one bounded region is selected from a group consisting of:

a bounding rectangle; and

as a mask that indicates the pixels within the reference frame of video data that are included in the at least one bounded region.

3. The system of claim 1, wherein the hand tracking application further configures the processor to detect an initialization gesture in a sequence of frames of image data obtained from the image capture system and the at least one bounded region is contained within an interaction zone within the reference frame of video data, where the interaction zone is a set of pixels within the reference frame of video data defined based upon the location within the reference frame of video data of the detected initialization gesture.

4. The system of claim 1, wherein the hand tracking application further configures the processor to:

obtain a third frame of video data from the image capture system;

identify pixels that occupy the same pixel locations in each of the reference and third frames of video data; and

identify at least one bounded region within the reference frame of video data containing pixels having corresponding pixels in the alternate frame of video data or the third frame of video data with intensity values that differ by a predetermined amount.

5. The system of claim 1, wherein:

the memory contains data concerning a finger detected in the alternate frame of video data obtained from the reference camera; and

the at least one bounded region within the reference frame of video data contains pixels that are within a specific range of locations within the reference frame of video data determined using the data concerning a finger detected in the alternate frame of video data.

26. The system of claim 1, wherein the reference and alternate frames of video data comprise intensity information for a plurality of pixels in a plurality of channels selected from the group consisting of blue, green, red, infrared, near-infrared, and ultraviolet portions of the spectrum.

6. A system for detecting hand posture using parts based template matching to detect individual human fingers, comprising:

a processor;

an image capture system configured to capture multiple images of a scene, where;

each image is a frame of video data comprising intensity information for a plurality of pixels; and

the image capture system comprises a reference camera configured to capture a reference frame of video data and an alternate view camera configured to capture an alternate frame of video data;

memory containing;

a hand tracking application; and

a plurality of templates that are rotated and scaled versions of a finger template; and

wherein the hand tracking application configures the processor to;

obtain a reference frame of video data and an alternate frame of video data from the image capture system;

identify corresponding pixels within the reference and alternate frames of video data by performing disparity searches to locate pixels within the alternate frame of video data that correspond to pixels within the reference frame of video data;

generate a depth map containing distances from the reference camera for pixels in the reference frame of video data using information including the disparity between corresponding pixels within the reference and alternate frames of video data; and

identify at least one bounded region within the reference frame of video data containing pixels having distances from the reference camera that are within a specific range of distances from the reference camera; and

detect at least one candidate finger within the at least one bounded region in the reference frame of video data, where each of the at least one candidate finger is a grouping of pixels identified by searching within the at least one bounded region in the reference frame of video data for a grouping of pixels that match one of the plurality of templates.

7. The system of claim 6, wherein the depth map contains distances from the reference camera for every pixel in the reference frame of video data.

8. The system of claim 6, wherein the depth map contains distances from the reference camera for a number of pixels in the reference frame of video data that is less than the total number of pixels in the reference frame of video data.

9. The system of claim 8, wherein the depth map contains distances from the reference camera for pixels in the reference frame of video data corresponding to pixel locations on a low resolution grid, where the low resolution grid has a resolution that is lower than the resolution of the reference frame of video data.

10. The system of claim 6, wherein the hand tracking application configures the processor to determine the specific range of distances relative to the distance of the pixel that is closest to the reference camera within the depth map.

11. The system of claim 6, wherein the at least one bounded region comprises a bounded region that encompasses the largest group of pixels within the reference frame of video data that satisfy criterion including that they are within the specific range of distances from the reference camera.

12. The system of claim 6, wherein at least one bounded region comprises a bounded region that encompasses the union of all pixels within the reference frame of video data that satisfy criterion including that they are within the specific range of distances from the reference camera.

13. The system of claim 6, wherein:

the memory contains data concerning a distance from the reference camera to a finger detected in a previous frame of video data obtained from the reference camera; and

the at least one bounded region within the reference frame of video data contains pixels that are within a specific range of distances from the reference camera determined relative to the distance from the reference camera of the finger detected in the previous frame of video data.

14. The system of claim 6, wherein:

the memory contains video data of a previous frame obtained from the reference camera; and

the hand tracking application configures the processor to;

compare the reference frame of video data to the previous frame obtained from the reference camera stored in memory to identify moving pixels; and

identify at least one bounded region within the reference frame of video data containing pixels that are moving and that have distances from the reference camera that are within a specific range of distances from the reference camera.

15. The system of claim 14, wherein the hand tracking application configures the processor to identify at least one bounded region within the reference frame of video data containing pixels that are moving and that have distances from the reference camera that are within a specific range of distances from the reference camera by:

identifying at least one preliminary bounded region within the reference frame of video data containing pixels that are moving;

generating the depth map based upon the identified at least one preliminary bounded region in the reference frame of video data so that the depth map contains distances from the reference camera for pixels within the at least one preliminary bounded region in the reference frame of video data; and

identify the at least one bounded region within the at least one preliminary bounded region in the reference frame of video data using the depth map.

16. The system of claim 6, wherein the hand tracking application further configures the processor to verify the correct detection of a candidate finger in the reference frame of video data by locating a grouping of pixels in the alternate frame of video data that correspond to the candidate finger.

17. The system of claim 16, wherein the hand tracking application is configured to locate a grouping of pixels in the alternate frame of video data that correspond to the candidate finger by searching along an epipolar line within the alternate frame of video data for a grouping of pixels that match one of the plurality of templates, where the epipolar line is defined by the relative location of the center of the reference camera and the center of the alternate view camera.

18. The system of claim 17, wherein the hand tracking application is configured to search a distance along the epipolar line within the alternate frame of video data for a grouping of pixels that match one of the plurality of templates based upon the distance of the candidate finger from the reference camera.

19. The system of claim 17, wherein the hand tracking application is configured to search a predetermined range of distances along the epipolar line within the alternate frame of video data for a grouping of pixels that match one of the plurality of templates, where the predetermined range of distances is determined relative to a disparity determined based upon the distance of the candidate finger from the reference camera.

20. The system of claim 17, wherein the hand tracking application is configured to search along an epipolar line within the alternate frame of video data for a grouping of pixels that match one of the plurality of templates by performing a search with respect to pixels within a predetermined margin relative to the epipolar line.

21. The system of claim 6, wherein:

each frame of video data captured by the reference view camera and the alternate view camera includes color information for a plurality of pixels comprising intensity information in a plurality of color channels; and

the hand tracking application further configures the processor to verify the correct detection of a candidate finger in the reference frame of video data by confirming that the colors of the pixels within the grouping of pixels identified as a candidate finger satisfy a skin color criterion.

22. The system of claim 6, wherein the finger template is an edge feature template.

23. The system of claim 22, wherein the edge feature template is a map of image gradient orientations.

24. The system of claim 22, wherein the hand tracking application configures the processor to search a frame of video data for a grouping of pixels that have image gradient orientations that match a given edge feature template from the plurality of edge feature templates by:

selecting a grouping of pixels;

searching within a predetermined neighborhood of pixels relative to each edge feature in the given edge feature template to find the image gradient orientation that is most similar to the image gradient orientation of the edge feature; and

determining the similarity of the grouping of pixels to the given edge feature template based upon a measure of the similarity of the most similar image gradient orientations found within the grouping of pixels for each of the edge features in the given edge feature template.

25. The system of claim 6, further comprising:

a display interface configured to drive a display device;

wherein the hand tracking application configures the processor to;

determine the orientation of the detected finger based upon at least the template from the plurality of templates that matched the detected finger;

map the distance and determined orientation of the detected finger to a location on the display device;

generate a target on the display device at the mapped location using the display interface.

28. The system of claim 6, wherein the at least one bounded region is selected from a group consisting of:

a bounding rectangle; and

as a mask that indicates the pixels within the reference frame of video data that are included in the at least one bounded region.

29. The system of claim 6, wherein the hand tracking application further configures the processor to detect an initialization gesture in a sequence of frames of image data obtained from the image capture system and the at least one bounded region is contained within an interaction zone within the reference frame of video data, where the interaction zone is a set of pixels within the reference frame of video data defined based upon the location within the reference frame of video data of the detected initialization gesture.

30. The system of claim 6, wherein the reference and alternate frames of video data comprise intensity information for a plurality of pixels in a plurality of channels selected from the group consisting of blue, green, red, infrared, near-infrared, and ultraviolet portions of the spectrum.

27. A system for detecting hand posture using parts based template matching to detect individual human fingers, comprising:

a processor;

a display interface configured to drive a display device;

a reference camera configured to capture sequences of frames of video data, where each frame of video data comprises color information for a plurality of pixels;

an alternate view camera configured to capture sequences of frames of video data, where each frame of video data comprises color information for a plurality of pixels;

memory containing;

a hand tracking application; and

a plurality of edge feature templates that are rotated and scaled versions of a finger template that are stored in a data structure that includes metadata describing the rotation and scaling applied to the finger template to obtain a given edge feature template, where the finger template comprises;

an edge features template; and

a plurality of surface color pixel sample locations defined relative to the edge features template;

wherein the hand tracking application configures the processor to;

obtain a reference frame of video data from the reference camera;

obtain an alternate view frame of video data from the alternate view camera;

generate a depth map containing distances from the reference camera for pixels in the reference frame of video data using information including the disparity between corresponding pixels within the reference and alternate view frames of video data; and

identify at least one bounded region within the reference frame of video data containing pixels having distances from the reference camera that are within a specific range of distances from the reference camera;

detect at least one candidate finger in the reference frame of video data, where each of the at least one candidate finger is a grouping of pixels identified by searching within the at least one bounded region in the reference frame of video data for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates;

verify the correct detection of a candidate finger in the reference frame of video data by confirming that the colors of the surface color pixel sample locations for the edge feature template from the plurality of edge feature templates that matches the grouping of pixels identified as a candidate finger satisfy a skin color criterion;

select a subset of edge feature templates from the plurality of edge feature templates based upon the metadata describing the rotation and scaling of the edge feature template matching the candidate finger in the reference frame of video data;

verify the correct detection of a candidate finger in the reference frame of video data by searching along an epipolar line within the alternate view frame of video data and with respect to pixels within a predetermined margin relative to the epipolar line for a grouping of pixels that correspond to the candidate finger, where a grouping of pixels corresponds to the candidate finger when they have image gradient orientations that match one of the subset of edge feature templates and the epipolar line is defined by the relative location of the center of the reference camera and the center of the alternate view camera;

verify the correct detection of a candidate finger in the reference frame of video data by confirming that the colors of the surface color pixel sample locations for the edge feature template from the plurality of edge feature templates that matches the grouping of pixels corresponding to the candidate finger in the alternate view frame of video data satisfy a skin color criterion;

determine the orientation of the detected finger based upon at least the edge feature template from the plurality of edge feature templates that matched the detected finger;

map the distance and orientation of the detected finger to a location on the display device; and

generate a target on the display device at the mapped location using the display interface.