SYSTEMS AND METHODS FOR TRACKING HUMAN HANDS BY PERFORMING PARTS BASED TEMPLATE MATCHING USING IMAGES FROM MULTIPLE VIEWPOINTS

• US 20130343606A1
• Filed: 05/21/2013
• Published: 12/26/2013
• Est. Priority Date: 06/25/2012
• Status: Active Grant

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Imimtek, Inc.
• Inventors: Dal Mutto, Carlo, Hummel, Britta, Stoppa, Michele, Pasqualotto, Giulian, Mao, Teresa, Giridhar, Murali
• Granted Patent: US 8,934,675 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Systems and methods for tracking human hands by performing parts based template matching using images captured from multiple viewpoints are described. One embodiment of the invention includes a processor, a reference camera, an alternate view camera, and memory containing: a hand tracking application; and a plurality of edge feature templates that are rotated and scaled versions of a finger template that includes an edge features template. In addition, the hand tracking application configures the processor to: detect at least one candidate finger in a reference frame, where each candidate finger is a grouping of pixels identified by searching the reference frame for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates; and verify the correct detection of a candidate finger in the reference frame by locating a grouping of pixels in an alternate view frame that correspond to the candidate finger.

Claims

1. A system for detecting hand posture using parts based template matching to detect individual human fingers, comprising:

a processor;

a reference camera configured to capture sequences of frames of video data, where each frame of video data comprises intensity information for a plurality of pixels;

an alternate view camera configured to capture sequences of frames of video data, where each frame of video data comprises intensity information for a plurality of pixels;

memory containing;

a hand tracking application; and

a plurality of edge feature templates that are rotated and scaled versions of a finger template that comprises an edge features template; and

wherein the hand tracking application configures the processor to;

obtain a sequence of frames of video data including a reference frame of video data and a previous frame of video data from the reference camera;

obtain a sequence of frames of video data including an alternate view frame of video data from the alternate view camera;

identify at least one region in the reference frame of video data where pixels changed from the previous frame of video data to the reference frame of video data;

detect at least one candidate finger in the reference frame of video data, where each of the at least one candidate finger is a grouping of pixels identified by searching the identified at least one region in the reference frame of video data for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates; and

verify the correct detection of a candidate finger in the reference frame of video data by locating a grouping of pixels in the alternate view frame of video data that correspond to the candidate finger.

2. The system of claim 1, wherein the hand tracking application is configured to locate a grouping of pixels in the alternate view frame of video data that correspond to the candidate finger by searching along an epipolar line within the alternate view image for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates, where the epipolar line is defined by the relative location of the center of the reference camera and the center of the alternate view camera.

3. The system of claim 2, wherein the hand tracking application is configured to search along an epipolar line within the alternate view frame of video data for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates by performing a search with respect to pixels within a predetermined margin relative to the epipolar line.

4. The system of claim 2, wherein the hand tracking application is configured to search along an epipolar line within the alternate view frame of video data for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates that is rotated and scaled within predetermined limits relative to the finger template.

5. The system of claim 2, wherein the hand tracking application is configured to search along an epipolar line within the alternate view frame of video data for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates that is rotated and scaled within permitted limits relative to the finger template that are determined based upon the disparity between the grouping of pixels in the alternate view image and the candidate finger in the reference image.

6. The system of claim 5, wherein the permitted rotation limits increase with increased disparity.

7. The system of claim 2, wherein the distance from the reference camera to the candidate finger is determined based upon the disparity between the candidate finger in the reference frame of video data and grouping of pixels in the alternate view frame of video data that correspond to the candidate finger.

8. The system of claim 7, wherein the hand tracking application is configured to search along the epipolar line within the alternate view frame of video data to obtain disparity measurements with pixel precision.

9. The system of claim 7, wherein the hand tracking application is configured to perform a search along the epipolar line within the alternate view frame of video data involving resampling the alternate view frame of video data to obtain disparity measurements with sub-pixel precision.

10. The system of claim 2, wherein the hand tracking application is configured to rectify the alternate view frame of video data relative to the reference frame of video data.

11. The system of claim 2, wherein the hand tracking application is configured to use calibration to perform rectification of pixels that lie in regions surrounding the epipolar line in the alternate view frame of video data to search for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates.

12. The system of claim 2, wherein the edge features of the finger template are synthetically generated.

13. The system of claim 2, wherein the plurality of edge feature templates are stored in a data structure that includes metadata describing the rotation and scaling applied to the finger template to obtain a given edge feature template within the data structure.

14. The system of claim 13, wherein the hand tracking application is configured to select a subset of edge feature templates from the plurality of edge feature templates to use when searching along an epipolar line within the alternate view frame of video data based upon the metadata describing the rotation and scaling of the edge feature template matching the candidate finger in the reference frame of video data.

15. The system of claim 14, wherein the hand tracking application is further configured to select the subset of edge feature templates from the plurality of edge feature templates to use when searching a specific location along the epipolar line within the alternate view image based upon the disparity at the specific location along the epipolar line.

16. The system of claim 2, wherein the edge feature template is a binary edge map.

17. The system of claim 2, wherein the edge feature template is a map of image gradient orientations.

18. The system of claim 2, wherein the hand tracking application configures the processor to search a frame of video data for a grouping of pixels that have image gradient orientations that match a given edge feature template from the plurality of edge feature templates by:

selecting a grouping of pixels;

searching within a predetermined neighborhood of pixels relative to each edge feature in the given edge feature template to find the image gradient orientation that is most similar to the image gradient orientation of the edge feature; and

determining the similarity of the grouping of pixels to the given edge feature template based upon a measure of the similarity of the most similar image gradient orientations found within the grouping of pixels for each of the edge features in the given edge feature template.

19. The system of claim 18, wherein the hand tracking application configures the processor to determine image gradient orientation based upon a change in intensity of neighboring pixels in the frame of video data.

20. The system of claim 18, wherein:

each frame of video data captured by the reference view camera and the alternate view camera comprises intensity information for a plurality of pixels in a plurality of color channels; and

the hand tracking application configures the processor to determine image gradient orientation based upon a change in the color of neighboring pixels in the frame of video data in at least one color channel.

21. The system of claim 20, wherein the frame of video data comprises color information for a plurality of pixels in blue, green and red color channels.

22. The system of claim 21, wherein the hand tracking application configures the processor to determine image gradient orientation by computing the orientation of gradients in red and green color channels separately and using the gradient orientation of the channel whose magnitude is the largest.

23. The system of claim 21, wherein the hand tracking application configures the processor to determine image gradient orientation by computing the orientation of gradients in blue, green, and red color channels separately and using the gradient orientation of the channel whose magnitude is the largest.

24. The system of claim 2, wherein the hand tracking application configures the processor to search a frame of video data (I) for a grouping of pixels that have image gradient orientations that match a given edge feature template (T) from the plurality of edge feature templates by determining whether the similarity of a number of gradient orientations extracted from the given edge feature template at a list P of locations r to be considered from edge features (O) in the given edge feature template and image gradient orientations extracted from the frame of video data relative to a pixel location c exceed a predetermined matching threshold using the following similarity measure:

25. The system of claim 24, wherein the hand tracking application configures the processor to quantize the image gradient orientations into a predetermined number (n_o) of values that can each be represented by a single bit.

26. The system of claim 25, wherein the hand tracking application configures the processor to calculate the similarity measure by spreading the quantized image gradient orientation at pixel location t (ori(I,t)) in a T×

T window around the pixel location t and encoding the spread image gradient orientations using a binary string of length n_o, where each bit in the string corresponds to each quantized image gradient orientation present in the T×

T window.

27. The system of claim 26, wherein the hand tracking application configures the processor to:

precompute lookup tables of max<sub>tε

R(c+r)</sub>|cos(ori(O,r)−

ori(I,t))| for each possible orientation ori(O,r) of an edge feature at a given pixel location t in the video image data; and

evaluate the similarity measure for the given edge feature template using the precomputed lookup tables.

28. The system of claim 24, wherein the hand tracking application determines the pixel locations (c) searched within a reference frame of video data using information including the pixel location of a previously identified finger.

29. The system of claim 28, wherein the hand tracking application determines the pixel locations (c) searched within a reference frame of video data using information further comprising the velocity of a previously identified finger.

30. The system in claim 24, wherein the hand tracking application configures the processor to:

detect the presence and initial location of at least one finger by recognizing an initialization gesture; and

determine the pixel locations (c) searched within a reference frame of video data using information including the initial location of the at least one finger.

31. The system of claim 24, wherein:

the memory contains video data of a previous frame obtained from the reference camera; and

the hand tracking application configures the processor to determine the pixel locations (c) searched within a reference frame of video data using information including pixels in the frame of video data that have changed relative to the pixels in the video data of the previous frame obtained from the reference camera.

32. The system of claim 2, wherein:

each frame of video data captured by the reference view camera and the alternate view camera includes color information for a plurality of pixels comprising intensity information in a plurality of color channels; and

the hand tracking application further configures the processor to verify the correct detection of a candidate finger in the reference frame of video data by confirming that the colors of the pixels within the grouping of pixels identified as a candidate finger satisfy a skin color criterion.

33. The system of claim 32, wherein the hand tracking application further configures the processor to verify the correct detection of a candidate finger in the frame of video data obtained from the reference camera by confirming that the colors of the pixels within the grouping of pixels in the alternate view frame of video data identified as corresponding to the candidate finger satisfy a skin color criterion.

34. The system of claim 33, wherein:

the finger template further comprises a plurality of surface color pixel sample locations defined relative to the edge features template; and

the hand tracking application configures the processor to confirm that the colors of the pixels within a grouping of pixels identified within a frame of video data satisfy a skin color criterion by confirming that the colors of the surface color pixel sample locations for the edge feature template from the plurality of edge feature templates that matches the grouping of pixels satisfy a skin color criterion.

35. The system of claim 34, wherein the skin color criterion is a number of pixel sample locations having a color corresponding to a predetermined range of colors.

36. The system of claim 35, wherein the skin color criterion is a ratio of pixel sample locations having a color corresponding to a predetermined range of colors relative to the overall number of pixel sample locations.

38. The system of claim 2, wherein the hand tracking application further configures the processor to verify the correct detection of a candidate finger by confirming that the detected finger belongs to a possible hand pose indicative of a correct identification of a finger.

39. The system of claim 2, further comprising:

a display interface configured to drive a display device;

wherein the hand tracking application configures the processor to;

determine distance from the reference camera to a detected finger based upon the disparity between the detected finger in the reference frame of video data and the grouping of pixels in the alternate view frame of video data that correspond to the detected finger

determine the orientation of the detected finger based upon at least the edge feature template from the plurality of edge feature templates that matched the detected finger;

map the determined distance and orientation of the detected finger to a location on the display device;

generate a target on the display device at the mapped location using the display interface.

37. The system of 35, wherein the hand tracking application is configured to adapt the skin color criterion based upon at least one characteristic of the given edge feature template selected from the group consisting of the scale, and rotation of the given edge feature template relative to the finger template.

40. A system for detecting hand posture using parts based template matching to detect individual human fingers, comprising:

a processor;

a display interface configured to drive a display device;

a reference camera configured to capture sequences of frames of video data, where each frame of video data comprises color information for a plurality of pixels;

an alternate view camera configured to capture sequences of frames of video data, where each frame of video data comprises color information for a plurality of pixels;

memory containing;

a hand tracking application; and

a plurality of edge feature templates that are rotated and scaled versions of a finger template that are stored in a data structure that includes metadata describing the rotation and scaling applied to the finger template to obtain a given edge feature template, where the finger template comprises;

an edge features template; and

a plurality of surface color pixel sample locations defined relative to the edge features template;

wherein the hand tracking application configures the processor to;

obtain a sequence of frames of video data including a reference frame of video data and a previous frame of video data from the reference camera;

obtain a sequence of frames of video data including an alternate view frame of video data from the alternate view camera;

identify at least one region in the reference frame of video data where pixels changed from the previous frame of video data to the reference frame of video data;

detect at least one candidate finger in the reference frame of video data, where each of the at least one candidate finger is a grouping of pixels identified by searching the identified at least one region in the reference frame of video data for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates; and

verify the correct detection of a candidate finger in the reference frame of video data by confirming that the colors of the surface color pixel sample locations for the edge feature template from the plurality of edge feature templates that matches the grouping of pixels identified as a candidate finger satisfy a skin color criterion;

select a subset of edge feature templates from the plurality of edge feature templates based upon the metadata describing the rotation and scaling of the edge feature template matching the candidate finger in the reference frame of video data;

verify the correct detection of a candidate finger in the reference frame of video data by searching along an epipolar line within the alternate view image and with respect to pixels within a predetermined margin relative to the epipolar line for a grouping of pixels that correspond to the candidate finger, where a grouping of pixels corresponds to the candidate finger when they have image gradient orientations that match one of the subset of edge feature templates and the epipolar line is defined by the relative location of the center of the reference camera and the center of the alternate view camera;

verify the correct detection of a candidate finger in the reference frame of video data by confirming that the colors of the surface color pixel sample locations for the edge feature template from the plurality of edge feature templates that matches the grouping of pixels corresponding to the candidate finger in the alternate view frame of video data satisfy a skin color criterion;

determine distance from the reference camera to a detected finger based upon the disparity between the detected finger in the reference frame of video data and grouping of pixels in the alternate view frame of video data that correspond to the detected finger

determine the orientation of the detected finger based upon at least the edge feature template from the plurality of edge feature templates that matched the detected finger;

map the determined distance and orientation of the detected finger to a location on the display device;

generate a target on the display device at the mapped location using the display interface.

41. A method of detecting hand posture by performing parts based template matching to detect individual human fingers, comprising:

detect at least one candidate finger in the reference frame of video data, where each of the at least one candidate finger is a grouping of pixels identified by searching the identified at least one region in the reference frame of video data for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates; and

capturing a sequence of frames of video data including a reference frame of video data and a previous frame of video data using a reference camera, where the reference frame of video data comprises intensity information for a plurality of pixels;

capturing a sequence of frames of video data including an alternate view frame of video data using an alternate view camera, where the alternate view frame of video data comprises intensity information for a plurality of pixels;

identify at least one region in the reference frame of video data where pixels changed from the previous frame of video data to the reference frame of video data using a processor configured by a hand tracking application;

detecting at least one candidate finger in the reference frame of video data using the processor configured by the hand tracking application, where each of the at least one candidate finger is a grouping of pixels identified by searching the reference frame of video data for a grouping of pixels that have image gradient orientations that match one of a plurality of edge feature templates that are rotated and scaled versions of a finger template; and

verifying the correct detection of a candidate finger in the reference frame of video data by locating a grouping of pixels in the alternate view frame of video data that correspond to the candidate finger using the processor configured by the hand tracking application.