Systems and methods for tracking human hands using parts based template matching

• US 9,098,739 B2
• Filed: 05/21/2013
• Issued: 08/04/2015
• Est. Priority Date: 06/25/2012
• Status: Active Grant

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Aquifi, Inc.
• Inventors: Dal Mutto, Carlo, Hummel, Britta, Stoppa, Michele, Pasqualotto, Giuliano, Mao, Teresa, Murali, Giridhar
• Primary Examiner(s): Abdi, Amara
• Application Number:
  US13/899,520
• Publication Number:
  US 20130343605A1
• Time in Patent Office: 805 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Systems and methods for tracking human hands using parts based template matching are described. One embodiment of the invention includes a processor, a reference camera and memory containing: a hand tracking application; and a finger template including an edge features template. In addition, the hand tracking application configures the processor to: detect at least one candidate finger in a frame of video data received from the reference camera, where each candidate finger is a grouping of pixels identified by searching the frame of video data for a grouping of pixels that have image gradient orientations that match the edge features of the finger template accounting for rotation and scaling differences; and verify the correct detection of a candidate finger by confirming that the colors of the pixels within the grouping of pixels identified as a candidate finger satisfy a skin color criterion.

Claims

1. A system for detecting hand posture using parts based template matching to detect individual human fingers, comprising:

a processor;

a reference camera configured to capture sequences of frames of video data, where each frame of video data comprises color information for a plurality of pixels;

memory containing;

a hand tracking application; and

a plurality of edge feature templates wherein each of the plurality of edge feature templates is a rotated and scaled version of a finger template;

wherein the hand tracking application configures the processor to;

obtain a frame of video data from the reference camera;

detect at least one candidate finger in the frame of video data received from the reference camera, where each of the at least one candidate finger is a grouping of pixels identified by searching the frame of video data (I) for a grouping of pixels that have image gradient orientations that match a given edge feature template () from the plurality of edge feature templates by determining whether the similarity of a number of gradient orientations extracted from the given edge feature template at a list P of locations r to be considered from edge features () in the given edge feature template and image gradient orientations extracted from the frame of video data relative to a pixel location c exceed a predetermined matching threshold using the following similarity measure;

2. The system of claim 1, wherein the edge features of the finger template are synthetically generated.

3. The system of claim 1, wherein the plurality of edge feature templates are stored in a data structure that includes metadata describing the rotation and scaling applied to the finger template to obtain a given edge feature template within the data structure.

4. The system of claim 1, wherein the edge feature template is a binary edge map.

5. The system of claim 1, wherein the edge feature template is a map of image gradient orientations.

6. The system of claim 1, wherein the hand tracking application configures the processor to search the frame of video data for a grouping of pixels that have image gradient orientations that match a given edge feature template from the plurality of edge feature templates by:

selecting a grouping of pixels;

searching within a predetermined neighborhood of pixels relative to each edge feature in the given edge feature template to find the image gradient orientation that is most similar to the image gradient orientation of the edge feature; and

determining the similarity of the grouping of pixels to the given edge feature template based upon a measure of the similarity of the most similar image gradient orientations found within the grouping of pixels for each of the edge features in the given edge feature template.

7. The system of claim 6, wherein the hand tracking application configures the processor to determine image gradient orientation based upon a change in intensity of neighboring pixels in the video data.

8. The system of claim 6, wherein the hand tracking application configures the processor to determine image gradient orientation based upon a change in the color of neighboring pixels in the video data in at least one color channel.

9. The system of claim 8, wherein the video data comprises color information for a plurality of pixels in blue, green and red color channels.

10. The system of claim 9, wherein the hand tracking application configures the processor to determine image gradient orientation by computing the orientation of gradients in red and green color channels separately and using the gradient orientation of the channel whose magnitude is the largest.

11. The system of claim 9, wherein the hand tracking application configures the processor to determine image gradient orientation by computing the orientation of gradients in blue, green, and red color channels separately and using the gradient orientation of the channel whose magnitude is the largest.

12. The system of claim 1, wherein the hand tracking application configures the processor to quantize the image gradient orientations into a predetermined number (n₀) of values that can each be represented by a single bit.

13. The system of claim 12, wherein the hand tracking application configures the processor to calculate the similarity measure by spreading the quantized image gradient orientation at pixel location t (ori(I, t)) in a T×

T window around the pixel location t and encoding the spread image gradient orientations using a binary string of length n₀, where each bit in the string corresponds to each quantized image gradient orientation present in the T×

T window.

14. The system of claim 13, wherein the hand tracking application configures the processor to:

precompute lookup tables of max<sub>tε

R(c+r)</sub>|cos(ori(,r)−

ori(I,t))| for each possible orientation ori(,r) of an edge feature at a given pixel location t in the video data; and

evaluate the similarity measure for the given edge feature template using the precomputed lookup tables.

15. The system of claim 1, wherein the hand tracking application determines the pixel locations (c) searched within the frame of video data using information including the pixel location of a previously identified finger.

16. The system of claim 15, wherein the hand tracking application determines the pixel locations (c) searched using information further comprising the velocity of a previously identified finger.

17. The system in claim 1, where the hand tracking application configures the processor to:

detect the presence and initial location of at least one finger by recognizing an initialization gesture; and

determine the pixel locations (c) searched within the frame of video data using information including the initial location of the at least one finger.

18. The system of claim 1, wherein:

the memory contains video data of a previous frame; and

the hand tracking application configures the processor to determine the pixel locations (c) searched within the frame of video data using information including pixels in the frame of video data that have changed relative to the pixels in the video data of the previous frame.

19. The system of claim 1, wherein:

the finger template further comprises a plurality of surface color pixel sample locations defined relative to the edge features template; and

the hand tracking application configures the processor to confirm that the colors of the pixels within the grouping of pixels identified as a candidate finger satisfy a skin color criterion by confirming that the colors of the surface color pixel sample locations for the edge feature template from the plurality of edge feature templates that matches the grouping of pixels identified as a candidate finger satisfy a skin color criterion.

20. The system of claim 19, wherein the skin color criterion is a number of pixel sample locations having a color corresponding to a predetermined range of colors.

21. The system of claim 19, wherein the skin color criterion is a ratio of pixel sample locations having a color corresponding to a predetermined range of colors relative to the overall number of pixel sample locations.

22. The system of claim 21, wherein the hand tracking application is configured to adapt the skin color criterion based upon at least one characteristic of the given edge feature template selected from the group consisting of the scale, and rotation of the given edge feature template relative to the finger template.

23. The system of claim 1, wherein the hand tracking application further configures the processor to determine the position, rotation, and size of the at least one detected finger and to determine a hand pose from the location, orientation, and size of the at least one detected finger.

24. The system of claim 23, wherein the hand tracking application further configures the processor to verify the correct detection of a candidate finger by confirming that the detected finger belongs to a possible hand pose indicative of a correct identification of a finger.

25. The system of claim 23, wherein the hand tracking application configures the processor to determine a hand pose using a 3D model of a human hand based upon the location and orientation of the at least one detected finger.

26. The system of claim 23, wherein the hand tracking application configures the processor to determine a hand pose using priors concerning relative size of fingers and thumbs of a human hand based upon the location, orientation and size of the at least one detected finger.

27. The system of claim 23, wherein the hand tracking application configures the processor to determine a hand pose using priors concerning the manner in which finger joints enable manipulation of fingers relative to the pose of a human hand based upon the location, orientation and size of the at least one detected finger.

28. The system of claim 1, further comprising:

a display interface configured to drive a display device;

wherein the hand tracking application configures the processor to;

determine a distance and orientation of a detected finger;

map the determined distance and orientation of the detected finger to a location on the display device; and

generate a target on the display device at the mapped location using the display interface.

29. The system of claim 28, wherein the hand tracking application configures the processor to:

determine the size of a tracked finger at a known distance from at least one initial frame of video data obtained from the reference camera during a registration process based upon the scaling of an edge feature template from the plurality of edge feature templates that matches the tracked finger; and

determine the distance to a detected finger in the frame of video data obtained from the reference camera based upon the relative scale of the edge feature template from the plurality of edge feature templates that matches the grouping of pixels corresponding to the detected finger to the scaling of the edge feature template from the plurality of edge feature templates that matched the tracked finger during the registration process.

30. The system of claim 28, further comprising:

a depth sensor;

wherein the hand tracking application configures the processor to;

obtain a depth map from the depth sensor; and

determine a distance to the detected finger using the depth map.

31. The system of claim 28, further comprising:

an alternate view camera configured to capture sequences of frames of video data, where each frame of video data comprises color information for a plurality of pixels;

wherein the hand tracking application configures the processor to;

obtain a frame of video data from the alternate view camera; and

determine a distance to the detected finger using disparity between the location of the detected finger in the frame of video data obtained from the reference camera and the frame of video data obtained from the alternate view camera.

32. The system of claim 1, wherein the hand tracking application configures the processor to:

obtain a sequence of frames of video data from the reference camera; and

determine hand pose based upon the trajectory in the sequence of frames of video data of at least one detected finger.

33. The system of claim 32, wherein the hand tracking application configures the processor to:

identify at least one detected finger forming a static gesture; and

execute a command in response to the identification of the static gesture.

34. The system of claim 32, wherein the hand tracking application configures the processor to:

identify a trajectory in the sequence of frames of video data of at least one detected finger forming a dynamic gesture; and

execute a command in response to the identification of the dynamic gesture.

35. A system for detecting hand posture using parts based template matching to detect individual human fingers, comprising:

a processor;

a reference camera configured to capture sequences of frames of video data, where each frame of video data comprises color information for a plurality of pixels;

a display interface configured to drive a display device;

memory containing;

a hand tracking application; and

a plurality of edge feature templates that are rotated and scaled versions of a finger template, which comprises;

an edge features template; and

a plurality of surface color pixel sample locations defined relative to the edge features template;

wherein the hand tracking application configures the processor to;

obtain a frame of video data from the reference camera;

detect at least one candidate finger in the frame of video data received from the reference camera, where;

each of the at least one candidate finger is a grouping of pixels identified by searching the frame of video data for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates; and

searching the frame of video data for a grouping of pixels that have image gradient orientations that match a given edge feature template () from the plurality of edge feature templates comprises;

selecting a grouping of pixels;

searching within a predetermined neighborhood of pixels relative to each edge feature in the given edge feature template to find the image gradient orientation that is most similar to the image gradient orientation of the edge feature; and

determining whether the similarity of a number of gradient orientations extracted from the given edge feature template at a list P of locations r to be considered from edge features () in the given edge feature template and image gradient orientations extracted from the frame of video data relative to a pixel location c exceed a predetermined matching threshold using the following similarity measure;

36. A method of detecting hand posture by performing parts based template matching to detect individual human fingers, comprising:

capturing a frame of video data using a reference camera, where the frame of video data comprises color information for a plurality of pixels;

detecting at least one candidate finger in the frame of video data received from the reference camera using a processor configured using a hand tracking application, where each of the at least one candidate finger is a grouping of pixels identified by searching the frame of video data (I) for a grouping of pixels that have image gradient orientations that match a given edge feature template () from a plurality of edge feature templates by determining whether the similarity of a number of gradient orientations extracted from the given edge feature template at a list P of locations r to be considered from edge features () in the given edge feature template and image gradient orientations extracted from the frame of video data relative to a pixel location c exceed a predetermined matching threshold using the following similarity measure;