Systems and methods for initializing motion tracking of human hands

• US 9,092,665 B2
• Filed: 05/22/2013
• Issued: 07/28/2015
• Est. Priority Date: 01/30/2013
• Status: Active Grant

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Aquifi, Inc.
• Inventors: Stoppa, Michele, Hummel, Britta, Dal Mutto, Carlo, Pasqualotto, Giuliano
• Primary Examiner(s): Fujita, Katrina
• Application Number:
  US13/900,015
• Publication Number:
  US 20140211991A1
• Time in Patent Office: 797 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Systems and methods for initializing motion tracking of human hands are disclosed. One embodiment includes a processor; a reference camera; and memory containing: a hand tracking application; and a plurality of edge feature templates that are rotated and scaled versions of a base template. The hand tracking application configures the processor to: determine whether any pixels in a frame of video are part of a human hand, where a part of a human hand is identified by searching the frame of video data for a grouping of pixels that have image gradient orientations that match the edge features of one of the plurality of edge feature templates; track the motion of the part of the human hand visible in a sequence of frames of video; confirm that the tracked motion corresponds to an initialization gesture; and commence tracking the human hand as part of a gesture based interactive session.

Claims

1. A real-time gesture based interactive system, comprising:

a processor;

a reference camera configured to capture sequences of frames of video data, where each frame of video data comprises intensity information for a plurality of pixels;

memory containing;

a hand tracking application; and

a set of edge feature templates comprising a plurality of edge feature templates that are rotated and scaled versions of a base template;

wherein the hand tracking application configures the processor to;

obtain a sequence of frames of video data from the reference camera;

compare successive frames of video data from the sequence of frames of video data for pixels that are moving;

determine whether any of the pixels that changed are part of a human hand visible in the sequence of frames of video data, where a part of a human hand is identified by searching the frame of video data for a grouping of pixels that have image gradient orientations that match the edge features of one of the plurality of edge feature templates; and

track the motion of the part of the human hand visible in the sequence of frames of video data;

confirm that the tracked motion of the part of the human hand visible in the sequence of frames of video data corresponds to a predetermined initialization gesture, wherein the predetermined initialization gesture comprises a finger oscillating from side to side within a predetermined gesture range; and

commence tracking the human hand as part of a gesture based interactive session.

2. The real-time gesture based interactive system of claim 1, wherein the hand tracking application configures the processor to identify pixels in successive frames of video that change from one frame to the next by subtracting the successive frames of video to identify pixel value differences exceeding a predetermined threshold.

3. The real-time gesture based interactive system of claim 1, wherein the hand tracking application configures the processor to confirm that the tracked motion of the part of the human hand visible in the sequence of frames of video data corresponds to a predetermined initialization gesture by verifying that the extent of the motion of a human hand visible in the sequence of frames of video data is below a predetermined threshold.

4. The real-time gesture based interactive system of claim 1, wherein the hand tracking application further configures the processor to initialize the image capture settings of the reference camera used during the gesture based interactive session.

5. The real-time gesture based interactive system of claim 4, wherein the hand tracking application configures the processor to:

obtain the sequence of frames of video data from the reference camera using initial image capture settings; and

adjust the exposure and gain of the reference camera as additional frames of video data are captured by the reference camera so that the brightness of at least one pixel that is part of a human hand visible in the additional frames of video data satisfies a predetermined criterion.

6. The real-time gesture based interactive system of claim 5, wherein the hand tracking application configures the processor to determine the initial image capture settings using auto-exposure and auto-gain controls of the reference camera.

7. The real-time gesture based interactive system of claim 6, wherein the hand tracking application configures the processor to determine an initial white balance setting using automatic white balance controls of the reference camera.

8. The real-time gesture based interactive system of claim 5, wherein the hand tracking application configures the processor to adjust the exposure and gain of the reference camera as additional frames of video data are captured by the reference camera by adjusting the exposure and gain of the reference camera as additional frames of video data are captured by the reference camera so that the brightest pixel that remains after performing outlier rejection and that is part of a human hand visible in the additional frames of video data is a predetermined brightness value.

9. The real-time gesture based interactive system of claim 8, wherein the hand tracking application configures the processor to determine the brightest pixel remains after performing outlier rejection and that is part of a human hand visible in one of the additional frames of video data by performing a search within a region defined by the one of the plurality of edge feature templates used to identify the part of the human hand visible in the additional frame of video data.

10. The real-time gesture based interactive system of claim 5, wherein the hand tracking application configures the processor to adjust the exposure and gain of the reference camera as additional frames of video data are captured by the reference camera so that pixels that are part of a human hand visible in the additional frames of video data satisfy a predetermined brightness criterion.

11. The real-time gesture based interactive system of claim 5, wherein the hand tracking application configures the processor to adjust the exposure and gain of the reference camera as additional frames of video data are captured by the reference camera so that pixels that are part of a human hand visible in the additional frames of video data satisfy a predetermined average brightness criterion.

12. The real-time gesture based interactive system of claim 5, wherein the hand tracking application configures the processor to adjust the exposure and gain of the reference camera as additional frames of video data are captured by the reference camera so that the darkest pixel that remains after performing outlier rejection and that is part of a human hand visible in the additional frames of video data satisfies a predetermined brightness criterion.

13. The real-time gesture based interactive system of claim 5, wherein the hand tracking application configures the processor to adjust the white level balance of the reference camera based upon the exposure and gain settings of the reference camera.

14. The real-time gesture based interactive system of claim 1, wherein:

each frame of video data captured by the reference view camera includes color information for a plurality of pixels comprising intensity information in a plurality of color channels; and

the hand tracking application further configures the processor to initialize a skin color classifier used during the gesture based interactive session.

15. The real-time gesture based interactive system of claim 14, wherein the hand tracking application further configures the processor to initialize a skin color classifier based upon the colors of a plurality of pixels that are part of a human hand visible in the sequence of frames of video data.

16. The real-time gesture based interactive system of claim 15, wherein the hand tracking application further configures the processor to initialize a skin color classifier based upon the colors of a plurality of pixels that are part of a human hand visible in the sequence of frames of video data by forming a histogram of the colors of the plurality of pixels.

17. The real-time gesture based interactive system of claim 16, wherein the histogram is a two dimensional histogram of pixel intensity values in Red and Green color channels.

18. The real-time gesture based interactive system of claim 16, wherein the histogram is determined using normalized intensities in the Red channel R/(R+G+B) and normalized intensities in the Green channel R/(R+G+B).

19. The real-time gesture based interactive system of claim 16, wherein the hand tracking application further configures the processor to determine skin color information by identifying the pixel colors within the histogram satisfying a predetermined skin tone criterion.

20. The real-time gesture based interactive system of claim 19, wherein the hand tracking application further configures the processor to apply a threshold to the histogram to determine pixel intensities in Red and Green color channels that will be considered to correspond to skin colors.

21. The real-time gesture based interactive system of claim 19, wherein the criterion involves the pixel color being within a threshold distance of at least one selected from the group consisting of the mean, median and mode pixel color of the histogram.

22. The real-time gesture based interactive system of claim 19, wherein the criterion involves the pixel colors of a threshold percentage of pixels within the histogram being considered to be skin colors.

23. The real-time gesture based interactive system of claim 1, wherein the hand tracking application further configures the processor to define a 3D interaction zone relative to the motion of the part of the human hand visible in the sequence of frames of video data.

24. The real-time gesture based interactive system of claim 23, wherein the hand tracking application configures the processor to identify gestures within an interaction zone during a gesture based interactive session.

25. The real-time gesture based interactive system of claim 23, wherein the interaction zone is defined relative to the location of a detected part of a human hand based upon a predetermined gesture range.

26. The real-time gesture based interactive system of claim 1, wherein the hand tracking application further configures the processor to:

determine the size of a part of a human hand visible in the sequence of frames of video data by determining the scale of the edge feature template that matches the part of the human hand visible in the sequence of frames of video data at a known distance from the reference camera; and

use the size of the part of a human hand visible in the sequence of frames to determine the distance to the part of a human hand visible in additional frames of video captured by the reference camera during the gesture based interactive session.

27. The real-time gesture based interactive system of claim 1 wherein the edge features of the base template are synthetically generated.

28. The real-time gesture based interactive system of claim 1, wherein the plurality of edge feature templates are stored in a data structure that includes metadata describing the rotation and scaling applied to the finger template to obtain a given edge feature template within the data structure.

29. The real-time gesture based interactive system of claim 1, wherein the hand tracking application configures the processor to search the frame of video data for a grouping of pixels that have image gradient orientations that match a given edge feature template from the plurality of edge feature templates by:

selecting a grouping of pixels;

searching within a predetermined neighborhood of pixels relative to each edge feature in the given edge feature template to find the image gradient orientation that is most similar to the image gradient orientation of the edge feature; and

determining the similarity of the grouping of pixels to the given edge feature template based upon a measure of the similarity of the most similar image gradient orientations found within the grouping of pixels for each of the edge features in the given edge feature template.

30. The real-time gesture based interactive system of claim 29, wherein the hand tracking application configures the processor to determine image gradient orientation based upon a change in intensity of neighboring pixels in the video data.

31. The real-time gesture based interactive system of claim 29, wherein:

each frame of video data captured by the reference view camera includes color information for a plurality of pixels comprising intensity information in a plurality of color channels; and

the hand tracking application configures the processor to determine image gradient orientation based upon a change in the color of neighboring pixels in the video data in at least one color channel.

32. The real-time gesture based interactive system of claim 31, wherein the video data comprises color information for a plurality of pixels in blue, green and red color channels.

33. The real-time gesture based interactive system of claim 31, wherein the hand tracking application configures the processor to determine image gradient orientation by computing the orientation of gradients in red and green color channels separately and using the gradient orientation of the channel whose magnitude is the largest.

34. The real-time gesture based interactive system of claim 31, wherein the hand tracking application configures the processor to determine image gradient orientation by computing the orientation of gradients in blue, green, and red color channels separately and using the gradient orientation of the channel whose magnitude is the largest.

35. The real-time gesture based interactive system of claim 1 wherein the hand tracking application configures the processor to search the frame of video data (I) for a grouping of pixels that have image gradient orientations that match a given edge feature template () from the plurality of edge feature templates by determining whether the similarity of a number of gradient orientations extracted from the given edge feature template at a list P of locations r to be considered from edge features () in the given edge feature template and image gradient orientations extracted from the frame of video data relative to a pixel location c exceed a predetermined matching threshold using the following similarity measure:

36. The real-time gesture based interactive system of claim 35, wherein the hand tracking application configures the processor to quantize the image gradient orientations into a predetermined number (n_o) of values that can each be represented by a single bit.

37. The real-time gesture based interactive system of claim 36, wherein the hand tracking application configures the processor to calculate the similarity measure by spreading the quantized image gradient orientation at pixel location t (ori(l,t)) in a T×

T window around the pixel location t and encoding the spread image gradient orientations using a binary string of length n_o, where each bit in the string corresponds to each quantized image gradient orientation present in the T×

T window.

38. The real-time gesture based interactive system of claim 37, wherein the hand tracking application configures the processor to:

precompute lookup tables of max<sub>tε

R(c+r)</sub>|cos(ori(,r)−

ori(l,t))| for each possible orientation ori(,r) of an edge feature at a given pixel location t in the video image data; and

evaluate the similarity measure for the given edge feature template using the precomputed lookup tables.

39. The real-time gesture based interactive system of claim 35, wherein the hand tracking application determines the pixel locations (c) searched within the frame of video data using information including the pixel location of a previously identified finger.

40. The real-time gesture based interactive system of claim 39, wherein the hand tracking application determines the pixel locations (c) searched using information further comprising the velocity of a previously identified finger.

41. The real-time gesture based interactive system of claim 35, where the hand tracking application configures the processor to:

detect the presence and initial location of at least one finger by recognizing an initialization gesture; and

determine the pixel locations (c) searched within the frame of video data using information including the initial location of the at least one finger.

42. The real-time gesture based interactive system of claim 35, wherein:

the memory contains video data of a previous frame; and

the hand tracking application configures the processor to determine the pixel locations (c) searched within the frame of video data using information including pixels in the frame of video data that have changed relative to the pixels in the video data of the previous frame.

43. The real-time gesture based interactive system of claim 1, further comprising:

an alternate view camera configured to capture sequences of frames of video data, where each frame of video data comprises intensity information for a plurality of pixels;

wherein the hand tracking application further configures the processor to;

obtain a sequence of frames of video from the alternate view camera; and

verify that a part of a human hand visible in the sequence of frames of video data obtained from the reference camera is part of a human hand by locating a grouping of pixels in the alternate view frame of video data that correspond to the part of a human hand.

44. The real-time gesture based interactive system of claim 43, wherein the hand tracking application configures the processor to locate a grouping of pixels in the alternate view frame of video data that correspond to the part of a human hand by searching along an epipolar line within the alternate view image for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates, where the epipolar line is defined by the relative location of the center of the reference camera and the center of the alternate view camera.

45. The real-time gesture based interactive system of claim 44, wherein the hand tracking application is configured to search along an epipolar line within the alternate view frame of video for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates by performing a search with respect to pixels within a predetermined margin relative to the epipolar line.

46. The real-time gesture based interactive system of claim 44, wherein the hand tracking application is configured to search along an epipolar line within the alternate view frame of video for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates that is rotated and scaled within predetermined limits relative to the base template.

47. The real-time gesture based interactive system of claim 44, wherein the hand tracking application is configured to search along an epipolar line within the alternate view frame of video for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates that is rotated and scaled within permitted limits relative to the base template that are determined based upon the disparity between the grouping of pixels in the alternate view image and the candidate finger in the reference image.

48. The real-time gesture based interactive system of claim 47, wherein the permitted rotation limits increase with increased disparity.

49. The real-time gesture based interactive system of claim 42, wherein the distance from the reference camera to the part of a human hand is determined based upon the disparity between the part of the human hand visible in the sequence of frames of video data obtained from the reference camera and the grouping of pixels in the frames of video data obtained from the alternate view camera that correspond to the part of the human hand.

50. The real-time gesture based interactive system of claim 49, wherein the hand tracking application is configured to search along the epipolar line within the alternate view frame of video to obtain disparity measurements with pixel precision.

51. The real-time gesture based interactive system of claim 49, wherein the hand tracking application is configured to perform a search along the epipolar line within the alternate view frame of video involving resampling the alternate view frame of video to obtain disparity measurements with sub-pixel precision.

52. The real-time gesture based interactive system of claim 49, wherein the hand tracking application is configured to rectify the alternate view frame of video relative to the reference frame of video.

53. The real-time gesture based interactive system of claim 44, wherein the hand tracking application is configured to use calibration to perform rectification of pixels that lie in regions surrounding the epipolar line in the alternate view frame of video data to search for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates.

54. A real-time gesture based interactive system, comprising:

a processor;

a reference camera configured to capture sequences of frames of video data, where each frame of video data comprises color information for a plurality of pixels;

memory containing;

a hand tracking application; and

a set of edge feature templates comprising a plurality of edge feature templates that are rotated and scaled versions of a base template;

wherein the hand tracking application configures the processor to;

obtain a sequence of frames of video data from the reference camera using initial image capture settings;

compare successive frames of video data from the sequence of frames of video data for pixels that change from one frame to the next;

determine whether any of the pixels that changed are part of a human hand visible in the sequence of frames of video data, where a part of a human hand is identified by searching the frame of video data for a grouping of pixels that have image gradient orientations that match the edge features of one of the plurality of edge feature templates; and

track the motion of the part of the human hand visible in the sequence of frames of video data;

confirm that the tracked motion of the part of the human hand visible in the sequence of frames of video data corresponds to a predetermined initialization gesture, where the predetermined initialization gesture comprises a finger oscillating from side to side within a predetermined gesture range;

initialize the image capture settings of the reference camera used during the gesture based interactive session by adjusting the exposure and gain of the reference camera as additional frames of video data are captured by the reference camera so that the brightness of at least one pixel that is part of a human hand visible in the additional frames of video data satisfies a predetermined criterion; and

commence tracking the human hand as part of a gesture based interactive session.

55. A method of commencing tracking of a human hand using a real-time gesture based interactive system, comprising:

obtaining a sequence of frames of video data from the reference camera;

comparing successive frames of video data from the sequence of frames of video data for pixels that change from one frame to the next using a processor configured using a hand tracking application;

determining whether any of the pixels that changed are part of a human hand visible in the sequence of frames of video data using the processor configured using the hand tracking application, where a part of a human hand is identified by searching the frame of video data for a grouping of pixels that have image gradient orientations that match the edge features of one of the plurality of edge feature templates; and

tracking the motion of the part of the human hand visible in the sequence of frames of video data using the processor configured using the hand tracking application;

confirming that the tracked motion of the part of the human hand visible in the sequence of frames of video data corresponds to a predetermined initialization gesture using the processor configured using the hand tracking application, wherein the predetermined initialization gesture comprises a finger oscillating from side to side within a predetermined gesture range; and

commence tracking the human hand as part of a gesture based interactive session using the processor configured using the hand tracking application.