Systems and methods for initializing motion tracking of human hands

• US 8,615,108 B1
• Filed: 07/22/2013
• Issued: 12/24/2013
• Est. Priority Date: 01/30/2013
• Status: Expired due to Fees

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Imimtek, Inc.
• Inventors: Stoppa, Michele, Hummel, Britta, Dal Mutto, Carlo, Pasqualotto, Giuliano
• Primary Examiner(s): Krasnic, Bernard
• Assistant Examiner(s): YANG, WEI WEN
• Application Number:
  US13/948,117
• Time in Patent Office: 155 Days
• Field of Search: 382/103, 382/199, 382/209
• US Class Current: 382/103
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Systems and methods for initializing motion tracking of human hands within bounded regions are disclosed. One embodiment includes: a processor; reference and alternate view cameras; and memory containing a plurality of templates that are rotated and scaled versions of a base template. In addition, a hand tracking application configures the processor to: obtain reference and alternate view frames of video data; generate a depth map; identify at least one bounded region within the reference frame of video data containing pixels having distances from the reference camera that are within a specific range of distances; determine whether any of the pixels within the at least one bounded region are part of a human hand; track the motion of the part of the human hand in a sequence of frames of video data obtained from the reference camera; and confirm that the tracked motion corresponds to a predetermined initialization gesture.

Claims

1. A real-time gesture based interactive system, comprising:

a processor;

a reference camera configured to capture sequences of frames of video data, where each frame of video data comprises intensity information for a plurality of pixels;

memory containing;

a hand tracking application; and

a set of edge feature templates comprising a plurality of edge feature templates that are rotated and scaled versions of a base template; and

wherein the hand tracking application configures the processor to;

obtain a sequence of frames of video data from the reference camera;

compare successive frames of video data from the sequence of frames of video data for pixels that change from one frame to the next;

determine whether any of the pixels that changed are part of a human hand visible in the sequence of frames of video data by searching the frame of video data for a grouping of pixels that have image gradient orientations that match the edge features of one of the plurality of edge feature templates; and

track a motion of the part of the human hand visible in the sequence of frames of video data;

confirm that the tracked motion of the part of the human hand visible in the sequence of frames of video data corresponds to a predetermined initialization gesture;

define a 3D interaction zone relative to the tracked motion of the part of the human hand visible in the sequence of frames of video data corresponding to the predetermined initialization gesture, where the 3D interaction zone corresponds to a bounded region that is less than the frame of video and that contains the part of the human hand visible in the sequence of frames of video data; and

commence tracking the human hand within the 3D interaction zone as part of a gesture based interactive session by performing a search restricted to the bounded region corresponding to the 3D interaction zone in a sequence of frames of video data for a grouping of pixels that have image gradient orientations that match the edge features of one of the plurality of edge feature templates.

2. The real-time gesture based interactive system of claim 1, wherein the hand tracking application configures the processor to identify pixels in successive frames of video that change from one frame to the next by subtracting the successive frames of video to identify pixel value differences exceeding a predetermined threshold.

3. The real-time gesture based interactive system of claim 1, wherein the predetermined initialization gesture comprises a finger oscillating from side to side within a predetermined gesture range.

4. The real-time gesture based interactive system of claim 3, wherein the hand tracking application configures the processor to confirm that the tracked motion of the part of the human hand visible in the sequence of frames of video data corresponds to a predetermined initialization gesture by verifying that the extent of the motion of a human hand visible in the sequence of frames of video data is below a predetermined threshold.

5. The real-time gesture based interactive system of claim 1, wherein the hand tracking application further configures the processor to initialize the image capture settings of the reference camera used during the gesture based interactive session.

6. The real-time gesture based interactive system of claim 5, wherein the hand tracking application configures the processor to:

obtain the sequence of frames of video data from the reference camera using initial image capture settings; and

adjust the exposure and gain of the reference camera as additional frames of video data are captured by the reference camera so that the brightness of at least one pixel that is part of a human hand visible in the additional frames of video data satisfies a predetermined criterion.

7. The real-time gesture based interactive system of claim 6, wherein the hand tracking application configures the processor to determine the initial image capture settings using auto-exposure and auto-gain controls of the reference camera.

8. The real-time gesture based interactive system of claim 7, wherein the hand tracking application configures the processor to determine an initial white balance setting using automatic white balance controls of the reference camera.

9. The real-time gesture based interactive system of claim 1, wherein:

each frame of video data captured by the reference view camera includes color information for a plurality of pixels comprising intensity information in a plurality of color channels; and

the hand tracking application further configures the processor to initialize a skin color classifier used during the gesture based interactive session.

10. The real-time gesture based interactive system of claim 9, wherein the hand tracking application further configures the processor to initialize a skin color classifier based upon the colors of a plurality of pixels that are part of a human hand visible in the sequence of frames of video data.

11. The real-time gesture based interactive system of claim 1, wherein the 3D interaction zone is defined relative to the location of a detected part of a human hand based upon a predetermined gesture range.

12. The real-time gesture based interactive system of claim 1, wherein the hand tracking application further configures the processor to:

determine the size of a part of a human hand visible in the sequence of frames of video data by determining the scale of the edge feature template that matches the part of the human hand visible in the sequence of frames of video data at a known distance from the reference camera;

use the size of the part of a human hand visible in the sequence of frames to determine the distance to the part of a human hand visible in additional frames of video captured by the reference camera during the gesture based interactive session.

13. The real-time gesture based interactive system of claim 1, wherein the hand tracking application configures the processor to search the frame of video data for a grouping of pixels that have image gradient orientations that match a given edge feature template from the plurality of edge feature templates by:

selecting a grouping of pixels;

searching within a predetermined neighborhood of pixels relative to each edge feature in the given edge feature template to find the image gradient orientation that is most similar to the image gradient orientation of the edge feature; and

determining the similarity of the grouping of pixels to the given edge feature template based upon a measure of the similarity of the most similar image gradient orientations found within the grouping of pixels for each of the edge features in the given edge feature template.

14. The real-time gesture based interactive system of claim 13, wherein the hand tracking application configures the processor to determine image gradient orientation based upon a change in intensity of neighboring pixels in the video data.

15. The real-time gesture based interactive system of claim 13, wherein:

each frame of video data captured by the reference view camera includes color information for a plurality of pixels comprising intensity information in a plurality of color channels; and

the hand tracking application configures the processor to determine image gradient orientation based upon a change in the color of neighboring pixels in the video data in at least one color channel.

16. The real-time gesture based interactive system of claim 15, wherein the video data comprises color information for a plurality of pixels in color channels selected from the group consisting of the blue, green, red, infrared, near-infrared, and ultraviolet portions of the spectrum.

17. The real-time gesture based interactive system of claim 1, wherein the hand tracking application configures the processor to search the frame of video data (I) for a grouping of pixels that have image gradient orientations that match a given edge feature template () from the plurality of edge feature templates by determining whether the similarity of a number of gradient orientations extracted from the given edge feature template at a list P of locations r to be considered from edge features () in the given edge feature template and image gradient orientations extracted from the frame of video data relative to a pixel location c exceed a predetermined matching threshold using the following similarity measure:

18. The real-time gesture based interactive system of claim 17, wherein the hand tracking application configures the processor to quantize the image gradient orientations into a predetermined number (n_o) of values that can each be represented by a single bit.

19. The real-time gesture based interactive system of claim 18, wherein the hand tracking application configures the processor to calculate the similarity measure by spreading the quantized image gradient orientation at pixel location t (ori(I,t)) in a T×

T window around the pixel location t and encoding the spread image gradient orientations using a binary string of length n_o, where each bit in the string corresponds to each quantized image gradient orientation present in the T×

T window.

20. The real-time gesture based interactive system of claim 19, wherein the hand tracking application configures the processor to:

precompute lookup tables of max<sub>tε

R(c+r)</sub>|cos(ori(,r)−

ori(I,t))| for each possible orientation ori(,r) of an edge feature at a given pixel location t in the video image data; and

evaluate the similarity measure for the given edge feature template using the precomputed lookup tables.

21. The real-time gesture based interactive system of claim 17, wherein the hand tracking application determines the pixel locations (c) searched within the frame of video data using information selected from the group consisting of:

the pixel location of a previously identified finger; and

the velocity of a previously identified finger.

22. The real-time gesture based interactive system of claim 17, where the hand tracking application configures the processor to:

detect the presence and initial location of at least one finger by recognizing an initialization gesture; and

determine the pixel locations (c) searched within the frame of video data using information including the initial location of the at least one finger.

23. The real-time gesture based interactive system of claim 17, wherein:

the memory contains video data of a previous frame; and

the hand tracking application configures the processor to determine the pixel locations (c) searched within the frame of video data using information including pixels in the frame of video data that have changed relative to the pixels in the video data of the previous frame.

24. The real-time gesture based interactive system of claim 1, further comprising:

an alternate view camera configured to capture sequences of frames of video data, where each frame of video data comprises intensity information for a plurality of pixels;

wherein the hand tracking application further configures the processor to;

obtain a sequence of frames of video from the alternate view camera; and

verify that a part of a human hand visible in the sequence of frames of video data obtained from the reference camera is part of a human hand by locating a grouping of pixels in the alternate view frame of video data that correspond to the part of a human hand.

25. The real-time gesture based interactive system of claim 24, wherein

the hand tracking application configures the processor to:

determine whether any of the pixels that changed are part of a human hand visible in the sequence of frames of video data by searching the frame of video data for a grouping of pixels that have image gradient orientations that match the edge features of one of the plurality of edge feature templates; and

locate a grouping of pixels in the alternate view frame of video data that correspond to the part of a human hand by searching along an epipolar line within the alternate view image for a grouping of pixels that have image gradient orientations that match one of the plurality of edge feature templates, where the epipolar line is defined by the relative location of the center of the reference camera and the center of the alternate view camera.

29. The real-time gesture based interactive system of claim 1, wherein the 3D interaction zone corresponds to a 3D zone that is no greater than 160 mm×

90 mm×

200 mm.

26. A real-time gesture based interactive system, comprising:

a processor;

a reference camera configured to capture sequences of frames of video data, where each frame of video data comprises color information for a plurality of pixels;

memory containing;

a hand tracking application; and

a set of edge feature templates comprising a plurality of edge feature templates that are rotated and scaled versions of a base template;

wherein the hand tracking application configures the processor to;

obtain a sequence of frames of video data from the reference camera using initial image capture settings;

compare successive frames of video data from the sequence of frames of video data for pixels that change from one frame to the next;

determine whether any of the pixels that changed are part of a human hand visible in the sequence of frames of video data, where a part of a human hand is identified by searching the frame of video data for a grouping of pixels that have image gradient orientations that match the edge features of one of the plurality of edge feature templates; and

track a motion of the part of the human hand visible in the sequence of frames of video data;

confirm that the tracked motion of the part of the human hand visible in the sequence of frames of video data corresponds to a predetermined initialization gesture, where the predetermined initialization gesture comprises a finger oscillating from side to side within a predetermined gesture range;

initialize the image capture settings of the reference camera used during the gesture based interactive session by adjusting the exposure and gain of the reference camera as additional frames of video data are captured by the reference camera so that the brightness of at least one pixel that is part of a human hand visible in the additional frames of video data satisfies a predetermined criterion;

define a 3D interaction zone relative to the tracked motion of the part of the human hand visible in the sequence of frames of video data corresponding to the predetermined initialization gesture, where the 3D interaction zone corresponds to a bounded region that is less than the frame of video and that contains the part of the human hand visible in the sequence of frames of video data; and

commence tracking the human hand within the 3D interaction zone as part of a gesture based interactive session performing a search restricted to the bounded region corresponding to the 3D interaction zone in a sequence of frames of video data for a grouping of pixels that have image gradient orientations that match the edge features of one of the plurality of edge feature templates.

30. The real-time gesture based interactive system of claim 26, wherein the 3D interaction zone corresponds to a 3D zone that is no greater than 160 mm×

90 mm×

200 mm.

27. A method of commencing tracking of a human hand using a real-time gesture based interactive system and a set of edge feature templates comprising a plurality of edge feature templates that are rotated and scaled versions of a base template, the method comprising:

obtaining a sequence of frames of video data from the reference camera;

comparing successive frames of video data from the sequence of frames of video data for pixels that change from one frame to the next using a processor configured by a hand tracking application;

determining whether any of the pixels that changed are part of a human hand visible in the sequence of frames of video data using the processor configured by the hand tracking application, where a part of a human hand is identified by searching the frame of video data for a grouping of pixels that have image gradient orientations that match the edge features of one of the plurality of edge feature templates; and

tracking a motion of the part of the human hand visible in the sequence of frames of video data using the processor configured by the hand tracking application;

confirming that the tracked motion of the part of the human hand visible in the sequence of frames of video data corresponds to a predetermined initialization gesture using the processor configured by the hand tracking application;

defining a 3D interaction zone relative to the tracked motion of the part of the human hand visible in the sequence of frames of video data corresponding to the predetermined initialization gesture using the processor configured by the hand tracking application, where the 3D interaction zone corresponds to a bounded region that is less than the frame of video and that contains the part of the human hand visible in the sequence of frames of video data; and

commence tracking the human hand within the 3D interaction zone as part of a gesture based interactive session by performing a search of the bounded region corresponding to the 3D interaction zone in a sequence of frames of video data for a grouping of pixels that have image gradient orientations that match the edge features of one of the plurality of edge feature templates using the processor configured by the hand tracking application.

28. The method of claim 27, wherein the 3D interaction zone corresponds to a 3D zone that is no greater than 160 mm×

90 mm×

200 mm.