Systems and methods for initializing motion tracking of human hands using template matching within bounded regions determined using a depth map

• US 9,129,155 B2
• Filed: 06/11/2013
• Issued: 09/08/2015
• Est. Priority Date: 01/30/2013
• Status: Active Grant

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Aquifi, Inc.
• Inventors: Hummel, Britta, Dal Mutto, Carlo, Pasqualotto, Giuliano, Stoppa, Michele
• Primary Examiner(s): ANSARI, TAHMINA N
• Application Number:
  US13/915,532
• Publication Number:
  US 20140211992A1
• Time in Patent Office: 819 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Systems and methods for initializing motion tracking of human hands within bounded regions are disclosed. One embodiment includes: a processor; reference and alternate view cameras; and memory containing a plurality of templates that are rotated and scaled versions of a base template. In addition, a hand tracking application configures the processor to: obtain reference and alternate view frames of video data; generate a depth map; identify at least one bounded region within the reference frame of video data containing pixels having distances from the reference camera that are within a specific range of distances; determine whether any of the pixels within the at least one bounded region are part of a human hand; track the motion of the part of the human hand in a sequence of frames of video data obtained from the reference camera; and confirm that the tracked motion corresponds to a predetermined initialization gesture.

Claims

1. A real-time gesture based interactive system, comprising:

a processor;

a reference camera configured to capture sequences of frames of video data, where each frame of video data comprises intensity information for a plurality of pixels;

an alternate view camera configured to capture sequences of frames of video data, where each frame of video data comprises intensity information for a plurality of pixels; and

memory containing a hand tracking application; and

wherein the hand tracking application configures the processor to;

obtain a reference frame of video data from the reference camera where the reference frame of video data is part of a sequence of frames of video data obtained from the reference camera;

obtain an alternate view frame of video data from the alternate view camera;

identify moving pixels by comparing a previous frame of video data from the sequence of frames of video data with the reference frame of video data to identify pixel value differences exceeding a predetermined threshold;

generate a depth map containing distances from the reference camera for pixels in the reference frame of video data using information including the disparity between corresponding pixels within the reference and alternate view frames of video data;

identify at least one bounded region within the reference frame of video data containing moving pixels having distances from the reference camera that are within a specific range of distances from the reference camera by;

identifying at least one preliminary bounded region within the reference frame of video data containing pixels that are moving,

generating the depth map based upon the identified at least one preliminary bounded region in the reference frame of video data so that the depth map contains distances from the reference camera for pixels within the at least one preliminary bounded region in the reference frame of video data, and

identifying the at least one bounded region within the at least one preliminary bounded region in the reference frame of video data using the depth map;

determine whether any of the pixels within the at least one bounded region within the reference frame are part of a human hand;

obtain the sequence of frames of video data from the reference camera;

track the motion of the part of the human hand visible in the sequence of frames of video data;

confirm that the tracked motion of the part of the human hand visible in the sequence of frames of video data corresponds to a predetermined initialization gesture; and

commence tracking the human hand as part of a gesture based interactive session.

2. The real-time gesture based interactive system of claim 1, wherein the hand tracking application further configures the processor to verify that the part of a human hand visible in the sequence of frames of video data obtained from the reference camera is part of a human hand by locating a grouping of pixels in the alternate view frame of video data that correspond to the part of a human hand.

3. The real-time gesture based interactive system of claim 1, wherein the at least one bounded region is selected from the group consisting of:

a bounding rectangle; and

a mask that indicates the pixels within the reference frame of video data that are included in the at least one bounded region.

4. The real-time gesture based interactive system of claim 1, wherein the depth map contains distances from the reference camera for every pixel in the reference frame of video data.

5. The real-time gesture based interactive system of claim 1, wherein the depth map contains distances from the reference camera for a number of pixels in the reference frame of video data that is less than the total number of pixels in the reference frame of video data.

6. The real-time gesture based interactive system of claim 1, wherein the hand tracking application further configures the processor to:

obtain a third frame of video data from the reference camera;

identify pixels that occupy the same pixel locations in each of the reference and third frames of video data; and

identify at least one bounded region within the reference frame of video data containing pixels having corresponding pixels in the previous frame of video data or the third frame of video data with intensity values that differ by a predetermined amount.

7. The real-time gesture based interactive system of claim 1, wherein the predetermined initialization gesture comprises a finger oscillating from side to side within a predetermined gesture range.

8. The real-time gesture based interactive system of claim 1, wherein:

the at least one bounded region within the reference frame of video data containing pixels having distances from the reference camera that are within a specific range of distances from the reference camera includes a candidate part of a human hand comprising a cluster of pixels that are within a specific range of distances from the reference camera; and

the hand tracking application configures the processor to confirm that the tracked motion of the part of the human hand visible in the sequence of frames of video data corresponds to a predetermined initialization gesture by;

determining an optical flow of the candidate part of a human hand by comparing frames of video data in the sequence of frames of video data from the reference camera; and

confirming that the optical flow of the candidate part of a human hand corresponds to a predetermined motion sequence.

9. The real-time gesture based interactive system of claim 8, wherein the predetermined motion sequence comprises:

movement in a first direction;

followed by movement in a second opposite direction; and

followed by movement in the first direction.

10. The real-time gesture based interactive system of claim 1, wherein the hand tracking application further configures the processor to initialize the image capture settings of the reference camera used during the gesture based interactive session.

11. The real-time gesture based interactive system of claim 10, wherein the hand tracking application configures the processor to:

obtain the sequence of frames of video data from the reference camera using initial image capture settings; and

adjust the exposure and gain of the reference camera as additional frames of video data are captured by the reference camera so that the brightness of at least one pixel that is part of a human hand visible in the additional frames of video data satisfies a predetermined criterion.

12. The real-time gesture based interactive system of claim 11, wherein the hand tracking application configures the processor to determine the initial image capture settings using auto-exposure and auto-gain controls of the reference camera.

13. The real-time gesture based interactive system of claim 12, wherein the hand tracking application configures the processor to determine an initial white balance setting using automatic white balance controls of the reference camera.

14. The real-time gesture based interactive system of claim 11, wherein the hand tracking application configures the processor to adjust the exposure and gain of the reference camera as additional frames of video data are captured by the reference camera by adjusting the exposure and gain of the reference camera as additional frames of video data are captured by the reference camera so that the brightest pixel that is part of a human hand visible in the additional frames of video data is a predetermined brightness value.

15. The real-time gesture based interactive system of claim 1, wherein:

each frame of video data captured by the reference view camera includes information for a plurality of pixels comprising intensity information in a plurality of channels selected from the group consisting of blue, green, red, infrared, near-infrared, and ultraviolet portions of the spectrum; and

the hand tracking application further configures the processor to initialize a skin color classifier used during the gesture based interactive session.

16. The real-time gesture based interactive system of claim 15, wherein the hand tracking application further configures the processor to initialize the skin color classifier based upon the intensity information of a plurality of pixels that are part of a human hand visible in the sequence of frames of video data.

17. The real-time gesture based interactive system of claim 15, wherein the hand tracking application further configures the processor to initialize the skin color classifier based upon the intensity information of a plurality of pixels that are part of a human hand visible in the sequence of frames of video data by forming a histogram of the colors of the plurality of pixels.

18. The real-time gesture based interactive system of claim 17, wherein the hand tracking application further configures the processor to determine skin color information by identifying the pixel colors within the histogram satisfying a predetermined skin tone criterion.

19. A real-time gesture based interactive system comprising:

a processor;

a reference camera configured to capture sequences of frames of video data, where each frame of video data comprises intensity information for a plurality of pixels;

an alternate view camera configured to capture sequences of frames of video data, where each frame of video data comprises intensity information for a plurality of pixels; and

memory containing a hand tracking application and a plurality of templates that are rotated and scaled versions of a base template; and

wherein the hand tracking application configures the processor to;

obtain a reference frame of video data from the reference camera;

obtain an alternate view frame of video data from the alternate view camera;

generate a depth map containing distances from the reference camera for pixels in the reference frame of video data using information including the disparity between corresponding pixels within the reference and alternate view frames of video data; and

identify at least one bounded region within the reference frame of video data containing pixels having distances from the reference camera that are within a specific range of distances from the reference camera;

determine whether any of the pixels within the at least one bounded region within the reference frame are part of a human hand by searching the frame of video data for a grouping of pixels that match one of the plurality of templates;

obtain a sequence of frames of video data from the reference camera;

track the motion of the part of the human hand visible in the sequence of frames of video data;

confirm that the tracked motion of the part of the human hand visible in the sequence of frames of video data corresponds to a predetermined initialization gesture; and

commence tracking the human hand as part of a gesture based interactive session.

20. The real-time gesture based interactive system of claim 19, wherein the base template is an edge feature template.

21. The real-time gesture based interactive system of claim 19, wherein the hand tracking application configures the processor to search the frame of video data for a grouping of pixels that have image gradient orientations that match a given edge feature template from the plurality of edge feature templates by:

selecting a grouping of pixels;

searching within a predetermined neighborhood of pixels relative to each edge feature in the given edge feature template to find the image gradient orientation that is most similar to the image gradient orientation of the edge feature; and

determining the similarity of the grouping of pixels to the given edge feature template based upon a measure of the similarity of the most similar image gradient orientations found within the grouping of pixels for each of the edge features in the given edge feature template.

22. The real-time gesture based interactive system of claim 19, wherein the hand tracking application configures the processor to search the frame of video data (I) for a grouping of pixels that have image gradient orientations that match a given edge feature template (T) from the plurality of edge feature templates by determining whether the similarity of a number of gradient orientations extracted from the given edge feature template at a list P of locations r to be considered from edge features (O) in the given edge feature template and image gradient orientations extracted from the frame of video data relative to a pixel location c exceed a predetermined matching threshold using the following similarity measure:

23. The real-time gesture based interactive system of claim 19, wherein:

the hand tracking application further configures the processor to verify that a part of a human hand visible in the sequence of frames of video data obtained from the reference camera is part of a human hand by locating a grouping of pixels in the alternate view frame of video data that correspond to the part of a human hand; and

the hand tracking application is configured to locate a grouping of pixels in the alternate view frame of video data that correspond to the part of a human hand by searching along an epipolar line within the alternate view image for a grouping of pixels that match one of the plurality of templates, where the epipolar line is defined by the relative location of the center of the reference camera and the center of the alternate view camera.

24. The system of claim 23, wherein the hand tracking application is configured to search a distance along the epipolar line within the alternate view frame of video data for a grouping of pixels that match one of the plurality of templates based upon the distance of the part of a human hand visible in the sequence of frames of video data obtained from the reference camera.

25. The system of claim 23, wherein the hand tracking application is configured to search a predetermined range of distances along the epipolar line within the alternate view frame of video data for a grouping of pixels that match one of the plurality of templates, where the predetermined range of distances is determined relative to a disparity determined based upon the distance from the reference camera of the part of a human hand visible in the sequence of frames of video data obtained from the reference camera.

26. The real-time gesture based interactive system of claim 23, wherein the hand tracking application is configured to search along an epipolar line within the alternate view frame of video for a grouping of pixels that match one of the plurality of templates by performing a search with respect to pixels within a predetermined margin relative to the epipolar line.

27. A real-time gesture based interactive system, comprising:

a processor;

a reference camera configured to capture sequences of frames of video data, where each frame of video data comprises color information for a plurality of pixels;

an alternate view camera configured to capture sequences of frames of video data, where each frame of video data comprises color information for a plurality of pixels; and

memory containing;

a hand tracking application; and

a set of edge feature templates comprising a plurality of edge feature templates that are rotated and scaled versions of a base template;

wherein the hand tracking application configures the processor to;

obtain a reference frame of video data from the reference camera;

obtain an alternate view frame of video data from the alternate view camera;

generate a depth map containing distances from the reference camera for pixels in the reference frame of video data using information including the disparity between corresponding pixels within the reference and alternate view frames of video data;

identify at least one bounded region within the reference frame of video data containing pixels having distances from the reference camera that are within a specific range of distances from the reference camera;

determine whether any of the pixels within the at least one bounded region within the reference frame are part of a human hand visible in the sequence of frames of video data, where a part of a human hand is identified by searching the frame of video data for a grouping of pixels that have image gradient orientations that match the edge features of one of the plurality of edge feature templates;

obtain a sequence of frames of video data from the reference camera;

track the motion of the part of the human hand visible in the sequence of frames of video data;

confirm that the tracked motion of the part of the human hand visible in the sequence of frames of video data corresponds to a predetermined initialization gesture, where the predetermined initialization gesture comprises a finger oscillating from side to side within a predetermined gesture range;

initialize the image capture settings of the reference camera used during the gesture based interactive session by adjusting the exposure and gain of the reference camera as additional frames of video data are captured by the reference camera so that the brightness of at least one pixel that is part of a human hand visible in the additional frames of video data satisfies a predetermined criterion; and

commence tracking the human hand as part of a gesture based interactive session.

28. A method of commencing tracking of a human hand using a real-time gesture based interactive system, comprising:

capturing a reference frame of video data using a reference camera, where the reference frame of video data comprises intensity information for a plurality of pixels;

capturing an alternate view frame of video data using an alternate view camera, where the alternate view frame of video data comprises intensity information for a plurality of pixels;

generating a depth map containing distances from the reference camera for pixels in the reference frame of video data using a processor configured by a hand tracking application and information including the disparity between corresponding pixels within the reference and alternate view frames of video data;

identifying at least one bounded region within the reference frame of video data containing pixels having distances from the reference camera that are within a specific range of distances from the reference camera using the processor configured by the hand tracking application;

determining whether any of the pixels within the at least one bounded region within the reference frame are part of a human hand visible in the reference frame of video data using the processor configured using the hand tracking application, where a part of a human hand is identified by searching the reference frame of video data for a grouping of pixels that have image gradient orientations that match the edge features of one of the plurality of edge feature templates;

obtaining a sequence of frames of video data from the reference camera;

tracking the motion of the part of the human hand visible in the sequence of frames of video data using the processor configured using the hand tracking application;

confirming that the tracked motion of the part of the human hand visible in the sequence of frames of video data corresponds to a predetermined initialization gesture using the processor configured using the hand tracking application; and

commence tracking the human hand as part of a gesture based interactive session using the processor configured using the hand tracking application.