Hand gesture detection

• US 8,750,573 B2
• Filed: 09/16/2010
• Issued: 06/10/2014
• Est. Priority Date: 08/02/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Ming-Chang, Liu, Liangyin, Yu, Quan, Yuan
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): RICE, ELISA M
• Application Number:
  US12/883,708
• Publication Number:
  US 20120027263A1
• Time in Patent Office: 1,363 Days
• Field of Search: 382/181
• US Class Current: 382/107
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A method for detecting presence of a hand gesture in video frames includes receiving video frames having an original resolution, downscaling the received video frames into video frames having a lower resolution, and detecting a motion corresponding to the predefined hand gesture in the downscaled video frames based on temporal motion information in the downscaled video frames. The method also includes detecting a hand shape corresponding to the predefined hand gesture in a candidate search window within one of the downscaled video frames using a binary classifier. The candidate search window corresponds to a motion region containing the detected motion. The method further includes determining whether the received video frames contain the predefined hand gesture based on the hand shape detection.

Claims

1. A computer-implemented method for detecting presence of a hand gesture in video frames, the method comprising:

detecting a motion corresponding to a predefined hand gesture in downscaled video frames based on temporal motion information in the downscaled video frames;

detecting a hand shape corresponding to the predefined hand gesture in a candidate search window within one of the downscaled video frames using a binary classifier, wherein the candidate search window corresponds to a motion region containing the detected motion; and

determining whether the received video frames contain the predefined hand gesture based on the hand shape detection.

2. The method of claim 1, wherein detecting the motion comprises:

generating a motion history image (MHI) based on the temporal motion information, wherein the MHI represents recency of motions in the downscaled video frames;

perform template matching by a normalized cross-correlation of a plurality of motion templates in the MHI, wherein the plurality of motion templates correspond to the predefined hand gesture performed at different speed; and

detecting the motion based on correlation scores.

3. The method of claim 1, further comprising:

refining the motion region by filtering out a false motion based on an area size of and/or a pattern of the detected motion; and

determining the candidate search window based on the refined motion region.

4. The method of claim 1, further comprising:

refining the candidate search window by removing background edges based on gradient intensity values of the background edges in the candidate search window and in a corresponding region of a reference downscaled video frame; and

detecting the hand shape in the updated candidate search window using the binary classifier.

5. The method of claim 1, wherein detecting the hand shape comprises:

generating histogram of oriented gradient (HOG) features based on the candidate search window, wherein the HOG features correspond to gradient intensity values and edge orientation values of each region in the candidate search window; and

detecting the hand shape in the candidate search window using the binary classifier based on the generated HOG features.

6. The method of claim 5, wherein detecting the hand shape in the candidate search window using the binary classifier based on the generated HOG features comprises:

scanning the candidate search window using a two-dimensional filter corresponding to the binary classifier based on the generated HOG features;

generating a convolution response map based on the scan;

detecting a high response location in the convolution response map; and

detecting the hand shape in the candidate search window based on the high response location.

7. The method of claim 1, wherein the binary classifier can be a linear support vector machine (SVM) classifier or a nonlinear SVM classifier.

8. The method of claim 1, further comprising:

verifying the detected hand shape based on predefined hand skin colors; and

determining whether the received video frames contain the predefined hand gesture based on the verification.

9. The method of claim 1, further comprising:

determining a hand region containing the detected hand shape;

refining the hand region by filtering out false skin regions based on predefined hand skin colors;

verifying whether the refined hand region contains a hand shape corresponding to the predefined hand gesture using a nonlinear classifier; and

determining whether the received video frames contain the predefined hand gesture based on the verification.

10. The method of claim 1, further comprising:

verifying the detected hand shape using a nonlinear classifier based on histogram of oriented gradient (HOG) features, wherein the HOG features correspond to gradient intensity values and edge orientation values of a hand region corresponding to the detected hand shape; and

determining whether the received video frames contain the predefined hand gesture based on the verification.

11. An apparatus for detecting a hand gesture in video frames having an original resolution, the apparatus comprising:

a non-transitory memory for storing;

a motion detector to determine a motion region by detecting a motion corresponding to a predefined hand gesture in downscaled video frames based on temporal motion information in the downscaled video frames;

a hand detector to determine a hand region by detecting a hand shape corresponding to the predefined hand gesture in a candidate search window within one of the downscaled video frames using a linear classifier, wherein the candidate search window corresponds to the motion region; and

a hand verifier to verify whether the hand region contains a hand shape of the predefined hand gesture using a nonlinear classifier, and to determine whether the received video frames contain the predefined hand gesture based on the verification; and

a processor for executing the motion detector, the hand detector and the hand verifier.

12. The apparatus of claim 11, wherein the motion detector is configured to:

generate a motion history image (MHI) based on the temporal motion information, wherein the MHI represents recency of motions in the downscaled video frames;

perform template matching by a normalized cross-correlation of a motion template in the MHI, wherein the motion template corresponds to the predefined hand gesture; and

determine the motion region based on correlation scores.

13. The apparatus of claim 11, wherein the motion detector is configured to:

define a plurality of motion templates corresponding to the predefined hand gesture performed at different speed; and

detect the motion corresponding to the predefined hand gesture in the downscaled video frames based on the temporal motion information and the plurality of motion templates.

14. The apparatus of claim 11, further comprising:

a false motion filter to refine the motion region by filtering out false positives based on a definition of the predefined hand gesture and/or a predefined maximum or minimum size of hands, and to determine the candidate search window based on the refined motion region.

15. The apparatus of claim 11, wherein the hand detector is configured to:

generate histogram of oriented gradient (HOG) features based on the candidate search window, wherein the HOG features correspond to gradient intensity values and edge orientation values of each region in the candidate search window;

update the HOG features by resetting gradient intensity values of background edges to a predefined lower value based on comparing gradient intensity values of a region in the candidate search window with gradient intensity values of a corresponding region in an earlier downscaled video frame; and

detect the hand shape in the candidate search window using a linear support vector machine (SVM) classifier based on the updated HOG features.

16. The apparatus of claim 11, further comprising:

a skin color verifier to refine the hand region by filtering out a false skin region based on predefined hand skin colors.

17. The apparatus of claim 16, wherein the hand verifier is configured to:

verify whether the refined hand region contains the hand shape of the predefined hand gesture using a nonlinear support vector machine (SVM) classifier based on histogram of oriented gradient (HOG) features, wherein the HOG features correspond to gradient intensity values and edge orientation values of the refined hand region.

18. A system comprising:

a camera configured to capture images, the captured images having an original resolution; and

a gesture detector coupled to the camera and configured to

determine a motion region by detecting a motion corresponding to a predefined hand gesture in downscaled images based on a motion history image (MHI) and a plurality of motion templates, wherein the MHI represents recency of motions in the downscaled images, and the plurality of motion templates correspond to the predefined hand gesture performed at different speed,

refine the motion region by filtering out a false motion based on configuration information of the predefined hand gesture and the detected motion,

determine a hand region by detecting a hand shape corresponding to the predefined hand gesture in a candidate search window using a linear classifier, wherein the candidate search window is within one of the downscaled images and corresponds to the refined motion region,

refine the hand region by filtering out a false skin region based on hand skin colors, and

determine whether the captured images contain the predefined hand gesture by verifying the detected hand shape in the refined hand region using a nonlinear classifier.

19. The system of claim 18, wherein the gesture detector has been trained using a support vector machine (SVM) classifier based on histogram of oriented gradient (HOG) features of training images to recognize a hand of different races, genders, shapes, and sizes, and to distinguish a hand from background objects and other non-hand objects.

20. A non-transitory computer-readable medium storing instructions that, when executed, cause a computer to perform a method for hand gesture detection, the method comprising:

detecting a motion corresponding to the predefined hand gesture in downscaled video frames based on temporal motion information in the downscaled video frames;

detecting a hand shape corresponding to the predefined hand gesture in a candidate search window within one of the downscaled video frames using a binary classifier, wherein the candidate search window corresponds to a motion region containing the detected motion; and

determining whether the received video frames contain the predefined hand gesture based on the hand shape detection.