Capturing and recognizing hand postures using inner distance shape contexts

• US 8,428,311 B2
• Filed: 02/19/2010
• Issued: 04/23/2013
• Est. Priority Date: 02/25/2009
• Status: Expired due to Fees

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Dariush, Behzad, Gopalan, Raghuraman
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): Rosario, Dennis
• Application Number:
  US12/709,287
• Publication Number:
  US 20100215257A1
• Time in Patent Office: 1,159 Days
• Field of Search: 382/224, 382/115, 382/225
• US Class Current: 382/115
• CPC Class Codes: G06V 10/46 Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features colour feature extraction G06V10/56
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A system, method, and computer program product for recognizing hand postures are described. According to one aspect, a set of training images is provided with labels identifying hand states captured in the training images. Inner Distance Shape Context (IDSC) descriptors are determined for the hand regions in the training images, and fed into a Support Vector Machine (SVM) classifier to train it to classify hand shapes into posture classes. An IDSC descriptor is determined for a hand region in a testing image, and classified by the SVM classifier into one of the posture classes the SVM classifier was trained for. The hand posture captured in the testing image is recognized based on the classification.

Claims

1. A computer based method for recognizing a hand posture captured in an image, the method comprising:

identifying a hand region in the image;

estimating a scatter direction of the hand region in the image;

determining an Inner Distance Shape Context (IDSC) descriptor for the hand region identified in the image, the IDSC descriptor not based on the scatter direction;

matching the scatter direction to one of a plurality orientation bins, the matching orientation bin corresponding to that scatter direction;

identifying a classifier associated with the matching orientation bin, the classifier comprising a plurality of reference IDSC descriptors corresponding to a plurality of hand posture classes, the classifier generated based on training images having scatter directions that also match the orientation bin;

classifying the hand region into one of the plurality of hand posture classes by feeding the IDSC descriptor into the classifier.

2. The method of claim 1, further comprising:

determining reference IDSC descriptors for reference hand regions in training images;

labeling each of the reference hand regions to be in one of the plurality of hand posture classes; and

training the classifier to classify hand regions into the plurality of hand posture classes by feeding the classifier with the reference IDSC descriptors and the corresponding hand posture classes.

3. The method of claim 2, further comprising:

segmenting hand regions from the training images and the image based on a skin color.

4. The method of claim 3, wherein segmenting the hand regions comprises:

applying a morphological operation to the segmented hand region.

5. The method of claim 2, further comprises:

estimating scatter directions of the reference hand regions in the training images;

grouping the reference hand regions in the training images into orientation bins based on associated scatter directions of the training images; and

wherein training the classifier comprises training a classifier for each of the orientation bins by feeding the classifier with the IDSC descriptors of the reference hand regions in the orientation bin.

6. The method of claim 1, wherein estimating the scatter directions comprises:

applying principal component analysis (PCA) to project the hand region along a direction of maximum scatter and determining the direction of maximum scatter as the scatter direction of the hand region.

7. The method of claim 1, wherein the image comprises a depth image taken by a single time-of-flight camera.

8. The method of claim 1, wherein the classifier is configured to determine a linear separating hyperplane that separates IDSC descriptors according to the corresponding hand posture classes.

9. The method of claim 8, wherein the classifier comprises a Support Vector Machine (SVM) classifier.

10. The method of claim 1, wherein identifying the hand region in the image comprises:

identifying the hand region in the image based on body features detected in the image.

11. A computer program product for recognizing a hand posture captured in an image, the computer program product comprising a non-transitory computer-readable storage medium containing executable computer program code for performing a method comprising:

identifying a hand region in the image;

estimating a scatter direction of the hand region in the image;

determining an Inner Distance Shape Context (IDSC) descriptor for the hand region identified in the image, the IDSC descriptor not based on the scatter direction;

matching the scatter direction to one of a plurality orientation bins, the matching orientation bin corresponding to that scatter direction;

identifying a classifier associated with the matching orientation bin, the classifier comprising a plurality of reference IDSC descriptors corresponding to a plurality of hand posture classes, the classifier generated based on training images having scatter directions that also match the orientation bin;

classifying the hand region into one of the plurality of hand posture classes by feeding the IDSC descriptor into the classifier.

12. The computer program product of claim 11, wherein the method further comprises:

determining reference IDSC descriptors for reference hand regions in training images;

labeling each of the reference hand regions to be in one of the plurality of hand posture classes; and

training the classifier to classify hand regions into the plurality of hand posture classes by feeding the classifier with the reference IDSC descriptors and the corresponding hand posture classes.

13. The computer program product of claim 12, wherein the method further comprises:

segmenting hand regions from the training images and the image based on a skin color.

14. The computer program product of claim 13, wherein segmenting the hand regions comprises:

applying a morphological operation to the segmented hand region.

15. The computer program product of claim 12, wherein the method further comprises:

estimating scatter directions of the reference hand regions in the training images;

grouping the reference hand regions in the training images into orientation bins based on associated scatter directions of the training images; and

wherein training the classifier comprises training a classifier for each of the orientation bins by feeding the classifier with the IDSC descriptors of the reference hand regions in the orientation bin.

16. A system for recognizing a hand posture captured in an image, the system comprising:

a computer processor for executing executable computer program code; and

a computer-readable storage medium containing the executable computer program code for performing a method comprising;

identifying a hand region in the image;

estimating a scatter direction of the hand region in the image;

determining an Inner Distance Shape Context (IDSC) descriptor for the hand region identified in the image, the IDSC descriptor not based on the scatter direction;

matching the scatter direction to one of a plurality orientation bins, the matching orientation bin corresponding to that scatter direction;

identifying a classifier associated with the matching orientation bin, the classifier comprising a plurality of reference IDSC descriptors corresponding to a plurality of hand posture classes, the classifier generated based on training images having scatter directions that also match the orientation bin;

classifying the hand region into one of the plurality of hand posture classes by feeding the IDSC descriptor into the classifier.

17. The system of claim 16, wherein the method further comprises:

determining reference IDSC descriptors for reference hand regions in training images;

labeling each of the reference hand regions to be in one of the plurality of hand posture classes; and

training the classifier to classify hand regions into the plurality of hand posture classes by feeding the classifier with the reference IDSC descriptors and the corresponding hand posture classes.

18. The system of claim 17, wherein the method further comprises:

segmenting hand regions from the training images and the image based on a skin color.

19. The system of claim 18, wherein segmenting the hand regions comprises:

applying a morphological operation to the segmented hand region.