HAND POSE RECOGNITION

• US 20120119984A1
• Filed: 01/07/2011
• Published: 05/17/2012
• Est. Priority Date: 11/15/2010
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: SANKARASUBRAMANIAM, Yogesh, Subramanian, Anbumani, Munnangi, Krusheel
• Granted Patent: US 8,730,157 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G06V 40/50 Maintenance of biometric data or enrolment thereof

Abstract

Hand pose recognition comprises determining an initial hand pose estimate for a captured input hand pose and performing iterations based upon hand pose estimates and residues between such estimates and hand pose estimates. One or more control signals are generated based upon the hand pose recognition.

Claims

1. A hand pose recognition method comprising:

capturing, with one or more sensors, a image frame of an input hand pose;

determining an initial hand pose estimate for the input hand pose;

performing iterations, each iteration comprising;

identifying a residue for a current hand pose estimate based upon data values for the input hand pose and data values for the current hand pose estimate, wherein the initial hand pose estimate serves as the current hand pose estimate during a first one of the iterations; and

determining a current hand pose estimate based upon the input hand pose and the residue from the previous current hand pose estimate;

outputting a hand pose estimate for the image frame of the input hand pose based upon results from the iterations; and

generating one or more control signals, based upon the first image frame hand pose estimate, wherein the input hand pose is transformed to one or more control signals.

2. The method of claim 1, wherein determining an initial hand pose estimate for the input hand pose comprises:

extracting an input hand pose feature vector from the image frame of the input hand pose;

identifying candidate feature vectors that best match the input hand pose feature vector from a general library of hand pose feature vectors for possible hand poses;

computing a score for each possible hand pose based upon the candidate feature vectors, wherein the possible hand pose having the highest score is the initial hand pose estimate.

3. The method of claim 2, wherein identifying candidate feature vectors that best match the input hand pose feature vector from a general library of feature vectors for possible hand poses comprises using a dot product of the input hand pose feature vector and feature vectors of the general library of feature vectors for possible hand poses.

4. The method of claim 2, wherein the score for each possible hand pose is computed using a pseudoinverse of the input hand pose feature vector and the candidate feature vectors for each possible hand pose.

5. The method of claim 2, wherein determining a new current hand pose estimate based upon the input hand pose and the residue from the previous current hand pose estimate comprises:

identifying residue based feature vectors that best match the residue from the general library of feature vectors;

adding the identified residue based feature vectors to the candidate feature vectors of the current hand pose estimate to form iteration feature vectors;

identifying best match feature vectors, the best match feature vectors being those iteration feature vectors that best matches the input feature vector;

computing a score for each possible hand pose based upon the best match feature vectors, wherein the possible hand pose having the highest score is the new current hand pose estimate.

6. The method of claim 5 wherein the score for each possible hand pose is computed using a pseudoinverse of the best match feature vectors for each possible hand pose and the input hand pose feature vector.

7. The method of claim 5, wherein identifying the best match feature vector comprises:

computing a pseudoinverse of the iteration feature vectors;

multiplying the pseudoinverse of the iteration feature vectors and the input hand pose feature vector, wherein the best match feature vectors are those iteration feature vectors having the highest scores after their pseudoinverse result is multiplied by the input hand pose feature vector.

8. The method of claim 1, wherein the initial hand pose estimate has one or more hand pose feature vectors, the method further comprising extracting an input hand pose feature vector from the image frame of the input hand pose.

9. The method of claim 8 wherein determining a new current hand pose estimate based upon the input hand pose and the residue from the previous current hand pose estimate comprises:

identifying residue based feature vectors that best match the residue from a general library of feature vectors;

adding the identified residue based feature vectors to the one or more feature vectors of the current hand pose estimate to form iteration feature vectors;

identifying best match feature vectors, the best match feature vectors being those iteration feature vectors that best matches the input hand pose feature vector;

computing a score for each possible hand pose based upon the best match feature vectors, wherein the possible hand pose having the highest score is the new current hand pose estimate.

10. The method of claim 9 wherein the score for each possible hand pose is computed using a pseudoinverse of the best match feature vectors for each possible hand pose and the input hand pose feature vector.

11. The method of claim 1, wherein the image frame hand pose estimate is output in response to a predetermined convergence of the current hand pose estimates from different iterations.

12. The method of claim 11, wherein the predetermined convergence comprises the new current hand pose class estimate being same as the previous current hand pose estimate, the residue associated with the new hand pose estimate being less than the residue associated with the previous hand pose estimate and a sum of differences between new current hand pose class estimates and previous current hand pose class estimates for the iterations being less than a predetermined threshold.

13. The method of claim 1 further comprising:

generating a plurality of image frame hand pose estimates including the first image frame hand pose class estimate;

determining a final hand pose estimate based upon the plurality of image frame hand pose estimates, wherein the control signals are generated with the processor based upon the final hand pose estimate.

14. The method of claim 13 further comprising rejecting a portion of the plurality of image frame hand pose estimates, wherein determining the final hand pose the estimate is based upon the plurality of image frame hand pose estimates excluding the rejected portion of the plurality of image frame hand pose estimates.

15. The method of claim 1, wherein the initial hand pose estimate is derived from a hand pose library of possible hand poses, the hand pose library comprising a matrix of one or more hand pose feature vectors for each possible hand pose.

16. The method of claim 15 further comprising adding a new possible hand pose to the hand pose library by appending one or more new hand pose feature vectors for the new possible hand pose to the matrix.

17. The method of claim 16 further comprising:

capturing, with one or more sensors, a image frame of the new possible hand pose;

extracting a new possible hand pose feature vector from the image frame of the new possible hand pose; and

determining whether the new possible hand pose feature vectors are too close to existing hand pose feature vectors of the library.

18. The method of claim 1, wherein capturing the first image frame of the input hand pose comprises:

capturing a color/grayscale image frame having color/grayscale features of the input hand pose;

capturing a depth image frame having depth features of the input hand pose, wherein the depth image frame is captured concurrently with the captured color/grayscale frame; and

forming a single input hand pose feature vector based upon the captured depth image frame and the captured color/grayscale image frame.

19. A hand pose recognition method comprising:

generating a plurality of image frame hand pose estimates for an input hand pose, each image frame hand pose estimate being generated by;

capturing, with one or more sensors, an image frame of the input hand pose;

determining an initial hand pose estimate for the input hand pose by;

extracting an input hand pose feature vector from the image frame of the input hand pose;

identifying candidate feature vectors that best match the input hand pose feature vector from a general library of hand pose feature vectors for possible hand poses;

computing a score for each possible hand pose based upon the candidate feature vectors, wherein the possible hand pose having the highest score is the initial hand pose estimate;

performing iterations, each iteration comprising;

identifying a residue for a current hand pose estimate based upon data values for the input hand pose and data values for the current hand pose estimate, wherein the initial hand pose estimate serves as the current hand pose estimate during a first one of the iterations; and

determining a new current hand pose estimate based upon the input hand pose and the residue from the previous current hand pose estimate by;

determining a new current hand pose estimate based upon the input hand pose and the residue from the previous current hand pose estimate comprises;

identifying residue based feature vectors that best match the residue from the general library of feature vectors;

adding the identified residue based feature vectors to the candidate feature vectors of the current hand pose estimate to form iteration feature vectors;

identifying best match feature vectors, the best match feature vectors being those iteration feature vectors that best matches the input feature vector; and

computing a score for each possible hand pose based upon the best match feature vectors, wherein the possible hand pose having the highest score is the new current hand pose estimate;

storing the image frame hand pose estimate for the image frame of the input hand pose in response to a predetermined convergence of the current hand pose estimates from different iterations;

determining a final hand pose estimate based upon the plurality of image frame hand pose estimates; and

generating one more control signals, with a processor, based upon the final hand pose estimate, wherein the input hand pose this transformed to the one or more control signals.

20. A hand pose recognition system comprising:

one or more sensors configured to capture an image frame of an input hand pose;

a memory containing a library of possible hand poses;

one or more processors in communication with anyone more sensors and the memory, the one or more processors configured to;

determine an initial hand pose estimate for the input hand pose;

perform iterations, each iteration comprising;

identifying a residue for a current hand pose estimate based upon data values for the input hand pose and data values for the current hand pose estimate, wherein the initial hand pose estimate serves as the current hand pose estimate during a first one of the iterations; and

determining a new hand pose estimate based upon the input hand pose and the residue from the previous current hand pose estimate;

output a hand pose estimate for the image frame of the input hand pose based upon results from the iterations; and

generate one more control signals based upon the hand pose estimate.