Hand pointing estimation for human computer interaction

• US 9,128,530 B2
• Filed: 03/02/2015
• Issued: 09/08/2015
• Est. Priority Date: 08/12/2011
• Status: Active Grant

• Current Assignee: The Research Foundation for The State University of New York (State University of New York)
• Original Assignee: The Research Foundation for The State University of New York (State University of New York)
• Inventors: Yin, Lijun, Canavan, Shaun, Hu, Kaoning
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US14/635,094
• Publication Number:
  US 20150177846A1
• Time in Patent Office: 190 Days
• Field of Search: 382/100, 382/103, 382106-107, 382/162, 382/168, 382/173, 382/181, 382/199, 382/209, 382/219, 382/232, 382/254, 382/274, 382285-297, 382/305, 382/312, 345/173, 345/677, 345/163, 715/773, 715/784
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/446 using Haar-like filters, e.g. using integral image techniques
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/7557 based on appearance, e.g. active appearance models [AAM]
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Hand pointing has been an intuitive gesture for human interaction with computers. A hand pointing estimation system is provided, based on two regular cameras, which includes hand region detection, hand finger estimation, two views'"'"' feature detection, and 3D pointing direction estimation. The technique may employ a polar coordinate system to represent the hand region, and tests show a good result in terms of the robustness to hand orientation variation. To estimate the pointing direction, Active Appearance Models are employed to detect and track, e.g., 14 feature points along the hand contour from a top view and a side view. Combining two views of the hand features, the 3D pointing direction is estimated.

Claims

1. A method for estimating a finger pointing direction, comprising:

providing an active appearance model of an object of interest, comprising a statistical model of a shape and image intensity of the object of interest, the object of interest being a hand comprising the finger, which tracks a plurality of landmarks on the hand, derived from characteristics of a plurality of different hands, in a memory;

receiving a plurality of images acquired from different angles representing a hand having at least one finger having a finger pointing direction;

detecting the hand in each of the plurality of images;

locating a center of the hand in each of the plurality of image;

locating a position of a wrist proximate to the hand in each of the plurality of image;

processing the plurality of images with an automated processor to warp the detected hand in each image from a Cartesian Coordinate representation to a Polar Coordinate representation, with the located center of the hand at the pole, and the polar angle determined by the determined location of the wrist proximate to the hand with respect to the center of the hand;

applying the active appearance model to find a best fit for visible landmarks of the hand in each of the plurality of images, comprising at least two visible features of the at least one finger having the finger pointing direction;

combining the best fit for the visible landmarks of the hand in each of the plurality of images to the active appearance model, to infer a three dimensional position of each visible landmark in each of the plurality of images; and

determining the pointing direction of the at least one finger having the finger pointing direction.

2. The method according to claim 1, wherein the detecting the hand comprises generating Haar-like features from the at least two images, and implementing a cascade detector using AdaBoost, wherein portions of each image are scanned over different translations, scales and rotations to find a best fit for the hand.

3. The method according to claim 1,

wherein the plurality of images are color images, and

wherein color sensitive image processing is employed to distinguish the hand from a background.

4. The method according to claim 1,

wherein the plurality of images are color images, and

wherein color sensitive image processing is employed to define a wrist location as a location on the hand having a most similar surrounding color consistent with skin tones.

5. The method according to claim 1, wherein the at least two visible features of the finger are selected based on a reliability of a resulting three dimensional pointing vector.

6. The method according to claim 1, further comprising providing a user feedback display of a target of the pointing gesture.

7. The method according to claim 1, wherein the determining the pointing direction of the at least one finger having the finger pointing direction comprises determining the pointing direction of the at least one finger having the finger pointing direction in each of the plurality of images, and the determining a target of the finger pointing direction by at least statistically averaging the determined pointing direction for each of the plurality of images over time.

8. The method according to claim 1, wherein a best fit for visible landmarks of the hand in each of the plurality of images to the active appearance model is updated for respective sequences of the plurality of images, to track a movement of the hand over time.

9. The method according to claim 1, wherein the plurality of images comprise at least two video signals, each representing a time sequence of images.

10. A method for estimating a finger pointing direction using an active appearance model of an object of interest, comprising a statistical model of a shape and image intensity of the object of interest, the object of interest being a hand comprising the finger, which tracks a plurality of landmarks on a hand, corresponding to landmarks of hands in a training image set to which a principal component analysis is applied to formulate a statistical model of the hand, comprising:

capturing a plurality of images of a hand from different perspectives;

locating a center of the hand in each image with at least one automated processor;

determining a position of a wrist associated with the hand in each image with the at least one automated processor;

converting a representation of the hand in each image from a Cartesian Coordinate representation to a Polar Coordinate representation with the at least one automated processor, with the center of the hand at the pole of the Polar Coordinate representation, and the polar angle determined by a position of the wrist with respect to the located center of the hand;

finding a best fit for the hand in each image to the active appearance model of the object of interest with the at least one automated processor;

inferring a three dimensional position of a plurality of landmarks on the hand, comprising at least two landmarks on a finger making a pointing gesture, in each image, with the at least one automated processor;

combining the fit of the hand to the active appearance model of the object of interest in each image, with the inferred three dimensional position of a plurality of landmarks on the hand, comprising at least two landmarks on a finger making a pointing gesture, to determine a pointing vector of the finger making a pointing gesture in each of the images, with the at least one automated processor, wherein the at least two landmarks on the finger making the pointing gesture are selected based on an estimated reliability of the resulting vector; and

determining a three dimensional pointing vector of the finger based on the determined pointing vector of the finger making a pointing gesture in each of the images with the at least one automated processor.

11. The method according to claim 10, wherein the hand is detected in the plurality of images by at least generating Haar-like features from the a plurality of images, and implementing a cascade detector using AdaBoost, wherein portions of each image are scanned over different translations, scales and rotations to find a best fit for the hand.

12. The method according to claim 10, wherein the a plurality of images are color images, and wherein color sensitive image processing is employed to distinguish the hand from a background, and color sensitive image processing is employed to define a wrist location as a location on the hand having a most similar surrounding color consistent with skin tones.

13. The method according to claim 10, wherein the at least two landmarks on the finger making the pointing gesture are selected independently from at least three landmarks of the finger making the pointing gesture, in each of the plurality of images based on the estimated reliability of a resulting three dimensional pointing vector.

14. The method according to claim 10, further comprising providing, with the at least one automated processor, a user feedback display of a target along the three dimensional pointing vector of the finger.

15. The method according to claim 10, wherein the determined three dimensional pointing vector of the finger represents a statistical averaging over time.

16. The method according to claim 10, wherein the best fit for the hand in each image to the active appearance model of the object of interest is updated for respective sequences of the plurality of images by the at least one automated processor, to track a movement of the hand.

17. The method according to claim 10, wherein the plurality of images comprise a plurality of video signals, each representing a time sequence of images.

18. An apparatus for estimating a finger pointing direction, comprising:

a memory configured to store an active appearance model of an object of interest, comprising a statistical model of a shape and image intensity of the object of interest, the object of interest being a hand comprising the finger, which tracks a plurality of landmarks on the hand, derived from characteristics of a plurality of different hands;

at least one input port configured to receive a plurality of images acquired from different angles representing a hand having at least one finger having a finger pointing direction;

at least one automated processor, configured to;

detect the hand in each of the plurality of images;

locate a center of the hand in each of the plurality of image;

locate a position of a wrist proximate to the hand in each of the plurality of image;

process the plurality of images with an automated processor to warp the detected hand in each image from a Cartesian Coordinate representation to a Polar Coordinate representation, with the located center of the hand at the pole, and the polar angle determined by the determined location of the wrist proximate to the hand with respect to the center of the hand;

apply the active appearance model of the object of interest to find a best fit for visible landmarks of the hand in each of the plurality of images, comprising at least two visible features of the at least one finger having the finger pointing direction;

combine the best fit for the visible landmarks of the hand in each of the plurality of images to the active appearance model of the object of interest, to infer a three dimensional position of each visible landmark in each of the plurality of images; and

determine the pointing direction of the at least one finger having the finger pointing direction.

19. The apparatus according to claim 18, further comprising a port configured to present information defining a target of the finger pointing direction.

20. The apparatus according to claim 18, wherein the plurality of images are color images, and wherein the at least one processor is further configured to:

perform color sensitive image processing to distinguish the hand from a background; and

detect the hand by generating Haar-like features from the a plurality of images, and implementing a cascade detector using AdaBoost, wherein portions of each image are scanned over different translations, scales and rotations to find a best fit for the hand.