Hand pointing estimation for human computer interaction

• US 9,372,546 B2
• Filed: 09/03/2015
• Issued: 06/21/2016
• Est. Priority Date: 08/12/2011
• Status: Active Grant

• Current Assignee: The Research Foundation for The State University of New York (State University of New York)
• Original Assignee: The Research Foundation for The State University of New York (State University of New York)
• Inventors: Yin, Lijun, Canavan, Shaun, Hu, Kaoning
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US14/844,815
• Publication Number:
  US 20150378444A1
• Time in Patent Office: 292 Days
• Field of Search: 382/100, 382/103, 382106-107, 382/162, 382/168, 382/173, 382/181, 382/199, 382/209, 382/219, 382/232, 382/254, 382/274, 382285-297, 382/305, 382/312, 715/773, 715/784, 345/173, 345/677, 345/163
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/446 using Haar-like filters, e.g. using integral image techniques
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/7557 based on appearance, e.g. active appearance models [AAM]
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Hand pointing has been an intuitive gesture for human interaction with computers. A hand pointing estimation system is provided, based on two regular cameras, which includes hand region detection, hand finger estimation, two views'"'"' feature detection, and 3D pointing direction estimation. The technique may employ a polar coordinate system to represent the hand region, and tests show a good result in terms of the robustness to hand orientation variation. To estimate the pointing direction, Active Appearance Models are employed to detect and track, e.g., 14 feature points along the hand contour from a top view and a side view. Combining two views of the hand features, the 3D pointing direction is estimated.

Claims

1. A method for determining a finger pointing target, comprising:

receiving a plurality of images comprising a sequence of images over time, including images acquired from different perspectives representing a hand having a finger with a finger pointing direction;

tracking a plurality of landmarks on the hand comprising at least two visible landmarks of the finger having the finger pointing direction, a center of the hand, and a position of a wrist proximate to the hand, in a series of images, the respective landmarks being derived from common characteristics of hands from a plurality of different people;

determining a best fit for visible landmarks of the hand in each of the series of images to an active appearance model;

inferring a three dimensional position of each visible landmark in the series of images;

determining the finger pointing directional vector of the finger based on the inferred three dimensional position of each visible landmark in the series of images; and

determining a finger pointing gesture based on at least a change in the determined directional vector over time.

2. The method according to claim 1, further comprising detecting the hand by generating Haar-like features from at least two images, and implementing a cascade detector using AdaBoost, wherein portions of each respective image of the sequence of images over time are scanned over different translations, scales and rotations to find a best fit for the hand.

3. The method according to claim 1, wherein color sensitive image processing is employed to distinguish the hand from a background in the series of images and to define a wrist location as a location adjacent to the hand having a most similar surrounding color consistent with skin tones.

4. The method according to claim 1, further comprising determining a finger pointing target along the finger pointing directional vector.

5. The method according to claim 1, wherein the finger pointing directional vector is a three dimensional vector, and the at least two visible landmarks of the finger are selected based on a reliability of a resulting three dimensional pointing vector.

6. The method according to claim 1, further comprising providing a user feedback display of a determined finger pointing directional vector.

7. The method according to claim 1, wherein the finger pointing gesture is determined by at least a statistical correspondence to a model of the change of the determined finger pointing direction vector over time.

8. The method according to claim 1, wherein a best fit for visible landmarks of the hand in each of the plurality of images to the active appearance model is updated for each respective image of the sequence of images over time, to track a movement of the hand over time.

9. The method according to claim 1, wherein the sequence of images over time comprises video signals in which the hand appears acquired from different positions.

10. A method for determining a hand gesture, comprising

capturing a plurality of images of a hand from different perspectives;

determining a center of the hand and a position of a wrist associated with the hand in each image;

finding a best fit for the hand in each image to an active appearance model;

inferring a three dimensional position of a plurality of landmarks on the hand,

corresponding to landmarks of human hands in a training image set to which a principal component analysis is applied to formulate a statistical model of the human hand;

combining the fit of the hand to the active appearance model in each image, with the inferred three dimensional position of a plurality of landmarks on the hand, to automatically determine at least one directional vector defined by the plurality of landmarks;

and determining a gesture based on a change in the at least one directional vector over time.

11. The method according to claim 10, wherein the at least two landmarks are selected based on an estimated reliability of the resulting directional vector.

12. The method according to claim 10, wherein the hand is detected in the plurality of images by at least generating Haar-like features from the a plurality of images, and implementing a cascade detector using AdaBoost, wherein portions of each image are scanned over different translations, scales and rotations to find a best fit for the hand.

13. The method according to claim 10, wherein the a plurality of images are color images, and wherein color sensitive image processing is employed to distinguish the hand from a background, and color sensitive image processing is employed to define a wrist location as a location on the hand having a most similar surrounding color consistent with skin tones.

14. The method according to claim 10, further comprising providing feedback to a user of the at least one directional vector.

15. The method according to claim 10, wherein the determined at least one directional vector represents a statistical averaging over time.

16. The method according to claim 10, wherein the best fit for the hand in each image to the active appearance model is updated for respective sequences of the plurality of images, to track a movement of the hand.

17. The method according to claim 10, wherein the plurality of images are acquired from at least two different perspectives by at least two different video cameras.

18. An apparatus for estimating a finger pointing direction, comprising:

a memory configured to store an appearance hand model which tracks a plurality of landmarks on the hand, derived from characteristics of a plurality of different hands;

a memory configured to store a plurality of images acquired from different angles representing a hand in a sequence of images over time;

at least one automated processor, configured to;

detect the hand in each of the plurality of images;

locate a center of the hand and a position of a wrist proximate to the hand in each of the plurality of images;

apply the active appearance model to find a best fit for visible landmarks of the hand in each of the plurality of images;

combine the best fit for the visible landmarks of the hand in each of the plurality of images to the active appearance model, to infer a three dimensional position of each visible landmark in each of the plurality of images;

determine a sequence of directional vector over time associated with the inferred three dimensional position of each visible landmark; and

determine a gesture based on at least a change in the determined sequence of directional vector over time.

19. The apparatus according to claim 18, wherein the plurality of landmarks comprise at least two landmarks of a pointing finger, and the at least one automated processor is further configured to determine at least one target of the pointing finger based on at least the determined directional vector.

20. The apparatus according to claim 18, wherein the at least one automated processor is further configured to determine the gesture based on a statistical correspondence to a model of the determined change of the sequence of a target of directional vector over time.