Real time hand tracking, pose classification, and interface control

• US 8,774,464 B2
• Filed: 09/27/2013
• Issued: 07/08/2014
• Est. Priority Date: 11/06/2009
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Adhikari, Suranjit
• Primary Examiner(s): Werner, Brian P
• Application Number:
  US14/039,217
• Publication Number:
  US 20140022164A1
• Time in Patent Office: 284 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A hand gesture from a camera input is detected using an image processing module of a consumer electronics device. The detected hand gesture is identified from a vocabulary of hand gestures. The electronics device is controlled in response to the identified hand gesture. This abstract is not to be considered limiting, since other embodiments may deviate from the features described in this abstract.

Claims

1. A method of hand position detection, comprising:

extracting, via an image processing module of an electronics device, a feature set associated with hand gesture detection and hand pose inference from at least one input image, the feature set including extracted image pyramids;

inferring a hand pose type using a trained multiclass support vector machine (SVM);

extracting pixel intensity or displacement features and scale invariant feature transforms (SIFT) features using the generated image pyramids; and

approximating the hand pose using inverse kinematics (IK) optimization.

2. The method according to claim 1, where the image pyramids are extracted from brightness gradients of the at least one input image.

3. The method according to claim 1, further comprising applying a cascade filter in association with extracting a pixel intensity or displacement features and the SIFT features from the generated image pyramids.

4. The method according to claim 1, where the at least one input image comprises a plurality of input images, and extracting, via the image processing module of the electronics device, the feature set associated with hand gesture detection and hand pose inference from the at least one input image comprises:

tracking a region of interest (ROI) between subsequent video frames of the plurality of input images as a flock of features;

triggering scale invariant feature transforms (SIFT) feature extraction; and

calculating an optical flow path of the flock of features.

5. The method according to claim 4, where tracking the ROI between subsequent video frames of the plurality of input images as the flock of features comprises:

tracking a two dimensional (2D) combination of pixel intensity/displacement features and a learned foreground color distribution.

6. The method according to claim 4, where calculating the optical flow path of the flock of features comprises applying at least one constraint on each of the flock of features such that the flock of features maintain a minimum distance from each other.

7. The method according to claim 1, where inferring a hand pose type using the trained multiclass SVM comprises:

detecting at least one feature within a training image and the at least one input image; and

performing a one-to-one mapping of instances of the at least one feature within the at least one input image with at least one label drawn from a finite set of elements.

8. The method according to claim 1, where approximating the hand pose using IK optimization comprises:

partitioning the at least one input image into a plurality of processing regions;

determining a centroid of features within each of the plurality of processing regions; and

mapping a location of each feature centroid onto three dimensional (3D) pose data associated with a motion capture data set.

9. The method according to claim 8, where determining the centroid of features within each of the plurality of processing regions comprises:

comparing variances from each feature centroid to a closest match within the 3D pose data; and

determining which of a plurality of joint constraints affect the IK optimization.

10. The method according to claim 9, further comprising mapping each feature centroid to a closest joint stored within the 3D pose data.

11. The method according to claim 10, further comprising:

minimizing a distance of each mapped closest joint within a training image based upon the 3D pose data; and

determining a final hand position based upon the minimized distance of each mapped closest joint within the training image.

12. The method according to claim 1, further comprising:

defining a configurable resolution threshold for image processing; and

adjusting the configurable resolution threshold.

13. An apparatus for hand position detection, comprising:

a camera; and

a processor programmed to;

extract a feature set associated with hand gesture detection and hand pose inference from at least one input image received via the camera, the feature set including image pyramids;

trigger scale invariant feature transforms (SIFT) feature extraction;

calculate an optical flow path of a flock of features of the at least one input image;

infer a hand pose type using a trained multiclass support vector machine (SVM); and

approximate the hand pose using inverse kinematics (IK) optimization.

14. The apparatus according to claim 13, where, in being programmed to extract the feature set associated with hand gesture detection and hand pose inference from the at least one input image received via the camera, the processor is programmed to generate the image pyramids from measured brightness gradients across the at least one input image.

15. The apparatus according to claim 13, where the at least one input image comprises a plurality of input images, and, where in being programmed to extract the feature set associated with hand gesture detection and hand pose inference from the at least one input image received via the camera, the processor is programmed to:

track a region of interest (ROI) between subsequent video frames of the plurality of input images as the flock of features.

16. The apparatus according to claim 15, where, in being programmed to track the ROI between subsequent video frames of the plurality of input images as the flock of features, the processor is programmed to:

track a two dimensional (2D) combination of pixel intensity/displacement features and a learned foreground color distribution.

17. The apparatus according to claim 15, where, in being programmed to calculate the optical flow path of the flock of features, the processor is programmed to apply at least one constraint on each of the flock of features such that the flock of features maintain a minimum distance from each other.

18. The apparatus according to claim 13, where, in being programmed to infer a hand pose type using the trained multiclass SVM, the processor is programmed to:

detect at least one feature within a training image and the at least one input image; and

perform a one-to-one mapping of instances of the at least one feature within the at least one input image with at least one label drawn from a finite set of elements.

19. The apparatus according to claim 18, where the at least one label comprises at least one label generated during a training phase based upon a motion capture three dimensional (3D) data set.

20. The apparatus according to claim 13, where, in being programmed to approximate the hand pose using IK optimization, the processor is programmed to:

partition the at least one input image into a plurality of processing regions;

determine a centroid of features within each of the plurality of processing regions; and

map a location of each feature centroid onto three dimensional (3D) pose data associated with a motion capture data set.

21. The apparatus according to claim 20, where, in being programmed to determine the centroid of features within each of the plurality of processing regions, the processor is programmed to:

compare variances from each feature centroid to a closest match within the 3D pose data; and

determine which of a plurality of joint constraints affect the IK optimization.

22. The apparatus according to claim 21, where the processor is further programmed to map each feature centroid to a closest joint stored within the 3D pose data.

23. The apparatus according to claim 22, where the processor is further programmed to:

minimize a distance of each mapped closest joint within a training image based upon the 3D pose data; and

determine a final hand position based upon the minimized distance of each mapped closest joint within the training image.

24. The apparatus according to claim 13, where the processor is further programmed to:

define a configurable resolution threshold for image processing; and

adjust the configurable resolution threshold.

25. An apparatus for hand position detection, comprising:

a camera; and

a processor programmed to;

extract a feature set associated with hand gesture detection and hand pose inference from a plurality of input images received via the camera by tracking a region of interest (ROI) between subsequent video frames of the plurality of input images as a flock of features, triggering scale invariant feature transforms (SIFT) feature extraction, and calculating an optical flow path of the flock of features;

infer a hand pose type using a trained multiclass support vector machine (SVM); and

approximate the hand pose using inverse kinematics (IK) optimization.