Real Time Hand Tracking, Pose Classification and Interface Control

• US 20110110560A1
• Filed: 10/04/2010
• Published: 05/12/2011
• Est. Priority Date: 11/06/2009
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Adhikari, Suranjit
• Granted Patent: US 8,600,166 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A hand gesture from a camera input is detected using an image processing module of a consumer electronics device. The detected hand gesture is identified from a vocabulary of hand gestures. The electronics device is controlled in response to the identified hand gesture. This abstract is not to be considered limiting, since other embodiments may deviate from the features described in this abstract.

Claims

1. A method of controlling an electronics device via hand gestures, comprising:

detecting, via an image processing module of the electronics device, a bare-hand position via a camera input based upon a detected sequence of bare-hand positions;

determining whether the bare-hand position has been detected for a threshold duration of time;

identifying the detected bare-hand position from a vocabulary of hand gestures, where the identified bare-hand position comprises a hand gesture associated with powering on the electronics device; and

controlling, in response to determining that the bare-hand position has been detected for the threshold duration of time, the electronics device in response to the identified bare-hand position by powering on the electronics device.

2. A computer readable storage medium storing instructions which, when executed on one or more programmed processors, carry out the method according to claim 1.

3. A method of controlling an electronics device via hand gestures, comprising:

detecting, via an image processing module of the electronics device, a hand gesture via a camera input;

identifying the detected hand gesture from a vocabulary of hand gestures; and

controlling the electronics device in response to the identified hand gesture.

4. The method according to claim 3, where detecting, via the image processing module of the electronics device, the hand gesture via the camera input comprises detecting a bare-hand position.

5. The method according to claim 3, where detecting, via the image processing module of the electronics device, the hand gesture via the camera input comprises detecting a sequence of bare-hand positions.

6. The method according to claim 3, where the identified hand gesture comprises a hand gesture associated with powering on the electronics device and where controlling the electronics device in response to the identified hand gesture comprises powering on the electronics device.

7. The method according to claim 3, where the identified hand gesture comprises a hand gesture associated with powering off the electronics device and where controlling the electronics device in response to the identified hand gesture comprises powering off the electronics device.

8. The method according to claim 3, further comprising:

determining whether the hand gesture associated with the control of the electronics device has been detected for a threshold duration of time; and

where detecting, via the image processing module of the electronics device, the hand gesture via the camera input comprises detecting the hand gesture associated with the control of the electronics device for the threshold duration of time.

9. The method according to claim 3, further comprising:

determining whether the hand gesture associated with the control of the electronics device has been detected for a threshold duration of time; and

where identifying the detected hand gesture from the vocabulary of hand gestures comprises identifying the detected hand gesture from the vocabulary of hand gestures in response to determining that the hand gesture associated with the control of the electronics device has been detected for the threshold duration of time.

10. The method according to claim 3, further comprising:

detecting user input indicating assignment of one of the vocabulary of hand gestures to a control function of the electronics device; and

assigning the one of the vocabulary of hand gestures to the control function of the electronics device.

11. The method according to claim 10, where detecting the user input indicating the assignment of the one of the vocabulary of hand gestures to the control function of the electronics device comprises detecting a hand gesture associated with the assignment of the one of the vocabulary of hand gestures to the control function of the electronics device.

12. A computer readable storage medium storing instructions which, when executed on one or more programmed processors, carry out the method according to claim 3.

13. A method of hand position detection, comprising:

extracting, via an image processing module of an electronics device, a feature set associated with hand gesture detection and hand pose inference from a plurality of input images by;

tracking a region of interest (ROI) between subsequent video frames of the plurality of input images as a flock of features;

triggering scale invariant feature transforms (SIFT) feature extraction;

calculating an optical flow path of the flock of features;

measuring brightness gradients in multiple directions across the plurality of input images;

generating image pyramids from the measured brightness gradients;

extracting pixel intensity/displacement features and the SIFT features using the generated image pyramids; and

applying a cascade filter in association with extracting the pixel intensity/displacement features and the SIFT features from the generated image pyramids;

inferring a hand pose type using a trained multiclass support vector machine (SVM) by;

detecting at least one feature within a training image and the plurality of input images; and

performing a one-to-one mapping of instances of the at least one feature within the plurality of input images with at least one label drawn from a finite set of elements, where the at least one label comprises at least one label generated during a training phase based upon a motion capture three dimensional (3D) data set; and

approximating the hand pose using inverse kinematics (IK) optimization by;

partitioning the plurality of input images into a plurality of processing regions;

determining a centroid of features within each of the plurality of processing regions;

mapping a location of each feature centroid onto three dimensional (3D) pose data associated with a motion capture data set;

comparing variances from each feature centroid to a closest match within the 3D pose data;

determining which of a plurality of joint constraints affect the IK optimization;

mapping each feature centroid to a closest joint stored within the 3D pose data;

minimizing a distance of each mapped closest joint within the training image based upon the 3D pose data; and

determining a final hand position based upon the minimized distance of each mapped closest joint within the training image.

14. A computer readable storage medium storing instructions which, when executed on one or more programmed processors, carry out the method according to claim 13.

15. A method of hand position detection, comprising:

extracting, via an image processing module of an electronics device, a feature set associated with hand gesture detection and hand pose inference from at least one input image;

inferring a hand pose type using a trained multiclass support vector machine (SVM); and

approximating the hand pose using inverse kinematics (IK) optimization.

16. The method according to claim 15, where extracting, via the image processing module of the electronics device, the feature set associated with hand gesture detection and hand pose inference from the at least one input image comprises:

measuring brightness gradients in multiple directions across the at least one input image; and

generating image pyramids from the measured brightness gradients.

17. The method according to claim 16, further comprising:

extracting pixel intensity/displacement features and scale invariant feature transforms (SIFT) features using the generated image pyramids.

18. The method according to claim 17, further comprising applying a cascade filter in association with extracting the pixel intensity/displacement features and the SIFT features from the generated image pyramids.

19. The method according to claim 15, where the at least one input image comprises a plurality of input images, and extracting, via the image processing module of the electronics device, the feature set associated with hand gesture detection and hand pose inference from the at least one input image comprises:

tracking a region of interest (ROI) between subsequent video frames of the plurality of input images as a flock of features;

triggering scale invariant feature transforms (SIFT) feature extraction; and

calculating an optical flow path of the flock of features.

20. The method according to claim 19, where tracking the ROI between subsequent video frames of the plurality of input images as the flock of features comprises:

tracking a two dimensional (2D) combination of pixel intensity/displacement features and a learned foreground color distribution.

21. The method according to claim 19, where calculating the optical flow path of the flock of features comprises applying at least one constraint on each of the flock of features such that the flock of features maintain a minimum distance from each other.

22. The method according to claim 15, where inferring a hand pose type using the trained multiclass SVM comprises:

detecting at least one feature within a training image and the at least one input image; and

performing a one-to-one mapping of instances of the at least one feature within the at least one input image with at least one label drawn from a finite set of elements.

23. The method according to claim 22, where the at least one label comprises at least one label generated during a training phase based upon a motion capture three dimensional (3D) data set.

24. The method according to claim 15, where approximating the hand pose using IK optimization comprises:

partitioning the at least one input image into a plurality of processing regions;

determining a centroid of features within each of the plurality of processing regions; and

mapping a location of each feature centroid onto three dimensional (3D) pose data associated with a motion capture data set.

25. The method according to claim 24, where determining the centroid of features within each of the plurality of processing regions comprises:

comparing variances from each feature centroid to a closest match within the 3D pose data; and

determining which of a plurality of joint constraints affect the IK optimization.

26. The method according to claim 25, further comprising mapping each feature centroid to a closest joint stored within the 3D pose data.

27. The method according to claim 26, further comprising:

minimizing a distance of each mapped closest joint within a training image based upon the 3D pose data; and

determining a final hand position based upon the minimized distance of each mapped closest joint within the training image.

28. The method according to claim 15, further comprising:

defining a configurable resolution threshold for image processing; and

adjusting the configurable resolution threshold.

29. The method according to claim 15, further comprising storing the extracted feature set associated with the hand gesture detection and the hand pose inference.

30. A computer readable storage medium storing instructions which, when executed on one or more programmed processors, carry out the method according to claim 15.

31. An apparatus for controlling an electronics device via hand gestures, comprising:

a camera; and

a processor programmed to;

detect a bare-hand position via the camera based upon a detected sequence of bare-hand positions;

determine whether the bare-hand position has been detected for a threshold duration of time;

identify the detected bare-hand position from a vocabulary of hand gestures, where the identified bare-hand position comprises a hand gesture associated with powering on the electronics device; and

control, in response to determining that the bare-hand position has been detected for the threshold duration of time, the electronics device in response to the identified bare-hand position by powering on the electronics device.

32. An apparatus for controlling an electronics device via hand gestures, comprising:

a camera; and

a processor programmed to;

detect a hand gesture via the camera;

identify the detected hand gesture from a vocabulary of hand gestures; and

control the electronics device in response to the identified hand gesture.

33. The apparatus according to claim 32, where, in being programmed to detect the hand gesture via the camera, the processor is programmed to detect a bare-hand position.

34. The apparatus according to claim 32, where, in being programmed to detect the hand gesture via the camera, the processor is programmed to detect a sequence of bare-hand positions.

35. The apparatus according to claim 32, where the identified hand gesture comprises a hand gesture associated with powering on the electronics device and where, in being programmed to control the electronics device in response to the identified hand gesture, the processor is programmed to power on the electronics device.

36. The apparatus according to claim 32, where the identified hand gesture comprises a hand gesture associated with powering off the electronics device and where, in being programmed to control the electronics device in response to the identified hand gesture, the processor is programmed to power off the electronics device.

37. The apparatus according to claim 32, where the processor is further programmed to:

determine whether the hand gesture associated with the control of the electronics device has been detected for a threshold duration of time; and

where, in being programmed to detect the hand gesture via the camera, the processor is programmed to detect the hand gesture associated with the control of the electronics device for the threshold duration of time.

38. The apparatus according to claim 32, where the processor is further programmed to:

determine whether the hand gesture associated with the control of the electronics device has been detected for a threshold duration of time; and

where, in being programmed to identify the detected hand gesture from the vocabulary of hand gestures, the processor is programmed to identify the detected hand gesture from the vocabulary of hand gestures in response to determining that the hand gesture associated with the control of the electronics device has been detected for the threshold duration of time.

39. The apparatus according to claim 32, where the processor is further programmed to:

detect user input indicating assignment of one of the vocabulary of hand gestures to a control function of the electronics device; and

assign the one of the vocabulary of hand gestures to the control function of the electronics device.

40. The apparatus according to claim 39, where, in being programmed to detect the user input indicating the assignment of the one of the vocabulary of hand gestures to the control function of the electronics device, the processor is programmed to detect a hand gesture associated with the assignment of the one of the vocabulary of hand gestures to the control function of the electronics device.

41. An apparatus for hand position detection, comprising:

a camera; and

a processor programmed to;

extract a feature set associated with hand gesture detection and hand pose inference from a plurality of input images received via the camera, where the processor is further programmed to;

track a region of interest (ROI) between subsequent video frames of the plurality of input images as a flock of features;

trigger scale invariant feature transforms (SIFT) feature extraction;

calculate an optical flow path of the flock of features;

measure brightness gradients in multiple directions across the plurality of input images;

generate image pyramids from the measured brightness gradients;

extract pixel intensity/displacement features and the SIFT features using the generated image pyramids; and

apply a cascade filter in association with extracting the pixel intensity/displacement features and the SIFT features from the generated image pyramids;

infer a hand pose type using a trained multiclass support vector machine (SVM), where the processor is further programmed to;

detect at least one feature within a training image and the plurality of input images; and

perform a one-to-one mapping of instances of the at least one feature within the plurality of input images with at least one label drawn from a finite set of elements, where the at least one label comprises at least one label generated during a training phase based upon a motion capture three dimensional (3D) data set; and

approximate the hand pose using inverse kinematics (IK) optimization, where the processor is further programmed to;

partition the plurality of input images into a plurality of processing regions;

determine a centroid of features within each of the plurality of processing regions;

map a location of each feature centroid onto three dimensional (3D) pose data associated with a motion capture data set;

compare variances from each feature centroid to a closest match within the 3D pose data;

determine which of a plurality of joint constraints affect the IK optimization;

map each feature centroid to a closest joint stored within the 3D pose data;

minimize a distance of each mapped closest joint within the training image based upon the 3D pose data; and

determine a final hand position based upon the minimized distance of each mapped closest joint within the training image.

42. An apparatus for hand position detection, comprising:

a camera; and

a processor programmed to;

extract a feature set associated with hand gesture detection and hand pose inference from at least one input image received via the camera;

infer a hand pose type using a trained multiclass support vector machine (SVM); and

approximate the hand pose using inverse kinematics (IK) optimization.

43. The apparatus according to claim 42, where, in being programmed to extract the feature set associated with hand gesture detection and hand pose inference from the at least one input image received via the camera, the processor is programmed to:

measure brightness gradients in multiple directions across the at least one input image; and

generate image pyramids from the measured brightness gradients.

44. The apparatus according to claim 43, where the processor is further programmed to:

extract pixel intensity/displacement features and scale invariant feature transforms (SIFT) features using the generated image pyramids.

45. The apparatus according to claim 44, where the processor is further programmed to apply a cascade filter in association with extracting the pixel intensity/displacement features and the SIFT features from the generated image pyramids.

46. The apparatus according to claim 42, where the at least one input image comprises a plurality of input images, and, where in being programmed to extract the feature set associated with hand gesture detection and hand pose inference from the at least one input image received via the camera, the processor is programmed to:

track a region of interest (ROI) between subsequent video frames of the plurality of input images as a flock of features;

trigger scale invariant feature transforms (SIFT) feature extraction; and

calculate an optical flow path of the flock of features.

47. The apparatus according to claim 46, where, in being programmed to track the ROI between subsequent video frames of the plurality of input images as the flock of features, the processor is programmed to:

track a two dimensional (2D) combination of pixel intensity/displacement features and a learned foreground color distribution.

48. The apparatus according to claim 46, where, in being programmed to calculate the optical flow path of the flock of features, the processor is programmed to apply at least one constraint on each of the flock of features such that the flock of features maintain a minimum distance from each other.

49. The apparatus according to claim 42, where, in being programmed to infer a hand pose type using the trained multiclass SVM, the processor is programmed to:

detect at least one feature within a training image and the at least one input image; and

perform a one-to-one mapping of instances of the at least one feature within the at least one input image with at least one label drawn from a finite set of elements.

50. The apparatus according to claim 49, where the at least one label comprises at least one label generated during a training phase based upon a motion capture three dimensional (3D) data set.

51. The apparatus according to claim 42, where, in being programmed to approximate the hand pose using IK optimization, the processor is programmed to:

partition the at least one input image into a plurality of processing regions;

determine a centroid of features within each of the plurality of processing regions; and

map a location of each feature centroid onto three dimensional (3D) pose data associated with a motion capture data set.

52. The apparatus according to claim 51, where, in being programmed to determine the centroid of features within each of the plurality of processing regions, the processor is programmed to:

compare variances from each feature centroid to a closest match within the 3D pose data; and

determine which of a plurality of joint constraints affect the 1K optimization.

53. The apparatus according to claim 52, where the processor is further programmed to map each feature centroid to a closest joint stored within the 3D pose data.

54. The apparatus according to claim 53, where the processor is further programmed to:

minimize a distance of each mapped closest joint within a training image based upon the 3D pose data; and

determine a final hand position based upon the minimized distance of each mapped closest joint within the training image.

55. The apparatus according to claim 42, where the processor is further programmed to:

define a configurable resolution threshold for image processing; and

adjust the configurable resolution threshold.

56. The apparatus according to claim 42, further comprising:

a memory; and

where the processor is further programmed to store the extracted feature set associated with the hand gesture detection and the hand pose inference in the memory.