Real time hand tracking, pose classification, and interface control

• US 8,755,568 B2
• Filed: 09/27/2013
• Issued: 06/17/2014
• Est. Priority Date: 11/06/2009
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Adhikari, Suranjit
• Primary Examiner(s): Werner, Brian P
• Application Number:
  US14/039,596
• Publication Number:
  US 20140028550A1
• Time in Patent Office: 263 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A hand gesture from a camera input is detected using an image processing module of a consumer electronics device. The detected hand gesture is identified from a vocabulary of hand gestures. The electronics device is controlled in response to the identified hand gesture. This abstract is not to be considered limiting, since other embodiments may deviate from the features described in this abstract.

Claims

1. A method of controlling an electronics device via hand gestures, comprising:

detecting, via an image processing module of the electronics device, a bare-hand position via a camera input based upon a detected sequence of bare-hand positions;

determining whether the bare-hand position has been detected for a threshold duration of time;

identifying the detected bare-hand position from a vocabulary of hand gestures by;

measuring brightness gradients in multiple directions across a plurality of images captured by the camera;

generating image pyramids from measured brightness gradients of the extracted hand gesture;

extracting pixel intensity/displacement features and scale invariant feature transform (SIFT) features using the generated pyramids;

applying a cascade filter in association with extracting the pixel intensity/displacement features and the SIFT features from the generated image pyramids;

classifying a hand pose type using a trained multiclass support vector machine (SVM); and

determining the hand gesture by a distance calculation that calculates and selects a closest gesture in the vocabulary of hand gestures, where the closest gesture comprises a gesture with hand joints closest to the hand joints in the vocabulary of gestures;

where the identified bare-hand position comprises a hand gesture associated with powering on the electronics device; and

controlling, in response to determining that the bare-hand position has been detected for the threshold duration of time, the electronics device in response to the identified bare-hand position by powering on the electronics device.

2. A computer readable storage medium storing instructions which, when executed on one or more programmed processors, carry out the method according to claim 1.

3. A method of controlling an electronics device via hand gestures, comprising:

detecting, via an image processing module of the electronics device, a hand gesture via a camera input;

identifying the detected hand gesture from a vocabulary of hand gestures by;

measuring brightness gradients in multiple directions across a plurality of images captured by the camera;

generating image pyramids from measured brightness gradients of the extracted hand gesture;

extracting pixel intensity/displacement features and scale invariant feature transform (SIFT) features using the generated pyramids;

applying a cascade filter in association with extracting the pixel intensity/displacement features and the SIFT features from the generated image pyramids;

classifying a hand pose type using a trained multiclass support vector machine (SVM); and

determining the hand gesture by a distance calculation that calculates and selects a closest gesture in the vocabulary of hand gestures, where the closest gesture comprises a gesture with hand joints closest to the hand joints in the vocabulary of gestures; and

controlling the electronics device in response to the identified hand gesture.

4. The method according to claim 3, where detecting, via the image processing module of the electronics device, the hand gesture via the camera input comprises detecting a bare-hand position.

5. The method according to claim 3, where detecting, via the image processing module of the electronics device, the hand gesture via the camera input comprises detecting a sequence of hand positions.

6. The method according to claim 3, where the identified hand gesture comprises a hand gesture associated with powering on the electronics device and where controlling the electronics device in response to the identified hand gesture comprises powering on the electronics device.

7. The method according to claim 3, where the identified hand gesture comprises a hand gesture associated with powering off the electronics device and where controlling the electronics device in response to the identified hand gesture comprises powering off the electronics device.

8. The method according to claim 3, further comprising:

determining whether the hand gesture associated with the control of the electronics device has been detected for a threshold duration of time; and

where detecting, via the image processing module of the electronics device, the hand gesture via the camera input comprises detecting the hand gesture associated with the control of the electronics device for the threshold duration of time.

9. The method according to claim 3, further comprising:

determining whether the hand gesture associated with the control of the electronics device has been detected for a threshold duration of time; and

where identifying the detected hand gesture from the vocabulary of hand gestures comprises identifying the detected hand gesture from the vocabulary of hand gestures in response to determining that the hand gesture associated with the control of the electronics device has been detected for the threshold duration of time.

10. The method according to claim 3, further comprising:

detecting user input indicating assignment of one of the vocabulary of hand gestures to a control function of the electronics device; and

assigning the one of the vocabulary of hand gestures to the control function of the electronics device.

11. The method according to claim 10, where detecting the user input indicating the assignment of the one of the vocabulary of hand gestures to the control function of the electronics device comprises detecting a hand gesture associated with the assignment of the one of the vocabulary of hand gestures to the control function of the electronics device.

12. A computer readable storage medium storing instructions which, when executed on one or more programmed processors, carry out the method according to claim 3.

13. An apparatus for controlling an electronics device via hand gestures, comprising:

a camera; and

a processor programmed to;

detect a bare-hand position via the camera based upon a detected sequence of bare-hand positions;

determine whether the bare-hand position has been detected for a threshold duration of time;

identify the detected bare-hand position from a vocabulary of hand gestures by;

measuring brightness gradients in multiple directions across a plurality of images captured by the camera;

generating image pyramids from measured brightness gradients of the extracted hand gesture;

extracting pixel intensity/displacement features and scale invariant feature transform (SIFT) features using the generated pyramids;

applying a cascade filter in association with extracting the pixel intensity/displacement features and the SIFT features from the generated image pyramids;

classifying a hand pose type using a trained multiclass support vector machine (SVM); and

determining the hand gesture by a distance calculation that calculates and selects a closest gesture in the vocabulary of hand gestures, where the closest gesture comprises a gesture with hand joints closest to the hand joints in the vocabulary of gestures;

where the identified bare-hand position comprises a hand gesture associated with powering on the electronics device; and

control, in response to determining that the bare-hand position has been detected for the threshold duration of time, the electronics device in response to the identified bare-hand position by powering on the electronics device.

14. An apparatus for controlling an electronics device via hand gestures, comprising:

a camera; and

a processor programmed to;

detect a hand gesture via the camera;

identify the detected hand gesture from a vocabulary of hand gestures by;

measuring brightness gradients in multiple directions across a plurality of images captured by the camera;

generating image pyramids from measured brightness gradients of the extracted hand gesture;

extracting pixel intensity/displacement features and scale invariant feature transform (SIFT) features using the generated pyramids;

applying a cascade filter in association with extracting the pixel intensity/displacement features and the SIFT features from the generated image pyramids;

classifying a hand pose type using a trained multiclass support vector machine (SVM); and

determining the hand gesture by a distance calculation that calculates and selects a closest gesture in the vocabulary of hand gestures, where the closest gesture comprises a gesture with hand joints closest to the hand joints in the vocabulary of gestures; and

control the electronics device in response to the identified hand gesture.

15. The apparatus according to claim 14, where, in being programmed to detect the hand gesture via the camera, the processor is programmed to detect a bare-hand position.

16. The apparatus according to claim 14, where, in being programmed to detect the hand gesture via the camera, the processor is programmed to detect a sequence of hand positions.

17. The apparatus according to claim 14, where the identified hand gesture comprises a hand gesture associated with powering on the electronics device and where, in being programmed to control the electronics device in response to the identified hand gesture, the processor is programmed to power on the electronics device.

18. The apparatus according to claim 14, where the identified hand gesture comprises a hand gesture associated with powering off the electronics device and where, in being programmed to control the electronics device in response to the identified hand gesture, the processor is programmed to power off the electronics device.

19. The apparatus according to claim 14, where the processor is further programmed to:

determine whether the hand gesture associated with the control of the electronics device has been detected for a threshold duration of time; and

where, in being programmed to detect the hand gesture via the camera, the processor is programmed to detect the hand gesture associated with the control of the electronics device for the threshold duration of time.

20. The apparatus according to claim 14, where the processor is further programmed to:

determine whether the hand gesture associated with the control of the electronics device has been detected for a threshold duration of time; and

where, in being programmed to identify the detected hand gesture from the vocabulary of hand gestures, the processor is programmed to identify the detected hand gesture from the vocabulary of hand gestures in response to determining that the hand gesture associated with the control of the electronics device has been detected for the threshold duration of time.

21. The apparatus according to claim 14, where the processor is further programmed to:

detect user input indicating assignment of one of the vocabulary of hand gestures to a control function of the electronics device; and

assign the one of the vocabulary of hand gestures to the control function of the electronics device.

22. The apparatus according to claim 21, where, in being programmed to detect the user input indicating the assignment of the one of the vocabulary of hand gestures to the control function of the electronics device, the processor is programmed to detect a hand gesture associated with the assignment of the one of the vocabulary of hand gestures to the control function of the electronics device.