METHOD AND SYSTEM FOR GESTURE RECOGNITION

• US 20100208038A1
• Filed: 02/17/2010
• Published: 08/19/2010
• Est. Priority Date: 02/17/2009
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Omek Interactive Ltd. (Intel Corporation)
• Inventors: Bleiweiss, Amit, Eilat, Eran, Kutliroff, Gershom
• Granted Patent: US 8,824,802 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A method of image acquisition and data pre-processing includes obtaining from a sensor an image of a subject making a movement. The sensor may be a depth camera. The method also includes selecting a plurality of features of interest from the image, sampling a plurality of depth values corresponding to the plurality of features of interest, projecting the plurality of features of interest onto a model utilizing the plurality of depth values, and constraining the projecting of the plurality of features of interest onto the model utilizing a constraint system. The constraint system may comprise an inverse kinematics solver.

Claims

1. A method of recognizing a gesture of interest comprising:

prompting a subject to perform the gesture of interest;

obtaining from a sensor a plurality of images of the subject making movements;

comparing each of the plurality of images to the gesture of interest as each of the plurality of images is obtained;

determining a tracking score based on the comparing; and

determining that the subject is performing the gesture of interest if the tracking score remains within a given threshold.

2. The method of claim 1, further comprising constraining each of the plurality of images to a model utilizing an inverse kinematics solver during the comparing.

3. The method of claim 1, further comprising selecting the gesture of interest from a gesture library.

4. The method of claim 1, wherein the comparing includes computing a similarity measure.

5. The method of claim 1, wherein the sensor comprises a depth camera.

21. A method as recited in claim 1, wherein determing the tracking score includes utilizing a support vector machine on the gesture of interest.

22. A method as recited in claim 1, wherein determining the tracking score includes utilizing a similarity measure.

6. A method of image acquisition and data pre-processing comprising:

obtaining from a sensor an image of a subject making a movement;

selecting a plurality of features of interest from the image;

sampling a plurality of depth values corresponding to the plurality of features of interest;

projecting the plurality of features of interest onto a model utilizing the plurality of depth values; and

constraining the projecting the plurality of features of interest onto the model utilizing a constraint system.

7. The method of claim 6, wherein the constraint system comprises an inverse kinematics solver.

8. The method of claim 6, wherein the sensor comprises a depth camera.

9. The method of claim 6, further comprising animation retargeting for scaling the model onto a standard model.

10. A system for recognizing gestures, comprising:

a camera for acquiring multiple frames of image depth data;

an image acquisition module configured to receive the multiple frames of image depth data from the camera and process the image depth data to determine feature positions of a subject;

a gesture training module configured to receive the feature positions of the subject from the image acquisition module and associate the feature positions with a pre-determined gesture;

a binary gesture recognition module configured to receive the feature positions of the subject from the image acquisition module and determine whether the feature positions match a particular gesture;

a real-time gesture recognition module configured to receive the feature positions of the subject from the image acquisition module and determine whether the particular gesture is being performed over more than one frame of image depth data.

11. The system of claim 10, further comprising:

a game engine module configured to select the particular gesture and prompt the subject to perform the particular gesture.

12. The system of claim 10, further comprising:

a gesture database that contains the pre-determined gestures.

13. The system of claim 10, further comprising:

a display for providing feedback to the subject about gestures performed by the subject.

14. The system of claim 10 wherein the camera further acquires color image data.

15. The system of claim 10 wherein the real-time gesture recognition module is further configured to calculate a similarity measure and a cumulative tracking score, wherein the similarity measure and the cumulative tracking score are updated for each frame, and further wherein the determination whether the particular gesture is being performed is based upon comparing the cumulative tracking score to a threshold for the particular gesture for each frame.

16. The system of claim 10 wherein the image acquisition module maps the image depth data onto a model.

17. The system of claim 16 wherein the image acquisition module uses an inverse kinematic solver to constrain movements of the model for feature positions not visible in at least one of the multiple frames.

18. The system of claim 10 wherein the image acquisition module scales the feature positions to a standard model.

19. The system of claim 10 wherein the gesture training module uses machine learning techniques to determine whether the feature positions match a particular gesture.

20. A system for recognizing gestures, comprising:

means for acquiring multiple frames of image depth data;

means for receiving the multiple frames of image depth data from the camera and processing the image depth data to determine feature positions of a subject;

means for receiving the feature positions of the subject from the image acquisition module and associating the feature positions with a pre-determined gesture;

means for receiving the feature positions of the subject from the image acquisition module and determining whether the feature positions match a particular gesture;

means for receiving the feature positions of the subject from the image acquisition module and determining whether the particular gesture is being performed over more than one frame of image depth data.

means for providing feedback to the subject about gestures performed by the subject.