System and method for automatically defining and identifying a gesture

• US 8,958,631 B2
• Filed: 12/02/2011
• Issued: 02/17/2015
• Est. Priority Date: 12/02/2011
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Kutliroff, Gershom, Ben-Ari, Itamar, Bleiweiss, Amit, Eshar, Dagan
• Primary Examiner(s): Werner, Brian P
• Application Number:
  US13/310,510
• Publication Number:
  US 20130142417A1
• Time in Patent Office: 1,173 Days
• Field of Search: None
• US Class Current: 382/159
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A system and method for creating a gesture and generating a classifier that can identify the gesture for use with an application is described. The designer constructs a training set of data containing positive and negative examples of the gesture. Machine learning algorithms are used to compute the optimal classification of the training data into positive and negative instances of the gesture. The machine learning algorithms generate a classifier which, given input data, makes a decision on whether the gesture was performed in the input data or not.

Claims

1. A method comprising:

recording test sequences of one or more users performing examples of sequences;

recording a training set of gestures;

marking the training set of gestures by;

marking examples of a correct performance of the gesture as positive examples; and

marking examples of an incorrect performance of the gesture as negative examples; and

applying a machine learning algorithm to the marked training set to generate a classifier to determine if movements not included in the training set of gestures is similar to a positive example;

testing the classifier on the test sequences to detect a predetermined number of false positives and false negatives, wherein the test sequences are distinct from the training set of gestures.

2. The method of claim 1, wherein applying the machine learning algorithm to the marked training further comprises:

generating a second classifier; and

selecting a best classifier from among the first and second classifiers.

3. The method of claim 1, wherein the machine learning algorithm is applied to different subsets of features points of a subject performing the gestures recorded for the training set and different features sets in the marked training set to generate a plurality of potential gesture recognition models, and further wherein a best performing one of the plurality of potential gesture recognition models is selected as the gesture recognition model, wherein features sets include raw image data, object segmentation data, and subsets of tracking data of user feature points.

4. The method of claim 1, wherein marking the training set comprises using voice recognition to analyze statements provided by a subject performing the gesture to indicate when the gesture begins.

5. The method of claim 1, wherein marking the training set comprises using a remote control by a subject performing the gesture to indicate a start time of the gesture.

6. The method of claim 1, wherein marking the training set comprises prompting a subject to perform the gesture at a specific time.

7. The method of claim 1, further comprising:

testing the gesture recognition model;

upon determining that performance of the model is unsatisfactory, recording an additional training set of gestures, and applying the machine learning algorithm to the additional training set to generate an improved gesture recognition model.

8. The method of claim 1, wherein the machine learning algorithm is applied to raw image data, object segmentation data or tracking data corresponding to at least a subset of feature paints of a subject performing the gestures recorded in the training set.

9. The method of claim 1, wherein the machine learning algorithm includes support vector machines, hidden Markov models, and/or neural networks.

10. The method of claim 1, wherein gestures recorded in the training set are performed by more than one subject.

11. A method comprising:

recording test sequences of one or more users performing examples of sequences;

prompting a user via a graphical user interface (GUI) to perform a training gesture;

recording a plurality of performances of the training gesture by the user;

applying a machine learning algorithm to the plurality of performances to train a classifier for the gesture recognition model; and

testing the classifier on the test sequences to detect a predetermined number of false positives and false negatives, wherein the test sequences are distinct from the training set of gestures.

12. The method of claim 11, further comprising:

determining whether a number of the recorded plurality of performances is sufficient;

upon determining that the number is insufficient, requesting the user to perform additional performances of the gesture for recording.

13. The method of claim 11, further comprising:

upon determining that performance of the model is unsatisfactory, requesting the user to perform additional performances of the gesture for recording, and applying the machine learning algorithm to the additional performances to generate an improved gesture recognition model.

14. The method of claim 11, wherein the machine learning algorithm is support vector machines, hidden Markov models and/or neural networks.

15. The method of claim 11, wherein the machine learning algorithm is applied to raw image data, object segmentation data, or tracking data corresponding to at least a subset of the user'"'"'s feature points in the plurality of performances.

16. An apparatus for generating a gesture recognition model for determining if a gesture has been performed by one of a plurality of users, the apparatus comprising:

a recording module to record sequences of one or more users performing examples of sequences and a training set of gestures;

a gesture information module to receive information associated with the gestures recorded in the training set from a designer;

a marking module to mark the training set of gestures by marking examples of a correct performance of the gesture as positive examples and marking examples of an incorrect performance of the gesture as negative examples

a machine learning module to apply a machine learning algorithm to generate a classifier to determine if movements not included in the training set of gestures is similar to a positive example; and

a testing module to test the classifier on the test sequences to detect a predetermined number of false positives and false negatives, wherein the test sequences are distinct from the training set of gestures.

17. The apparatus of claim 16, wherein machine learning module generates a second classifier and selects a best classifier from amongst the first and second classifiers.

18. The apparatus of claim 16, wherein the machine learning algorithm is applied to different subsets of the user'"'"'s features points and different features sets in the marked training set of gestures to generate a plurality of potential gesture recognition models, and further wherein a best performing one of the plurality of potential gesture recognition models is selected as the gesture recognition model, wherein features sets include raw image data, object segmentation data, and subsets of tracking data of user feature points.

19. The apparatus of claim 16, wherein information associated with the gestures comprises voice recognition of statements provided by a subject performing the gesture to indicate when the gesture begins.

20. The apparatus of claim 16, wherein information associated with the gestures comprises output from a remote control provided by a subject performing the gesture to indicate a start time of the gesture.

21. The apparatus of claim 16, wherein information associated with the gestures comprises timing information used for prompting a subject to perform the gesture at a specific time.

22. An apparatus for generating a gesture recognition model for determining if a gesture has been performed by a user, the apparatus comprising:

a recording module configured to record test sequences of one or more users performing examples of sequences;

a graphical user interface (GUI) to prompt a user to perform a training gesture;

a recording module configured to record sequences of one or more users performing examples of sequences and a plurality of performances of the training gesture by the user;

a machine learning module configured to apply a machine learning algorithm to the plurality of performances to train a classifier for the gesture recognition model; and

a testing module to test the classifier on the test sequences to detect a predetermined number of false positives and false negatives, wherein the test sequences are distinct from the training set of gestures.

23. The apparatus of claim 22, wherein the machine learning algorithm is applied to raw image data, object segmentation data, or tracking data corresponding to at least a subset of the user'"'"'s feature points of the plurality of performances.

24. The apparatus of claim 22, wherein the testing module determines whether a performance of the gesture recognition model is satisfactory.

25. A non-transitory computer readable medium having instructions, which when executed by a processor, cause the processor to form operations comprising:

recording test sequences of one or more users performing examples of sequences;

recording a training set of gestures;

marking the training set of gestures by;

marking examples of a correct performance of the gesture as positive examples; and

marking examples of an incorrect performance of the gesture as negative examples; and

applying a machine learning algorithm to the marked training set to generate a classifier to determine if movements not included in the training set of gestures is similar to a positive example;

testing the classifier on the test sequences to detect a predetermined number of false positives and false negatives, wherein the test sequences are distinct from the training set of gestures.

26. The computer readable medium of claim 25, wherein applying the machine learning algorithm to the marked training further comprises:

generating a second classifier; and

selecting a best classifier from among the first and second classifiers.

27. The computer readable medium of claim 25, wherein the machine learning algorithm is applied to different subsets of features points of a subject performing the gestures recorded for the training set and different features sets in the marked training set to generate a plurality of potential gesture recognition models, and further wherein a best performing one of the plurality of potential gesture recognition models is selected as the gesture recognition model, wherein features sets include raw image data, object segmentation data, and subsets of tracking data of user feature points.

28. A non-transitory computer readable medium having instructions, which when executed by a processor, cause the processor to form operations comprising:

recording test sequences of one or more users performing examples of sequences;

prompting a user via a graphical user interface (GUI) to perform a training gesture;

recording a plurality of performances of the training gesture by the user;

applying a machine learning algorithm to the plurality of performances to train a classifier for the gesture recognition model; and

testing the classifier on the test sequences to detect a predetermined number of false positives and false negatives, wherein the test sequences are distinct from the training set of gestures.

29. The computer readable medium having instructions of claim 28, which when executed by a processor, further cause the processor to form operations comprising:

determining whether a number of the recorded plurality of performances is sufficient;

upon determining that the number is insufficient, requesting the user to perform additional performances of the gesture for recording.

30. The computer readable medium having instructions of claim 28, which when executed by a processor, further cause the processor to form operations comprising:

upon determining that performance of the model is unsatisfactory, requesting the user to perform additional performances of the gesture for recording, and applying the machine learning algorithm to the additional performances to generate an improved gesture recognition model.