Recognizing finger gestures from forearm EMG signals

• US 8,892,479 B2
• Filed: 04/20/2013
• Issued: 11/18/2014
• Est. Priority Date: 06/26/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tan, Desney, Morris, Dan, Saponas, T. Scott, Balakrishnan, Ravin
• Primary Examiner(s): Wong, Lut
• Application Number:
  US13/867,060
• Publication Number:
  US 20130232095A1
• Time in Patent Office: 577 Days
• Field of Search: 706/12, 706/62
• US Class Current: 706/12
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

A machine learning model is trained by instructing a user to perform various predefined gestures, sampling signals from EMG sensors arranged arbitrarily on the user'"'"'s forearm with respect to locations of muscles in the forearm, extracting feature samples from the sampled signals, labeling the feature samples according to the corresponding gestures instructed to be performed, and training the machine learning model with the labeled feature samples. Subsequently, gestures may be recognized using the trained machine learning model by sampling signals from the EMG sensors, extracting from the signals unlabeled feature samples of a same type as those extracted during the training, passing the unlabeled feature samples to the machine learning model, and outputting from the machine learning model indicia of a gesture classified by the machine learning model.

Claims

1. A method for identifying individual finger movements, comprising:

arbitrarily arranging a plurality of electromyography (EMG) sensors on a user'"'"'s forearm;

performing one or more predefined finger gestures while using the EMG sensors to sample EMG signals generated by corresponding muscle contractions of the user;

concurrent with the sampling of the EMG signals, monitoring the predefined finger gestures using a secondary input mechanism to verify that correct gestures were performed;

extracting feature samples from the sampled EMG signals and labeling the feature samples according to corresponding finger gestures that have been verified as correct by the secondary input mechanism; and

training a machine learning model with the labeled feature samples for use in identifying arbitrary finger gestures from an evaluation of corresponding EMG signal samples.

2. The method of claim 1 wherein the predefined finger gestures comprise concurrent movement of two or more different fingers.

3. The method of claim 1 wherein the predefined finger gestures are finger taps.

4. The method of claim 3 wherein the secondary input mechanism is a keyboard that detects the finger taps concurrent with corresponding sampled EMG signals.

5. The method of claim 1 wherein the user is provided with instructions for performing one or more of the predefined finger gestures.

6. The method of claim 1 wherein one or more of the predefined finger gestures include intentional forearm muscle contractions corresponding to the predefined figure gestures performed by users missing at least a portion of corresponding fingers.

7. The method of claim 1 wherein one or more of the predefined finger gestures include attempted finger movement against a resistive force causing corresponding muscle contractions in the user'"'"'s forearm.

8. The method of claim 1 wherein one or more of the arbitrary finger gestures include attempted finger movement against a resistive force causing corresponding muscle contractions in the user'"'"'s forearm.

9. The method of claim 1 wherein one or more of the predefined finger gestures comprise a plurality of pressure levels applied by one or more fingers.

10. The method of claim 1 wherein the predefined finger gestures include using one or more particular fingers to perform one or more of tap gestures, curl gestures, extension gestures, press gestures, and lift gestures.

11. The method of claim 1 wherein using the trained machine learning model to evaluate EMG signal samples further comprises:

evaluating a sequence of two or more successive EMG signal samples to identify arbitrary finger gestures corresponding to each sample;

aggregating each sequential identified arbitrary finger gesture; and

evaluating the aggregate identifications to identify a single arbitrary finger gesture corresponding to the sequence of two or more successive EMG signal samples.

12. One or more computer readable memories storing information to enable a computing device to perform a process comprising:

using a plurality of electromyography (EMG) sensors arbitrarily arranged on a user'"'"'s forearm to obtain samples of EMG signals generated by muscle contractions of a user while the user is performing one or more predefined finger gestures;

monitoring the predefined finger gestures using a secondary input mechanism to verify that correct gestures were performed;

extracting feature samples from the sampled EMG signals and labeling feature samples according to the corresponding finger gestures that have been verified as correct by the secondary input mechanism;

training a machine learning model with the labeled feature samples; and

using the trained machine learning model to evaluate EMG signal samples obtained during arbitrary finger gestures to identify those arbitrary finger gestures relative to one or more of the predefined finger gestures performed by the user.

13. The computer readable memories of claim 12 wherein the trained machine learning model:

evaluates a sequence of two or more successive EMG signal samples to identify arbitrary finger gestures corresponding to each sample;

aggregates each sequential identified arbitrary finger gesture; and

evaluates the aggregate identifications to identify a single arbitrary finger gesture corresponding to the sequence of two or more successive EMG signal samples.

14. The computer readable memories of claim 12 wherein the predefined finger gestures comprise concurrent movement of two or more different fingers.

15. The computer readable memories of claim 12 wherein one or more of the predefined finger gestures include intentional forearm muscle contractions corresponding to the predefined figure gestures performed by users missing at least a portion of corresponding fingers.

16. The computer readable memories of claim 12 wherein one or more finger gestures include attempted finger movement against a resistive force causing corresponding muscle contractions in the user'"'"'s forearm.

17. A device for determining user gestures based on electromyography (EMG) signals derived from EMG sensors, comprising:

a plurality of EMG sensors arbitrarily arranged on a user'"'"'s forearm;

a processor for interacting with one or more modules;

a signal analysis module configured to obtain samples from one or more of the EMG sensors of EMG signals generated by muscle contractions of a user while the user is performing one or more predefined finger gestures;

wherein the signal analysis module further monitors the predefined finger gestures using a secondary input mechanism to verify that correct predefined finger gestures were performed;

a feature extraction module configured to extract feature samples from the sampled EMG signals and label the feature samples according to corresponding finger gestures that have been verified as correct by the secondary input mechanism; and

a training module configured to train a machine learning model using the labeled feature samples

to identify arbitrary finger gestures relative to one or more of the predefined finger gestures performed by the user.

18. The device of claim 17 wherein one or more of the predefined finger gestures are performed while the user is maintaining one or more predefined arm positions including, one or more of arm bent, arm extended, palm up, and palm down.

19. The device of claim 17 wherein the predefined finger gestures comprise concurrent movement of two or more different fingers.

20. The device of claim 17 wherein one or more finger gestures include attempted finger movement against a resistive force causing corresponding muscle contractions in the user'"'"'s forearm.