Recognizing gestures from forearm EMG signals

• US 8,447,704 B2
• Filed: 06/26/2008
• Issued: 05/21/2013
• Est. Priority Date: 06/26/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tan, Desney, Morris, Dan, Saponas, Scott, Balakrishnan, Ravin
• Primary Examiner(s): Wong, Lut
• Application Number:
  US12/146,471
• Publication Number:
  US 20090327171A1
• Time in Patent Office: 1,790 Days
• Field of Search: 706/62, 706/12
• US Class Current: 706/12
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

A machine learning model is trained by instructing a user to perform proscribed gestures, sampling signals from EMG sensors arranged arbitrarily on the user'"'"'s forearm with respect to locations of muscles in the forearm, extracting feature samples from the sampled signals, labeling the feature samples according to the corresponding gestures instructed to be performed, and training the machine learning model with the labeled feature samples. Subsequently, gestures may be recognized using the trained machine learning model by sampling signals from the EMG sensors, extracting from the signals unlabeled feature samples of a same type as those extracted during the training, passing the unlabeled feature samples to the machine learning model, and outputting from the machine learning model indicia of a gesture classified by the machine learning model.

Claims

1. One or more computer readable memories storing information to enable a computing device to perform a process, the process comprising:

receiving a plurality of electromyography (EMG) signals derived from respective EMG sensors, the sensors being arranged in a wearable device placed arbitrarily on the forearm;

dividing the EMG signals into a sequence of signal samples, each signal sample comprising signal segments of the EMG signals of the respective EMG sensors;

for each signal sample, forming a corresponding feature vector by extracting from the signal sample a plurality of values of different types of features based on the signal segments of the signal sample;

wherein feature vectors include two or more of amplitudes of individual channels, ratios of amplitudes for channel pairs, coherence ratios for pairs of channels, and frequency energy broken down over subbands;

passing the feature vectors to a machine learning module previously trained during a training session with feature vectors labeled with known gestures to determine gesture classifications of the respective feature vectors for each signal sample;

wherein the training session includes instructing a user to perform a gesture with different arm positions, including, one or more of arm bent, arm extended, palm up, and palm down; and

selecting a single one of the gesture classifications of the respective feature vectors and outputting a particular type of finger movement corresponding to the selected gesture classification.

2. One or more computer readable memories according to claim 1, wherein based on a first of the feature vectors the machine learning module outputs a first classification comprising identity of a first of a plurality of particular types of finger movements for which the machine learning module has been trained to classify, and based on a second of the feature vectors the machine learning module outputs a second classification comprising identity of a second of the plurality of types of finger movements.

3. One or more computer readable memories according to claim 2, wherein the process further comprises training the machine learning module by, when the user is performing known gestures, performing the dividing, and forming, and also labeling the corresponding feature vectors according to the known gestures being performed when the signal samples of the feature vectors were captured, where the labeled feature vectors train the machine learning module.

4. One or more computer readable memories according to claim 3, the process further comprising ignoring, during the training, an initial period of signal data after a known gesture training event begins.

5. One or more computer readable memories according to claim 4, wherein the training data is generated by:

instructing the user to perform a single particular specified finger gesture; and

during performance of the specified gesture, capturing the signal for a given sensor and obtaining a plurality of segments of the captured signal, for each segment deriving a sample comprising a plurality of signal characteristics of the corresponding segment, and labeling each of the samples as corresponding to the particular specified gesture, wherein the machine learning module is trained with the samples, and whereby a plurality of labeled samples are obtained corresponding to the single specified finger gesture.

6. One or more computer readable memories according to claim 1 wherein the different types of features in a feature vector include both individual features extracted only from corresponding individual segments of a signal sample, and combined features that are based on pairs of the individual features.

7. One or more computer readable memories according to claim 1, the process further comprising analyzing a plurality of the outputted classifications to identify a gesture.

8. A computer-implemented process for determining user gestures, comprising using a computing device to perform process actions for:

receiving a plurality of electromyography (EMG) signals derived from respective EMG sensors, the sensors being arranged in a wearable device placed arbitrarily on the forearm;

dividing the EMG signals into a sequence of signal samples, each signal sample comprising signal segments of the EMG signals of the respective EMG sensors;

for each signal sample forming a corresponding feature vector by extracting from the signal sample a plurality of values of different types of features based on the signal segments of the signal sample;

wherein feature vectors include two or more of amplitudes of individual channels, ratios of amplitudes for channel pairs, coherence ratios for pairs of channels, and frequency energy broken down over subbands;

passing the feature vectors to a machine learning module previously trained during a training session with feature vectors labeled with known gestures to determine gesture classifications of the respective feature vectors for each signal sample;

wherein the training session includes instructing a user to perform a gesture with different arm positions, including, one or more of arm bent, arm extended, palm up, and palm down; and

selecting a single one of the gesture classifications of the respective feature vectors and outputting a particular type of finger movement corresponding to the selected gesture classification.

9. The computer-implemented process of claim 8, wherein based on a first of the feature vectors the machine learning module outputs a first classification comprising identity of a first of a plurality of particular types of finger movements for which the machine learning module has been trained to classify, and based on a second of the feature vectors the machine learning module outputs a second classification comprising identity of a second of the plurality of types of finger movements.

10. The computer-implemented process of claim 9, wherein the process further comprises training the machine learning module by, when the user is performing known gestures, performing the dividing, and forming, and also labeling the corresponding feature vectors according to the known gestures being performed when the signal samples of the feature vectors were captured, where the labeled feature vectors train the machine learning module.

11. The computer-implemented process of claim 9, further comprising ignoring, during the training, an initial period of signal data after a known gesture training event begins.

12. The computer-implemented process of claim 11, wherein the training data is generated by:

instructing the user to perform a single particular specified finger gesture; and

during performance of the specified gesture, capturing the signal for a given sensor and obtaining a plurality of segments of the captured signal, for each segment deriving a sample comprising a plurality of signal characteristics of the corresponding segment, and labeling each of the samples as corresponding to the particular specified gesture, wherein the machine learning module is trained with the samples, and whereby a plurality of labeled samples are obtained corresponding to the single specified finger gesture.

13. The computer-implemented process of claim 8 wherein the different types of features in a feature vector include both individual features extracted only from corresponding individual segments of a signal sample, and combined features that are based on pairs of the individual features.

14. The computer-implemented process of claim 8 further comprising analyzing a plurality of the outputted classifications to identify a gesture.

15. A device for determining user gestures based on electromyography (EMG) signals derived from EMG sensors, comprising:

a processor for interacting with one or more modules;

a feature extraction module configured to;

receive a plurality of EMG signals derived from a plurality of EMG sensors arranged in a wearable device placed arbitrarily on the forearm,

divide the EMG signals into a sequence of signal samples, each signal sample comprising signal segments of the EMG signals of the EMG sensors,

form a corresponding feature vector for each signal sample by extracting from the signal sample a plurality of values of different types of features based on the signal segments of the signal sample, and

wherein feature vectors include two or more of amplitudes of individual channels, ratios of amplitudes for channel pairs, coherence ratios for pairs of channels, and frequency energy broken down over subbands;

a machine learning module configured to receive feature vectors, the machine learning module having been previously trained during a training session with feature vectors labeled with known gestures to determine gesture classifications of the respective feature vectors for each signal sample;

wherein the training session includes instructing a user to perform one or more gestures with different arm positions, including, one or more of arm bent, arm extended, palm up, and palm down; and

a gesture analysis module configured to select a single one of the gesture classifications of the respective feature vectors and outputting a particular type of finger movement corresponding to the selected gesture classification.

16. The device of claim 15 wherein, based on a first of the feature vectors, the machine learning module outputs a first classification comprising identity of a first of a plurality of particular types of finger movements for which the machine learning module has been trained to classify, and based on a second of the feature vectors the machine learning module outputs a second classification comprising identity of a second of the plurality of types of finger movements.

17. The device of claim 16 further comprising ignoring, during the training, an initial period of signal data after a known gesture training event begins.

18. The device of claim 17 wherein the training data is generated by:

instructing the user to perform a single particular specified finger gesture; and

during performance of the specified gesture, capturing the signal for a given sensor and obtaining a plurality of segments of the captured signal, for each segment deriving a sample comprising a plurality of signal characteristics of the corresponding segment, and labeling each of the samples as corresponding to the particular specified gesture, wherein the machine learning module is trained with the samples, and whereby a plurality of labeled samples are obtained corresponding to the single specified finger gesture.

19. The device of claim 15 wherein the different types of features in a feature vector include both individual features extracted only from corresponding individual segments of a signal sample, and combined features that are based on pairs of the individual features.

20. The device of claim 15 further comprising analyzing a plurality of the outputted classifications to identify a gesture.