Systems, articles, and methods for gesture identification in wearable electromyography devices

• US 9,483,123 B2
• Filed: 09/23/2014
• Issued: 11/01/2016
• Est. Priority Date: 09/23/2013
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Thalmic Labs Inc.
• Inventors: Aleem, Idris S., Ataee, Pedram, Bailey, Matthew
• Primary Examiner(s): Bukowski, Kenneth
• Application Number:
  US14/494,274
• Publication Number:
  US 20150084860A1
• Time in Patent Office: 770 Days
• Field of Search: 600/546
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

Systems, articles, and methods for performing gesture identification with improved robustness against variations in use parameters and without requiring a user to undergo an extensive training procedure are described. A wearable electromyography (“EMG”) device includes multiple EMG sensors, an on-board processor, and a non-transitory processor-readable storage medium that stores data and/or processor-executable instructions for performing gesture identification. The wearable EMG device detects, determines, and ranks features in the signal data provided by the EMG sensors and generates a digit string based on the ranked features. The permutation of the digit string is indicative of the gesture performed by the user, which is identified by testing the permutation of the digit string against multiple sets of defined permutation conditions. A single reference gesture may be performed by the user to (re-)calibrate the wearable EMG device before and/or during use.

Claims

1. A method performed by a wearable electromyography (“

EMG”

) device to identify a gesture performed by a user operating the wearable EMG device, wherein the wearable EMG device includes a set of EMG sensors and a processor, the set of EMG sensors communicatively coupled to the processor, and the method comprising;

detecting muscle activity of the user of the wearable EMG device by the set of EMG sensors, wherein the muscle activity corresponds to the gesture performed by the user;

in response to detecting muscle activity of the user by the set of EMG sensors, providing a set of signals from the set of EMG sensors to the processor;

determining a set of values from the set of signals by the processor;

ranking each value in the set of values by the processor, wherein ranking each value in the set of values by the processor includes assigning a rank number to each value in the set of values by the processor;

generating a permutation of a digit string based at least in part on the rank number of each value in the set of values by the processor; and

identifying the gesture by the processor based at least in part on the permutation of the digit string.

2. The method of claim 1 wherein determining a set of values from the set of signals by the processor includes determining a set of features from the set of signals by the processor, wherein each feature in the set of features is selected from the group consisting of:

an average value of each signal in the set of signals, a mean value of each signal in the set of signals, a median value of each signal in the set of signals, a mode value of each signal in the set of signals, a maximum value of each signal in the set of signals, a minimum value of each signal in the set of signals, a standard deviation of each signal in the set of signals, and/or a root mean squared (“

RMS”

) value of each signal in the set of signals.

3. The method of claim 2 wherein determining a set of values from the set of signals by the processor further includes determining a respective difference between each feature in the set of features and each of the other features in the set of features by the processor, and wherein ranking each value in the set of values by the processor includes ranking each respective difference between each feature in the set of features and each of the other features in the set of features by the processor.

4. The method of claim 2 wherein determining a set of values from the set of signals by the processor further includes determining a respective correlation between each feature in the set of features and each of the other features in the set of features by the processor, and wherein ranking each value in the set of values by the processor includes ranking each respective correlation between each feature in the set of features and each of the other features in the set of features by the processor.

5. The method of claim 1, further comprising determining an orientation of the wearable EMG device on the user by the wearable EMG device.

6. The method of claim 5 wherein the wearable EMG device includes at least one inertial sensor communicatively coupled to the processor, and wherein determining an orientation of the wearable EMG device on the user by the wearable EMG device comprises:

in response to performance of a reference gesture by the user, detecting muscle activity of the user of the wearable EMG device by the set of EMG sensors, wherein the muscle activity corresponds to the reference gesture performed by the user;

in response to detecting muscle activity of the user by the set of EMG sensors, providing a set of reference EMG signals from the set of EMG sensors to the processor;

in response to performance of the reference gesture by the user, detecting motion of the wearable EMG device by the at least one inertial sensor, wherein the motion corresponds to the reference gesture performed by the user;

in response to detecting motion of the wearable EMG device by the at least one inertial sensor, providing at least one motion signal from the at least one inertial sensor to the processor; and

determining an orientation of the wearable EMG device based on the set of reference EMG signals and the at least one motion signal by the processor.

7. The method of claim 6 wherein the wearable EMG device includes a non-transitory processor-readable storage medium that stores at least one permutation condition, and wherein the method further comprises:

calibrating the at least one permutation condition based at least in part on the orientation of the wearable EMG device.

8. The method of claim 6, further comprising:

prompting the user to perform at least one defined reference gesture.

9. The method of claim 1 wherein the wearable EMG device includes at least one inertial sensor communicatively coupled to the processor, and wherein the method further comprises:

detecting motion of the wearable EMG device by the at least one inertial sensor, wherein the motion corresponds to the gesture performed by the user;

in response to detecting motion of the wearable EMG device, providing at least one signal from the at least one inertial sensor to the processor; and

processing the at least one signal from the at least one inertial sensor by the processor, and wherein identifying the gesture by the processor based at least in part on the permutation of the digit string includes identifying the gesture by the processor based at least in part on a result of the processing the at least one signal from the at least one inertial sensor by the processor.

10. The method of claim 1 wherein the wearable EMG device includes a non-transitory processor-readable storage medium communicatively coupled to the processor, wherein the non-transitory processor-readable storage medium stores processor-executable gesture identification instructions, and wherein identifying the gesture by the processor based at least in part on the permutation of the digit string includes executing the gesture identification instructions by the processor to cause the processor to identify the gesture based at least in part on the permutation of the digit string.

11. The method of claim 10 wherein the non-transitory processor-readable storage medium stores multiple sets of permutation conditions and a set of gesture identifications, each gesture identification being associated with a respective set of permutation conditions, and wherein identifying the gesture by the processor based at least in part on the permutation of the digit string includes applying a set of permutation conditions to the permutation of the digit string and returning the corresponding gesture identification if the permutation of the digit string satisfies the set of permutation conditions by the processor.

12. The method of claim 11 wherein applying a set of permutation conditions to the permutation of the digit string and returning the corresponding gesture identification if the permutation of the digit string satisfies the set of permutation conditions by the processor includes effecting a many-to-one mapping between the multiple sets of permutation conditions and the set of gesture identifications by the processor.

13. The method of claim 10 wherein the non-transitory processor-readable storage medium stores a many-to-one look-up table that maps respective sets of multiple permutations of the digit string to respective gesture identifications, and wherein identifying the gesture by the processor based at least in part on the permutation of the digit string includes locating the permutation of the digit string in the many-to-one look-up table and returning the corresponding gesture identification by the processor.

14. The method of claim 1 wherein providing a set of signals from the set of EMG sensors to the processor includes providing a respective signal from each respective EMG sensor in the set of EMG sensors.

15. The method of claim 14 wherein ranking each value in the set of values by the processor includes ranking the set of values based at least in part on a respective magnitude of each respective value in the set of values by the processor.

16. The method of claim 15 wherein ranking the set of values based at least in part on a respective magnitude of each respective value in the set of values by the processor includes assigning a respective rank number between 1 and N to each respective value in the set of values, wherein N is the number of values in the set of values and the respective rank number that is assigned to each respective value in the set of values is indicative of the magnitude of that value relative to the magnitudes of the other values in the set of values.

17. The method of claim 15 wherein ranking the set of values based at least in part on a respective magnitude of each respective value in the set of values by the processor includes:

comparing each respective value in the set of values to at least one threshold value by the processor; and

assigning a first rank number to each respective value in the set of values that exceeds the at least one threshold value by the processor and assigning a second rank number to each respective value in the set of values that does not exceed the at least one threshold value by the processor.

18. The method of claim 1 wherein determining a set of values from the set of signals by the processor comprises:

grouping the set of signals into a set of time-separated data windows; and

determining a respective value corresponding to each respective data window in the set of time-separated data windows.

19. A wearable electromyography (“

EMG”

) device comprising;

a set of EMG sensors responsive to muscle activity of a user of the wearable EMG device, wherein in response to muscle activity of the user the set of EMG sensors provide a set of signals;

a processor communicatively coupled to the set of EMG sensors;

a non-transitory processor-readable storage medium communicatively coupled to the processor, wherein the non-transitory processor-readable storage medium stores processor-executable gesture identification instructions that, when executed by the processor, cause the processor to;

determine a set of values from the set of signals;

assign a rank number to each value in the set of values;

generate a permutation of a digit string based at least in part on the rank number of each value in the set of values; and

identify a gesture that corresponds to the muscle activity based at least in part on the permutation of the digit string.

20. The wearable EMG device of claim 19, further comprising:

at least one communication terminal communicatively coupled to the processor, the at least one communication terminal to transmit information about the gesture to a receiving device.

21. The wearable EMG device of claim 20 wherein the at least one communication terminal includes at least one of a wireless communication terminal and/or a tethered connector port.

22. The wearable EMG device of claim 19, further comprising:

at least one inertial sensor communicatively coupled to the processor, the at least one inertial sensor responsive to motion effected by the user of the wearable EMG device, wherein in response to motion effected by the user the at least one inertial sensor provides at least one signal to the processor, and wherein the processor-executable gesture identification instructions that, when executed by the processor, cause the processor to identify a gesture that corresponds to the muscle activity based at least in part on the permutation of the digit string include processor-executable gesture identification instructions that, when executed by the processor, cause the processor to identify a gesture that corresponds to both the muscle activity and the motion effected by the user based at least in part on both the permutation of the digit string and a result of processing the at least one signal provided by the at least one inertial sensor by the processor.

23. The wearable EMG device of claim 22 wherein the at least one inertial sensor includes at least one accelerometer and/or at least one gyroscope.

24. The wearable EMG device of claim 19 wherein the processor is selected from the group consisting of:

a digital microprocessor, a digital microcontroller, a digital signal processor, a graphics processing unit, an application specific integrated circuit, a programmable gate array, and a programmable logic unit.

25. The wearable EMG device of claim 19, further comprising:

a set of communicative pathways to route signals provided by the set of EMG sensors to the processor, wherein each EMG sensor in the set of EMG sensors is communicatively coupled to the processor by at least one communicative pathway from the set of communicative pathways.

26. The wearable EMG device of claim 25, further comprising:

a set of pod structures that form physically coupled links of the wearable EMG device, wherein each pod structure in the set of pod structures is positioned adjacent and physically coupled to at least one other pod structure in the set of pod structures, and wherein the set of pod structures comprises at least two sensor pods and a processor pod, each of the at least two sensor pods comprising a respective EMG sensor from the set of EMG sensors and the processor pod comprising the processor.

28. The wearable EMG device of claim 26, further comprising:

at least one adaptive coupler, wherein each respective pod structure in the set of pod structures is adaptively physically coupled to at least one adjacent pod structure in the set of pod structures by at least one adaptive coupler.

27. The wearable EMG device of 26 wherein each pod structure in the set of pod structures is positioned adjacent and in between two other pod structures in the set of pod structures and physically coupled to the two other pod structures in the set of pod structures, and wherein the set of pod structures forms a perimeter of an annular configuration.