Method and apparatus for analyzing capacitive EMG and IMU sensor signals for gesture control

• US 9,299,248 B2
• Filed: 02/21/2014
• Issued: 03/29/2016
• Est. Priority Date: 02/22/2013
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Thalmic Labs Inc.
• Inventors: Lake, Stephen, Bailey, Matthew, Grant, Aaron
• Primary Examiner(s): Pervan, Michael
• Application Number:
  US14/186,878
• Publication Number:
  US 20140240223A1
• Time in Patent Office: 767 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G08C 17/02 using a radio link
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device

Abstract

There is disclosed a muscle interface device for use with controllable connected devices. In an embodiment, the muscle interface device comprises a sensor worn on the forearm of a user, and the sensor is adapted to recognize a plurality of gestures made by a user to interact with a controllable connected device. The muscle interface device utilizes a plurality of sensors, including one or more of capacitive EMG sensors and an IMU sensor, to detect gestures made by a user. Other types of sensors including MMG sensors may also be used. The detected user gestures from the sensors are processed into a control signal for allowing the user to interact with content displayed on the controllable connected device.

Claims

1. An apparatus to detect and analyze signals for gesture control, comprising:

a band to be worn by a user;

a plurality of electromyography (EMG) sensors positioned radially around a circumference of the band, each EMG sensor in the plurality of EMG sensors to detect -electrical signals produced by muscle activity in response to the user performing a gesture;

an inertial sensor carried by the band, the inertial sensor to detect motion in response to the user performing the gesture; and

a processor carried by the band, the processor communicatively coupled to receive EMG signals from the plurality of EMG sensors and inertial sensor signals from the inertial sensor, and which in use;

analyzes the EMG signals and recognizes the gesture performed by the user based on the analysis of the EMG signals, and

analyzes the inertial sensor signals and determines a motion aspect of the gesture based on the analysis of the inertial sensor signals.

2. The apparatus of claim 1 wherein the processor in use further determines the position and orientation of the apparatus based on the analysis of the inertial sensor signals.

3. The apparatus of claim 1 wherein the processor in use analyzes the EMG signals via one or more of a Hidden Markov Model, Long-Short Term Neural Net, or another machine intelligence method to recognize the gesture performed by the user.

4. The apparatus of claim 1 wherein the processor in use further detects a static gesture or a dynamic gesture in dependence upon analyzing the inertial sensor signals.

5. The apparatus of claim 1 wherein the processor in use further determines whether the gesture is a long or short duration gesture by measuring a length of an activated segment of the EMG signals.

6. The apparatus of claim 1 wherein the processor in use further records one or more user defined gestures.

7. The apparatus of claim 1, further comprising:

at least one mechanomyography (MMG) sensor carried by the band to detect vibrations produced by muscle activity in response to the user performing the gesture.

8. The apparatus of claim 1, further comprising a haptic feedback module to provide haptic feedback to the user.

9. The apparatus of claim 8 wherein haptic feedback provided by the haptic feedback module provides confirmation of recognition of the gesture.

10. The apparatus of claim 8 wherein the haptic feedback module comprises a vibratory motor.

11. The apparatus of claim 1, further comprising a wireless transceiver module carried by the band and communicatively coupled to the processor, the wireless transceiver module to wirelessly transmit at least one signal in response to the processor recognizing the gesture performed by the user.

12. The apparatus of claim 1 wherein the on-board inertial sensor includes at least one sensor selected from the group consisting of:

an accelerometer and an inertial measurement unit.

13. The apparatus of claim 1, further comprising an on-board non-transitory computer readable storage memory carried by the band and communicatively coupled to the on-board processor, the on-board non-transitory computer-readable storage memory to store the EMG signals and the inertial sensor signals.

14. A method of operating an apparatus to detect and analyze signals for gesture control, the apparatus including a band worn by a user, a plurality of on-board electromyography (EMG) sensors positioned radially around a circumference of the band, an on-board inertial sensor carried by the band, and an on-board processor carried by the band, the on-board processor communicatively coupled to the plurality of on-board EMG sensors and to the on-board inertial sensor, wherein the method comprises:

receiving, by the on-board processor, at least one electrical signal from at least one on-board EMG sensor in the plurality of on-board EMG sensors, the at least one electrical signal from at least one on-board EMG sensor indicative of muscle activity in response to the user performing a gesture;

receiving, by the on-board processor, at least one electrical signal from the on-board inertial sensor, the at least one electrical signal from the on-board inertial sensor indicative of motion in response to the user performing the gesture;

analyzing, by the on-board processor, the at least one electrical signal received from the at least one on-board EMG sensor

recognizing the gesture performed by the user by the on-board processor based on analyzing the at least one electrical signal received from the at least one on-board EMG sensor;

analyzing, by the on-board processor, the at least one electrical signal received from the on-board inertial sensor; and

determining a motion aspect of the gesture based on analyzing the at least one electrical signal received from the on-board inertial sensor.

15. The method of claim 14, further comprising:

determining, by the on-board processor, the relative velocity and orientation of the apparatus based on the at least one electrical signal received from the on-board inertial sensor.

16. The method of claim 14 wherein analyzing, by the on-board processor, the at least on electrical signal received from the at least one on-board EMG sensor includes utilizing, by the on-board processor, one or more of a Hidden Markov Model, Long-Short Term Neural Net, or other machine intelligence method to analyze the at least one electrical signal received from the at least one on-board EMG sensor.

17. The method of claim 14, further comprising detecting, by the on-board processor, a static gesture or a dynamic gesture in dependence upon analyzing the at least one electrical signal received from the on-board inertial sensor.

18. The method of claim 14, further comprising measuring, by the on-board processor, a length of an activated segment of an RMS value of the at least one electrical signal received from the at least one on-board EMG sensor in the plurality of on-board EMG sensors.

19. The method of claim 14, further comprising recording, by the on-board processor, one or more user defined gestures.

20. The method of claim 14 wherein the apparatus includes an on-board haptic feedback module carried by the band, the method further comprising providing a haptic feedback to the user by the on-board haptic feedback module.

21. The method of claim 20 wherein the haptic feedback is provided to the user by the on-board haptic feedback module in response to the processor recognizing the gesture performed by the user.

22. The method of claim 14 wherein the apparatus includes an on-board wireless transceiver module, the method further comprising wirelessly transmitting a signal by the on-board wireless transceiver module in response to the on-board processor recognizing the gesture performed by the user.