Methods and apparatus for inferring user intent based on neuromuscular signals

• US 10,409,371 B2
• Filed: 07/25/2017
• Issued: 09/10/2019
• Est. Priority Date: 07/25/2016
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: CTRL-Labs Corp. (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Kaifosh, Patrick, Machado, Timothy, Reardon, Thomas, Schomburg, Erik
• Primary Examiner(s): Dicke, Chad M
• Application Number:
  US15/659,018
• Publication Number:
  US 20180024634A1
• Time in Patent Office: 777 Days
• Field of Search: None
• CPC Class Codes: A61B 5/389 Electromyography [EMG]
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: A61B 5/7282 Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G16H 50/70 for mining of medical data, e.g. analysing previous cases of other patients

Abstract

Methods and system for predicting the onset of a motor action using neuromuscular signals. The system comprises a plurality of sensors configured to continuously record a plurality of neuromuscular signals from a user and at least one computer processor programmed to provide as input to a trained statistical model, the plurality of neuromuscular signals or information based on the plurality of neuromuscular signals, predict, based on an output of the trained statistical model, whether an onset of a motor action will occur within a threshold amount of time; and send a control signal to at least one device based, at least in part, on the output probability, wherein the control signal is sent to the at least one device prior to completion of the motor action by the user.

Claims

1. A control system, comprising:

a plurality of sensors configured to continuously record a plurality of neuromuscular signals from a user; and

at least one computer processor programmed to;

provide as input to a trained statistical model, the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals, wherein the trained statistical model was trained based, at least in part, on neuromuscular data recorded during at least one previous performance of a motor action by one or more users and result data indicating an outcome of the motor action performed by the one or more users, wherein the trained statistical model is configured to predict the onset of the motor action prior to completion of the motor action;

predict, based on an output of the trained statistical model, whether an onset of the motor action will occur within a threshold amount of time; and

send a control signal to at least one device based, at least in part, on the prediction, wherein the control signal is sent to the at least one device prior to completion of the motor action by the user to control an operation of the at least one device.

2. The control system of claim 1, wherein the output of the trained statistical model comprises a probability that the onset of the motor action will occur within the threshold amount of time, and

wherein predicting that an onset of the motor action will occur within a threshold amount of time comprises determining whether the probability is greater than an operating threshold.

3. The control system of claim 2, wherein the at least one computer processor is further programmed to determine the operating threshold based, at least in part, on an analysis of data obtained from neuromuscular signals recorded by at least some of the plurality of sensors.

4. The control system of claim 3, wherein determining the operating threshold comprises:

determining a mean anticipation time based on the data; and

determining the operating threshold based, at least in part, on the mean anticipation time.

5. The control system of claim 2, wherein the at least one computer processor is further programmed to receive at least one performance characteristic for a task to be performed, and wherein the operating threshold is determined based, at least in part, on the received at least one performance characteristic.

6. The control system of claim 5, wherein the at least one performance characteristic includes a performance characteristic selected from the group consisting of a false positive rate, a false negative rate, a true positive rate, a true negative rate, a positive predictive value, a negative predictive value, and a mean anticipation time.

7. The control system of claim 1, wherein the output of the trained statistical model comprises a set of probabilities, each probability in the set indicating a probability that the onset of a respective motor action of a plurality of motor actions will occur within the threshold amount of time, and

wherein predicting that an onset of the motor action will occur within a threshold amount of time comprises determining whether any of the probabilities in the set of probabilities is greater than an operating threshold stored by the system.

8. The control system of claim 1, wherein the output of the trained statistical model comprises a probability that the onset of a sequence of motor actions will occur within the threshold amount of time, and

wherein predicting that an onset of the motor action will occur within a threshold amount of time comprises determining whether the probability is greater than an operating threshold.

9. The control system of claim 1, wherein the at least one computer processor is further programmed to retrain the trained statistical model based, at least in part, on at least some of the plurality of neuromuscular signals.

10. The control system of claim 1, wherein the trained statistical model comprises a recurrent neural network.

11. The control system of claim 1, wherein the at least one computer processor is further programmed to:

determine a set of features based on the plurality of neuromuscular signals; and

provide as input to a trained statistical model, the set of features as information based on the plurality of neuromuscular signals,

wherein predicting, based on an output of the trained statistical model, that an onset of the motor action will occur within a threshold amount of time comprises classifying the set of features using the trained statistical model.

12. The control system of claim 1, wherein the plurality of sensors are arranged on a wearable device configured to be worn on or around a body part of the user.

13. The control system of claim 12, wherein the wearable device comprises a flexible or elastic band configured to be worn around the body part of the user.

14. The control system of claim 13, wherein the wearable device comprises an armband configured to be worn around an arm of the user.

15. The control system of claim 1, further comprising hardware circuitry and/or software configured to process the neuromuscular signals prior to being provided as input to the trained statistical model, wherein processing the neuromuscular signals comprises processing selected from the group consisting of amplification, filtering, and rectification.

16. The control system of claim 1, wherein sending the control signal to the at least one device prior to completion of the motor action by the user comprises sending the control signal at least 25 ms prior to the completion of the motor action by the user.

17. The control system of claim 1, wherein the at least one device comprises at least one device selected from the group consisting of a video game system, a consumer electronics device, a manned vehicle, an unmanned vehicle, a robot, and a weapon.

18. The control system of claim 1, wherein the plurality of sensors comprises sensors selected from the group consisting of electromyography (EMG) sensors, mechanomyography (MMG) sensors, and sonomyography (SMG) sensors.

19. A method of predicting an onset of a motor action, the method comprising

providing, as input to a trained statistical model, a plurality of neuromuscular signals recorded from a plurality of sensors arranged on or around a part of a user'"'"'s body, wherein the trained statistical model was trained based, at least in part, on neuromuscular data recorded during at least one previous performance of a motor action by one or more users and result data indicating an outcome of the motor action performed by the one or more users, wherein the trained statistical model is configured to predict the onset of the motor action prior to completion of the motor action;

predicting, using at least one computer processor, whether an onset of the motor action will occur within a threshold amount of time; and

sending a control signal to at least one device based, at least in part, on the prediction, wherein the control signal is sent to the at least one device prior to completion of the motor action by the user to control an operation of the at least one device.

20. A computer system for training a statistical model to predict an onset of a motor action based, at least in part, on neuromuscular signal data, the computer system comprising:

an input interface configured to;

receive the neuromuscular signal data recorded during performance of a motor action performed by one or more users;

receive result data indicating an outcome of the motor action performed by the one or more users; and

at least one storage device configured to store a plurality of instructions that, when executed by at least one computer processor perform a method of;

generating training data based, at least on part, on the received neuromuscular signal data and the received result data;

training the statistical model using at least some of the generated training data to output a trained statistical model;

determining an operating threshold used to interpret output of the trained statistical model; and

storing, by the at least one storage device, the trained statistical model and the operating threshold, wherein the trained statistical model is configured to predict the onset of the motor action prior to completion of the motor action by a user.