Systems and methods for improved speech recognition using neuromuscular information

• US 10,592,001 B2
• Filed: 05/08/2018
• Issued: 03/17/2020
• Est. Priority Date: 05/08/2018
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Berenzweig, Adam, Kaifosh, Patrick, Du, Alan Huan, Seely, Jeffrey Scott
• Primary Examiner(s): Shah, Antim G
• Application Number:
  US15/974,384
• Publication Number:
  US 20190348025A1
• Time in Patent Office: 679 Days
• Field of Search: None
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G10L 15/14 using statistical models, e.g. Hidden Markov Models [HMMs] G10L15/18 takes precedence
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology

Abstract

Systems and methods for using neuromuscular information to improve speech recognition. The system includes a plurality of neuromuscular sensors, arranged on one or more wearable devices, wherein the plurality of neuromuscular sensors is configured to continuously record a plurality of neuromuscular signals from a user, at least one storage device configured to store one or more trained statistical models, and at least one computer processor programmed to provide as an input to the one or more trained statistical models, the plurality of neuromuscular signals or signals derived from the plurality of neuromuscular signals, determine based, at least in part, on an output of the one or more trained statistical models, at least one instruction for modifying an operation of a speech recognizer, and provide the at least one instruction to the speech recognizer.

Claims

1. A computerized system for using neuromuscular information to improve speech recognition, the system comprising:

a plurality of neuromuscular sensors, arranged on one or more wearable devices, wherein the plurality of neuromuscular sensors is configured to continuously record a plurality of neuromuscular signals from a user;

at least one storage device configured to store one or more trained statistical models; and

at least one computer processor programmed to;

provide as an input to the one or more trained statistical models, the plurality of neuromuscular signals or signals derived from the plurality of neuromuscular signals;

determine based, at least in part, on an output of the one or more trained statistical models whether the user is holding a gesture, wherein determining whether the user is holding the gesture comprises evaluating the output at two or more time points to determine whether the user is performing the gesture at each of the two or more time points;

in response to determining that the user is holding the gesture;

generate a first instruction for modifying a state of a speech recognizer to a first one of an activated state, in which the speech recognizer is configured to perform speech recognition, and a deactivated state, in which the speech recognizer is configured to not perform speech recognition; and

provide the first instruction to the speech recognizer;

in response to determining that the user is not holding the gesture;

generate a second instruction for modifying the state of the speech recognizer to a second one of the activated state and the deactivated state; and

provide the second instruction to the speech recognizer.

2. The computerized system of claim 1, wherein evaluating the output at two or more time points to determine whether the user is performing the gesture at each of the two or more time points comprises:

determining based, at least in part, on the output of the one or more trained statistical models, a musculo-skeletal representation of the user; and

determining, based on the determined musculo-skeletal representation, whether the user is performing the gesture at each of the two or more time points.

3. The computerized system of claim 1, further comprising:

a communications interface configured to provide the first instruction and/or the second instruction from the at least one computer processor to the speech recognizer.

4. The computerized system of claim 1, wherein generating the first instruction comprises generating the first instruction for modifying the state of the speech recognizer from the deactivated state to the activated state, and wherein generating the second instruction comprises generating the second instruction for modifying the state of the speech recognizer from the activated state to the deactivated state.

5. The computerized system of claim 1, wherein generating the first instruction comprises generating the first instruction for modifying the state of the speech recognizer from the activated state to the deactivated state, and wherein generating the second instruction comprises generating the second instruction for modifying the state of the speech recognizer from the deactivated state to the activated state.

6. The computerized system of claim 2, wherein the musculoskeletal representation comprises information relating to a movement, a force, a pose, a gesture, and/or a muscle activation.

7. A computer implemented method for using neuromuscular information to improve speech recognition, the method comprising:

providing, using at least one computer processor, as an input to one or more trained statistical models, a plurality of neuromuscular signals or signals derived from the plurality of neuromuscular signals, wherein the plurality of neuromuscular signals is received from a plurality of neuromuscular sensors arranged on one or more wearable devices, wherein the plurality of neuromuscular sensors is configured to continuously record the plurality of neuromuscular signals from a user;

determining, using the at least one computer processor, based, at least in part, on an output of the one or more trained statistical models whether the user is holding a gesture, wherein determining whether the user is holding the gesture comprises evaluating the output at two or more time points to determine whether the user is performing the gesture at each of the two or more time points;

in response to determining that the user is holding the gesture;

generating, using the at least one computer processor, a first instruction for modifying a state of a speech recognizer to a first one of an activated state, in which the speech recognizer is configured to perform speech recognition, and a deactivated state, in which the speech recognizer is configured to not perform speech recognition; and

providing, using the at least one computer processor, the first instruction to the speech recognizer;

in response to determining that the user is not holding the gesture;

generating, using the at least one computer processor, a second instruction for modifying the state of the speech recognizer to a second one of the activated state and the deactivated state; and

providing, using the at least one computer processor, the second instruction to the speech recognizer.

8. The method of claim 7, wherein evaluating the output at two or more time points to determine whether the user is performing the gesture at each of the two or more time points comprises:

determining based, at least in part, on the output of the one or more trained statistical models, a musculo-skeletal representation of the user; and

determining, based on the determined musculo-skeletal representation, whether the user is performing the gesture at each of the two or more time points.

9. The method of claim 7, further comprising:

providing, using a communications interface, the first instruction and/or the second instruction from the at least one computer processor to the speech recognizer.

10. The method of claim 7, wherein generating the first instruction comprises generating the first instruction for modifying the state of the speech recognizer from the deactivated state to the activated state, and wherein generating the second instruction comprises generating the second instruction for modifying the state of the speech recognizer from the activated state to the deactivated state.

11. The method of claim 7, wherein generating the first instruction comprises generating the first instruction for modifying the state of the speech recognizer from the activated state to the deactivated state, and wherein generating the second instruction comprises generating the second instruction for modifying the state of the speech recognizer from the deactivated state to the activated state.

12. The method of claim 8, wherein the musculoskeletal representation comprises information relating to a movement, a force, a pose, a gesture, and/or a muscle activation.