Methods of decoding speech from the brain and systems for practicing the same

• US 10,438,603 B2
• Filed: 02/26/2018
• Issued: 10/08/2019
• Est. Priority Date: 02/19/2013
• Status: Active Grant

• Current Assignee: Regents of the University of California (University of California)
• Original Assignee: Regents of the University of California (University of California)
• Inventors: Chang, Edward F., Bouchard, Kristofer E.
• Primary Examiner(s): Le, Thuykhanh
• Application Number:
  US15/905,378
• Publication Number:
  US 20180254049A1
• Time in Patent Office: 589 Days
• Field of Search: None
• CPC Class Codes: A61B 5/24 Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
• CPC Class Codes: A61B 5/4076 Diagnosing or monitoring particular conditions of the nervous system A61B5/4821, A61B5/4824 take precedence
• CPC Class Codes: A61B 5/4094 Diagnosing or monitoring seizure diseases, e.g. epilepsy
• CPC Class Codes: A61B 5/4803 Speech analysis specially adapted for diagnostic purposes
• CPC Class Codes: A61B 5/7203 for noise prevention, reduction or removal
• CPC Class Codes: A61B 5/7257 using Fourier transforms
• CPC Class Codes: A61B 5/726 using Wavelet transforms
• CPC Class Codes: G09B 21/00 Teaching, or communicating with, the blind, deaf or mute audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 21/00 Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility G10L19/00 takes precedence
• CPC Class Codes: G10L 25/03 characterised by the type of extracted parameters
• CPC Class Codes: G10L 25/06 the extracted parameters being correlation coefficients

Abstract

Provided are methods of decoding speech from the brain of a subject. The methods include detecting speech production signals from electrodes operably coupled to the speech motor cortex of a subject while the subject produces or imagines producing a speech sound. The methods further include deriving a speech production signal pattern from the detected speech production signals, and correlating the speech production signal pattern with a reference speech production signal pattern to decode speech from the brain of the subject. Speech communication systems and devices for practicing the subject methods are also provided.

Claims

1. A speech communication system, comprising:

a neurosensor, comprising;

at least three electrodes that detect speech production signals from the ventral sensorimotor cortex of a subject when operably coupled to the ventral sensorimotor cortex of the subject while the subject produces or imagines producing a speech sound, wherein the speech sound is selected from the group consisting of;

a phoneme, formant acoustics of a vowel, a diphone, a triphone, a consonant-vowel transition, a syllable, a word, a phrase, a sentence, a spectrotemporal feature, an acoustic-phonetic feature, and combinations thereof, and a wireless transmitter that transmits the detected speech production signals; and

a receiver unit, comprising;

a wireless receiver in communication with the wireless transmitter that receives the detected speech production signals;

a speech generator;

a processor; and

a memory comprising instructions which, when executed by the processor, derive a speech production signal pattern from the detected speech production signals from the ventral sensorimotor cortex of the subject, correlates the speech production signal pattern with a reference speech production signal pattern to decode the speech sound, and communicates the speech sound using the speech generator.

2. The system of claim 1, wherein the wireless transmitter is selected from the group consisting of:

a WiFi-based transmitter, a Bluetooth-based transmitter, and a radio frequency-based transmitter.

3. The system of claim 1, wherein the speech generator comprises a speaker that produces the speech sound in audible form.

4. The system of claim 1, wherein the speech generator comprises a display that displays the speech sound in text format.

5. The system of claim 1, wherein the speech generator produces the speech sound in audible form and displays the speech sound in text format.

6. The system of claim 1, wherein the receiver unit is selected from the group consisting of:

a desktop computer, a laptop computer, a tablet computer, a smartphone, and a teletypewriter (TTY) device.

7. The system of claim 1, wherein the at least three electrodes comprises at least 10 electrodes.

8. The system of claim 7, wherein the at least three electrodes comprises at least 20 electrodes.

9. The system of claim 1, wherein the speech production signals are detected from a region of the ventral sensorimotor cortex selected from:

the pre-central gyrus, the post-central gyms, the guenon, and combinations thereof.

10. The system of claim 1, wherein the at least three electrodes are operably coupled to the speech motor cortex by implantation on the surface of the speech motor cortex.

11. The system of claim 10, wherein the at least three electrodes comprises an array of electrocorticography (ECoG) electrodes.

12. The system of claim 1, wherein the at least three electrodes are operably coupled to the speech motor cortex by insertion of the electrodes into the speech motor cortex.

13. The system of claim 1, wherein the detected speech production signals are local field potentials generated from the speech motor cortex.

14. The system of claim 1, wherein the speech production signal pattern is derived from a time-course of speech production signals detected beginning prior to an onset of the speech sound.

15. The system of claim 1, wherein the speech production signal pattern is derived from a time-course of speech production signals detected beginning from 250 milliseconds or more prior to an onset of the speech sound.

16. The system of claim 1, wherein the speech sound is a syllable.

17. The system of claim 1, wherein the memory comprises instructions for performing time-frequency analysis of speech production signals detected from each of the at least three electrodes.

18. The system of claim 17, wherein the memory comprises instructions for performing a time-frequency analysis method selected from the group consisting of:

Fast Fourier Transform (FFT), wavelet transform, Hilbert transform, and bandpass filtering.

19. The system of claim 1, wherein the at least three electrodes are distributed across the speech motor cortex such that speech production signals corresponding to at least three speech articulators are detected.

20. The system of claim 19, wherein the at least three speech articulators are selected from the group consisting of:

the larynx, the lips, the jaw, and the tongue.