Methods of decoding speech from brain activity data and devices for practicing the same

• US 10,264,990 B2
• Filed: 10/25/2013
• Issued: 04/23/2019
• Est. Priority Date: 10/26/2012
• Status: Active Grant

• Current Assignee: Regents of the University of California (University of California)
• Original Assignee: Regents of the University of California (University of California)
• Inventors: Pasley, Brian, Knight, Robert T.
• Primary Examiner(s): Le, Thuykhanh
• Application Number:
  US14/438,496
• Publication Number:
  US 20150297106A1
• Time in Patent Office: 2,006 Days
• Field of Search: None
• CPC Class Codes: A61B 5/0006 ECG or EEG signals
• CPC Class Codes: A61B 5/291 for electroencephalography [EEG]
• CPC Class Codes: A61B 5/316 Modalities, i.e. specific diagnostic methods
• CPC Class Codes: A61B 5/369 Electroencephalography [EEG] devices for psychotechnics A61B5/16
• CPC Class Codes: A61B 5/374 Detecting the frequency distribution of signals, e.g. detecting delta, theta, alpha, beta or gamma waves
• CPC Class Codes: A61B 5/38 Acoustic or auditory stimuli
• CPC Class Codes: A61B 5/4803 Speech analysis specially adapted for diagnostic purposes
• CPC Class Codes: A61B 5/6814 Head
• CPC Class Codes: A61B 5/6868 Brain
• CPC Class Codes: A61B 5/6898 Portable consumer electronic devices, e.g. music players, telephones, tablet computers
• CPC Class Codes: A61B 5/741 using synthesised speech
• CPC Class Codes: A61B 5/742 using visual displays A61B5/339 takes precedence
• CPC Class Codes: G10L 25/48 specially adapted for particular use

Abstract

The present disclosure provides methods of decoding speech from brain activity data. Aspects of the methods include receiving brain speech activity data from a subject, and processing the brain speech activity data to output speech feature data. Also provided are devices and systems for practicing the subject methods.

Claims

1. A method of decoding brain speech activity of a subject into speech feature data, the method comprising:

(a) receiving brain speech activity data from two or more of;

the anterior superior temporal gyrus (STG), medial superior temporal gyrus and posterior superior temporal gyrus of a subject into a decoder device;

(b) processing the received brain speech activity data with a decoder module present in the decoder device, wherein the decoder module is configured to convert the received brain speech activity data by predictive statistical modeling into speech feature data; and

(c) outputting the speech feature data to an output device,

wherein the speech feature data is a representation of speech selected from the group consisting of auditory spectrograms, modulation representations, phonemes, phones, semantics, articulatory representations, nonlinear auditory representations, and motor representations of speech processing.

2. The method according to claim 1, wherein the brain speech activity data is obtained from an electrode device.

3. The method according to claim 2, wherein the electrode device comprises a plurality of electrodes.

4. The method according to claim 3, wherein the electrode device is implanted.

5. The method according to claim 4, wherein the brain speech activity data is electrocorticography (ECoG) data.

6. The method according to claim 3, wherein the electrode device is not implanted.

7. The method according to claim 6, wherein the brain speech activity data is electroencephalography (EEG) data.

8. The method according to claim 1, wherein the decoder module employs at least one of a linear spectrogram model and a non-linear modulation model.

9. The method according to claim 8, wherein the decoder module is configured to select between a linear spectrogram model and a non-linear modulation model.

10. The method according to claim 1, wherein the speech feature data is configured to be converted by an output device into readable text.

11. The method according to claim 1, wherein the speech feature data is configured to be converted by an output device into audible sound.

12. The method according to claim 11, wherein the audible sound is a human recognizable word.

13. The method according to claim 11, wherein the audible sound is a human recognizable collection of words.

14. The method according to claim 13, wherein the human recognizable collection comprises phrases or sentences.

15. The method according to claim 1, wherein the brain speech activity data is a result of heard speech.

16. The method according to claim 1, wherein the brain speech activity data is the result of imagined speech.

17. The method according to claim 1, wherein predictive statistical modeling comprises linear spectrogram modeling, non-linear modulation modeling or a combination thereof.

18. The method according to claim 17, wherein the non-linear modulation modeling comprises generating a modulation model by employing a plurality of modulation selective filters that analyze a two-dimensional spectrogram and extract modulation energy at different temporal rates and spectral scales.

19. The method according to claim 17, wherein linear spectrogram modeling further comprises mapping a stimulus spectrogram.

20. The method according to claim 19, wherein the stimulus spectrogram exhibits frequency selectivity.

21. The method according to claim 20, wherein frequency selectivity comprises neural response amplitudes based on stimulus frequency.

22. The method according to claim 1, wherein the method comprises receiving brain speech activity data from across the superior temporal gyms (STG) of the subject into the decoder device.

23. The method according to claim 1, further comprising receiving brain speech activity data from the middle temporal gyrus.

24. A system for decoding brain speech activity of a subject into speech feature data, the system comprising:

a decoder device comprising;

a data input component configured to receive brain speech activity data from two or more of;

the anterior superior temporal gyrus (STG)), medial superior temporal gyrus and posterior superior temporal gyrus of a subject;

a decoder module comprising;

a processor; and

a machine-readable medium encoding instructions operable to;

cause the processor to convert received brain speech activity data by predictive statistical modeling into speech feature data; and

output the speech feature data to an output component; and

a data output component configured to output speech feature data from the decoder module,

wherein the speech feature data is a representation of speech selected from the group consisting of auditory spectrograms, modulation representations, phonemes, phones, semantics, articulatory representations, nonlinear auditory representations, and motor representations of speech processing.

25. The system according to claim 24, wherein the data input component is configured to receive brain speech activity data from across the superior temporal gyrus (STG) of the subject into the decoder device.

26. The system according to claim 24, wherein the data input component is further configured to receive brain speech activity data from the middle temporal gyrus.