Method of sonifying brain electrical activity

• US 10,136,862 B2
• Filed: 05/30/2013
• Issued: 11/27/2018
• Est. Priority Date: 05/30/2012
• Status: Active Grant

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford University)
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford University)
• Inventors: Chafe, Christopher D., Parvizi, Josef
• Primary Examiner(s): WEARE, MEREDITH H
• Application Number:
  US13/905,377
• Publication Number:
  US 20130324878A1
• Time in Patent Office: 2,007 Days
• CPC Class Codes: A61B 5/316 Modalities, i.e. specific diagnostic methods
• CPC Class Codes: A61B 5/369 Electroencephalography [EEG] devices for psychotechnics A61B5/16
• CPC Class Codes: A61B 5/372 Analysis of electroencephalograms
• CPC Class Codes: A61B 5/4094 Diagnosing or monitoring seizure diseases, e.g. epilepsy
• CPC Class Codes: A61B 5/7415 Sound rendering of measured values, e.g. by pitch or volume variation A61B5/741 takes precedence

Abstract

A digital processor system having one or more processors and memory obtains a time-domain signal representing brain activity, the time-domain signal having a time varying signal value. The system concurrently generates a set of acoustic parameters, including a plurality of time-varying acoustic parameters, where one or more of the plurality of time-varying acoustic parameters is modulated in accordance with at least the signal value of the time-domain signal. The system combines the concurrently generated set of acoustic parameters to produce a representation of an acoustic signal corresponding to the time-domain signal, where the acoustic signal, in audible form, manifests one or more audibly discernible variations across a plurality of stages of a brain activity event.

Claims

1. A method of sonifying brain signals using a digital processor system that comprises at least one processing unit, the method comprising:

filtering, with a front-end module coupled to the digital processor system, a continuous time-domain sensor signal representing brain activity of a subject obtained from at least one sensor in order to generate a filtered continuous time-domain sensor signal;

producing a continuous time-domain signal representing the brain activity of the subject from the filtered continuous time-domain sensor signal using the digital processor system, the continuous time-domain signal having a time varying signal value;

concurrently generating a plurality of continuously time-varying acoustic parameters from the continuous time-domain signal using the digital processor system, the plurality of continuously time-varying acoustic parameters including a vowel-control parameter, an intensity-control parameter, and a pitch-control parameter, wherein at least three of the parameters in the plurality of continuously time-varying acoustic parameters are continuously modulated in proportion with at least the continuous time varying signal value of the time-domain signal during concurrent generation;

driving, with the digital processor system, an audio synthesizer using the concurrently generated plurality of continuously time-varying acoustic parameters, including the vowel-control parameter, the intensity-control parameter, and the pitch-control parameter, to produce a representation of an acoustic signal corresponding to the brain activity of the subject represented by the continuous time-domain signal;

performing continuous audio playback of the representation of the acoustic signal using the digital processor system, wherein performing the continuous audio playback comprises audibly providing, with at least one speaker, the representation of the acoustic signal for diagnosis and/or treatment of a condition of the subject.

2. The method of claim 1, wherein the at least one sensor is affixed externally to the subject.

3. The method of claim 1, wherein producing the continuous time-domain signal representing brain activity of the subject comprises upsampling the filtered continuous time-domain sensor signal.

4. The method of claim 1, wherein the method is concurrently performed on a plurality of continuous time-domain signals representing brain activity at a plurality of distinct locations in a brain of the subject to produce representations of a plurality of acoustic signals, each corresponding to one of the plurality of distinct locations in the brain.

5. The method of claim 4, wherein the plurality of acoustic signals, each corresponding to one of the plurality of distinct locations in the brain, comprise audibly distinct acoustic signals.

6. The method of claim 4, wherein the plurality of acoustic signals, each corresponding to one of the plurality of distinct locations in the brain, comprise audibly distinct sonic identities.

7. The method of claim 1 wherein the audio synthesizer comprises a voice synthesizer, and wherein driving the audio synthesizer to produce the representation of the acoustic signal corresponding to the brain activity of the subject comprises utilizing the voice synthesizer to produce the representation of an acoustic signal.

8. The method of claim 7, wherein the voice synthesizer uses frequency modulation synthesis.

9. The method of claim 1, wherein concurrently generating the plurality of continuously time-varying acoustic parameters further comprises:

generating a frequency-control parameter in proportion with the time varying signal value of the continuous time-domain signal.

10. The method of claim 1, wherein concurrently generating the plurality of continuously time-varying acoustic parameters includes generating a frequency-control parameter by:

obtaining a base frequency or pitch;

modulating the base frequency or pitch in proportion with the signal value of the continuous time-domain signal, to produce a signal-dependent variation of the base frequency; and

generating the frequency-control parameter corresponding to the signal-dependent variation of the base frequency, and

wherein the audio synthesizer is further driven using the frequency-control parameter to produce the representation of the acoustic signal corresponding to the brain activity of the subject.

11. The method of claim 1, wherein concurrently generating the plurality of continuously time-varying acoustic parameters includes generating the vowel-control parameter by:

selecting a sequence of acoustic waveform patterns from a set of N acoustic waveform patterns, the set of N acoustic waveform patterns arranged in a predefined order, where N is an integer valuing at least 2;

modulating a rate of selecting the sequence of acoustic waveform patterns in proportion with the time varying signal value of the continuous time-domain signal, to produce a signal-dependent rate of variation of acoustic waveform patterns; and

generating the vowel-control parameter corresponding to the signal-dependent rate of variation of acoustic waveform patterns.

12. The method of claim 11, wherein the sequence of acoustic waveform patterns comprises a sequence of vowel waveform patterns.

13. The method of claim 1, wherein concurrently generating the plurality of continuously time-varying acoustic parameters includes:

computing the time-varying intensity value in proportion with the time varying signal value of the continuous time-domain signal; and

generating a time-varying control parameter corresponding to the computed time-varying intensity value.

14. The method of claim 1, wherein concurrently generating the plurality of continuously time-varying acoustic parameters includes:

selecting a base frequency; and

generating the time-varying pitch-control parameter in proportion with the selected base frequency and the time varying signal value of the continuous time-domain signal.

15. The method of claim 1, wherein concurrently generating the plurality of continuously time-varying acoustic parameters includes:

selecting a sonic identity in proportion with a spatial location in the brain of sensing the continuous time-domain signal; and

generating an identity-control parameter in proportion with the selected sonic identity,

wherein the audio synthesizer is further driven using the identity-control parameter to produce the representation of the acoustic signal corresponding to the brain activity of the subject.

16. The method of claim 1, wherein producing the continuous time-domain signal representing brain activity comprises low pass filtering the filtered continuous time-domain sensor signal.

17. The method of claim 1 wherein the continuous audio playback of the representation of the acoustic signal represents a first sound modulation in response to a normal brain state, wherein the continuous audio playback of the representation of the acoustic signal represents a second sound modulation in response to a physiologically harmful brain event, and wherein the first sound modulation and second sound modulation are audibly distinct.

18. A digital processor system for sonifying brain signals, the digital processor system comprising:

at least one processing unit;

an interface for obtaining a continuous time-domain sensor signal representing brain activity of a subject from at least one sensor;

an analog to digital converter directed to produce a digital continuous time domain signal representing brain activity, the digital continuous time domain signal having a time-varying value;

a filter directed to filter the digital continuous time-domain sensor signal to generate a filtered continuous time-domain sensor signal;

a non-transitory memory storing a plurality of modules which when executed by the at least one processing unit perform processing operations, the plurality of modules comprising;

a plurality of signal modulators directed to concurrently generate a plurality of continuously time-varying acoustic parameters from the continuous time-domain signal, the plurality of continuously time-varying acoustic parameters including a vowel-control parameter, an intensity-control parameter, and a pitch-control parameter, wherein at least three of the parameters in the plurality of continuously time-varying acoustic parameters are continuously modulated in proportion with at least the continuous time-varying value of the time-domain signal during concurrent generation;

a voice synthesizer directed to combine the concurrently generated plurality of continuously time-varying acoustic parameters, including the vowel-control parameter, the intensity-control parameter, and the pitch-control parameter, to produce a representation of an acoustic signal corresponding to the brain activity represented by the sensor continuous time-domain signal and for performing continuous audio playback of the representation of the acoustic signal; and

at least one speaker to audibly provide the representation of the acoustic signal for diagnosis and/or treatment of a condition of the subject.

19. The system of claim 18 wherein the continuous audio playback of the representation of the acoustic signal represents a first sound modulation in response to a normal brain state, wherein the continuous audio playback of the representation of the acoustic signal represents a second sound modulation in response to a physiologically harmful brain event, and wherein the first sound modulation and second sound modulation are audibly distinct.

20. A system for sonifying brain signals, comprising:

at least one speaker;

at least one processor;

a non-transitory memory; and

at least one program, wherein the at least one program is stored in the non-transitory memory and configured to be executed by the at least one processors, the at least one program including instructions for;

producing a continuous time-domain signal representing the brain activity of a subject from a filtered continuous time-domain sensor signal, the continuous time-domain signal having a time-varying value, the filtered continuous time-domain signal being generated by filtering a continuous time-domain sensor signal representing brain activity of the subject obtained from at least one sensor;

concurrently generating a plurality of continuously time-varying acoustic parameters from the continuous time-domain signal, the plurality of continuously time-varying acoustic parameters including a vowel-control parameter, an intensity-control parameter, and a pitch-control parameter, wherein at least three parameters in the plurality of continuously time-varying acoustic parameters are continuously modulated in proportion with at least the time-varying value of the time-domain signal during concurrent generation;

driving an audio synthesizer using the concurrently generated plurality of continuously time-varying acoustic parameters, including the vowel-control parameter, the intensity-control parameter, and the pitch-control parameter, to produce a representation of an acoustic signal corresponding to the brain activity represented by the continuous time-domain signal;

performing continuous audio playback of the representation of the acoustic signal, wherein performing the continuous audio playback comprises audibly providing, with at least one speaker, the representation of the acoustic signal for diagnosis and/or treatment of a condition of the living subject.

21. The system of claim 20, wherein the at least one sensor is adapted to be affixed externally to a person.

22. The system of claim 20 wherein the instructions for producing a continuous time-domain signal representing brain activity comprise upsampling the filtered continuous time-domain sensor.

23. The system of claim 20, wherein performing continuous audio playback of the representation of the acoustic signal further comprises vocalization of the representation of the acoustic signal.

24. The system of claim 20 wherein the instructions for producing a continuous time-domain signal representing brain activity comprise instructions for low pass filtering the filtered continuous time-domain sensor signal.

25. The system of claim 20 wherein the continuous audio playback of the representation of the acoustic signal represents a first sound modulation in response to a normal brain state, wherein the continuous audio playback of the representation of the acoustic signal represents a second sound modulation in response to a physiologically harmful brain event, and wherein the first sound modulation and second sound modulation are audibly distinct.

26. A method of sonifying brain signals using a digital processor system that comprises at least one processing unit, the method comprising:

filtering, with a front-end module coupled to the digital processor system, a continuous time-domain sensor signal representing brain activity of a subject obtained from at least one sensor in order to generate a filtered continuous time-domain sensor signal;

producing a continuous time-domain signal representing the brain activity of the subject obtained from the at least one sensor from the filtered continuous time-domain sensor signal using the digital processor system, the continuous time-domain signal having a time varying signal value;

concurrently generating a plurality of continuously time-varying acoustic parameters from the continuous time-domain signal using the digital processor system, the plurality of continuously time-varying acoustic parameters including a vowel-control parameter, an intensity-control parameter, and a pitch-control parameter, wherein at least three of the parameters in the plurality of continuously time-varying acoustic parameters are continuously modulated in proportion with at least the continuous time varying signal value of the time-domain signal during concurrent generation;

driving, with the digital processor system, a vocal synthesizer using the concurrently generated plurality of continuously time-varying acoustic parameters, including the vowel-control parameter, the intensity-control parameter, and the pitch-control parameter, to produce a vocalized representation of an acoustic signal corresponding to the brain activity of the subject represented by the continuous time-domain signal; and

performing continuous audio playback of the vocalized representation of the acoustic signal using the digital processor system, wherein performing the continuous audio playback comprises audibly providing, with at least one speaker, the vocalized representation of the acoustic signal for diagnosis and/or treatment of a condition of the living subject,

wherein the vocalized representation of the acoustic signal manifests at least one audibly discernible variation across a plurality of stages of a brain activity event according to the continuously time-varying plurality of acoustic parameters.

27. The method of claim 26 wherein the continuous audio playback of the representation of the acoustic signal represents a first sound modulation in response to a normal brain state, wherein the continuous audio playback of the representation of the acoustic signal represents a second sound modulation in response to a physiologically harmful brain event, and wherein the first sound modulation and second sound modulation are audibly distinct.