Spatial audio array processing system and method

• US 10,735,887 B1
• Filed: 05/20/2020
• Issued: 08/04/2020
• Est. Priority Date: 09/19/2019
• Status: Active Grant

• Current Assignee: Wave Sciences, LLC
• Original Assignee: Wave Sciences, LLC
• Inventors: McElveen, James Keith, Nordlund, Jr., Gregory S., Krasny, Leonid
• Primary Examiner(s): Patel, Yogeshkumar
• Application Number:
  US16/879,470
• Time in Patent Office: 76 Days
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 21/0272 Voice signal separating
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 2201/401 2D or 3D arrays of transducers
• CPC Class Codes: H04R 2201/403 Linear arrays of transducers
• CPC Class Codes: H04R 2201/405 Non-uniform arrays of transducers or a plurality of uniform arrays with different transducer spacing
• CPC Class Codes: H04R 2430/03 Synergistic effects of band splitting and sub-band processing
• CPC Class Codes: H04R 2430/20 Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 2430/25 Array processing for suppression of unwanted side-lobes in directivity characteristics, e.g. a blocking matrix
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04S 7/307 Frequency adjustment, e.g. tone control H04S7/301 takes precedence

Abstract

A spatial audio processing system operable to enable audio signals to be spatially extracted from, or transmitted to, discrete locations within an acoustic space. Embodiments of the present disclosure enable an array of transducers being installed in an acoustic space to combine their signals via inverting physical and environmental models that are measured, learned, tracked, calculated, or estimated. The models may be combined with a whitening filter to establish a cooperative or non-cooperative information-bearing channel between the array and one or more discrete, targeted physical locations in the acoustic space by applying the inverted models with whitening filter to the received or transmitted acoustical signals. The spatial audio processing system may utilize a model of the combination of direct and indirect reflections in the acoustic space to receive or transmit acoustic information, regardless of ambient noise levels, reverberation, and positioning of physical interferers.

Claims

1. A method for spatial audio processing comprising:

receiving, with an audio processor, an audio input comprising audio signals captured by a plurality of transducers within an acoustic environment;

converting, with the audio processor, the audio input from a time domain to a frequency domain according to at least one transform function;

determining, with the audio processor, at least one acoustic propagation model for at least one source location within the acoustic environment according to a normalized cross power spectral density calculation, the at least one acoustic propagation model comprising at least one Green'"'"'s Function estimation;

processing, with the audio processor, the audio input according to the at least one acoustic propagation model to spatially filter at least one target audio signal from one or more non-target audio signals, wherein the target audio signal corresponds to the at least one source location; and

applying, with the audio processor, a whitening filter to a spatially filtered target audio signal to derive at least one separated audio output signal, wherein the whitening filter is applied concurrently or concomitantly with the at least one acoustic propagation model.

2. The method of claim 1 wherein the at least one transform function is selected from the group consisting of Fourier transform, Fast Fourier transform, Short Time Fourier transform and modulated complex lapped transform.

3. The method of claim 2 further comprising performing, with the audio processor, at least one inverse transform function to convert the at least one separated audio output signal from a frequency domain to a time domain.

4. The method of claim 3 further comprising rendering or outputting, with the audio processor, a digital audio output comprising the at least one separated audio output signal.

5. The method of claim 1 further comprising determining, with the audio processor, two or more acoustic propagation models associated with two or more source locations within the acoustic environment and storing each acoustic propagation model in the two or more acoustic propagation models in a computer-readable memory device.

6. The method of claim 5 further comprising creating, with the audio processor, a separate whitening filter for each acoustic propagation model in the two or more acoustic propagation models.

7. The method of claim 1 further comprising applying, with the audio processor, a spectral subtraction noise reduction filter to the at least one separated audio output signal.

8. The method of claim 1 further comprising applying, with the audio processor, a phase correction filter to the spatially filtered target audio signal.

9. The method of claim 5 further comprising receiving, in real-time, at least one sensor input comprising sound source localization data for at least one sound source.

10. The method of claim 9 further comprising determining, in real-time, the at least one source location according to the sound source localization data.

11. The method of claim 10 wherein the at least one sensor input comprises a camera or a motion sensor.

12. A spatial audio processing system, comprising:

a plurality of acoustic transducers being located within an acoustic environment and operably engaged to comprise an array, the plurality of acoustic transducers being configured to capture acoustic audio signals from sound sources within the acoustic environment;

a computing device comprising an audio processing module communicably engaged with the plurality of acoustic transducers to receive an audio input comprising the acoustic audio signals, the audio processing module comprising at least one processor and a non-transitory computer readable medium having instructions stored thereon that, when executed, cause the at least one processor to perform one or more spatial audio processing operations, the one or more spatial audio processing operations comprising;

converting the audio input from a time domain to a frequency domain according to at least one transform function;

determining at least one acoustic propagation model for at least one source location within the acoustic environment according to a normalized cross power spectral density calculation, the at least one acoustic propagation model comprising at least one Green'"'"'s Function estimation;

processing the audio input according to the at least one acoustic propagation model to spatially filter at least one target audio signal from one or more non-target audio signals, wherein the at least one target audio signal corresponds to the at least one source location; and

applying a whitening filter to a spatially filtered target audio signal to derive at least one separated audio output signal, wherein the whitening filter is applied concurrently or concomitantly with the at least one acoustic propagation model.

13. The system of claim 12 wherein the at least one transform function is selected from the group consisting of Fourier transform, Fast Fourier transform, Short Time Fourier transform and modulated complex lapped transform.

14. The system of claim 12 wherein the one or more spatial audio processing operations further comprise applying a spectral subtraction noise reduction filter to the at least one separated audio output signal.

15. The system of claim 12 wherein the one or more spatial audio processing operations further comprise applying a phase correction filter to the spatially filtered target audio signal.

16. The system of claim 13 wherein the one or more spatial audio processing operations further comprise applying at least one inverse transform function to convert the at least one separated audio output signal from a frequency domain to a time domain.

17. The system of claim 12 further comprising at least one sensor communicably engaged with the computing device to provide, in real-time, one or more sensor inputs comprising sound source localization data for at least one sound source.

18. The system of claim 17 wherein the computing device is configured to process the one or more sensor inputs in real-time to determine the at least one source location and communicate the at least one source location to the audio processing module.

19. The system of claim 17 wherein the at least one sensor comprises a camera or a motion sensor.

20. A non-transitory computer-readable medium encoded with instructions for commanding one or more processors to execute operations of a method for spatial audio processing, the operations comprising:

receiving an audio input comprising audio signals captured by a plurality of transducers within an acoustic environment;

converting the audio input from a time domain to a frequency domain according to at least one transform function;

determining at least one acoustic propagation model for at least one source location within the acoustic environment according to a normalized cross power spectral density calculation, the at least one acoustic propagation model comprising at least one Green'"'"'s Function estimation;

processing the audio input according to the at least one acoustic propagation model to spatially filter at least one target audio signal from one or more non-target audio signals, wherein the at least one target audio signal corresponds to the at least one source location; and

applying a whitening filter to a spatially filtered target audio signal to derive at least one separated audio output signal, wherein the whitening filter is applied concurrently or concomitantly with the at least one acoustic propagation model.