Position directed acoustic array and beamforming methods

• US 9,747,917 B2
• Filed: 06/14/2013
• Issued: 08/29/2017
• Est. Priority Date: 06/14/2013
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Tzirkel-Hancock, Eli, Bilik, Igal, Laifenfeld, Moshe
• Primary Examiner(s): Vo, Huyen
• Assistant Examiner(s): CHAVEZ, RODRIGO A
• Application Number:
  US13/918,752
• Publication Number:
  US 20140372129A1
• Time in Patent Office: 1,537 Days
• Field of Search: 704226, 704233, 704278
• CPC Class Codes: G01S 13/06 Systems determining position data of a target
• CPC Class Codes: G01S 15/06 Systems determining the position data of a target
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 21/00 Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility G10L19/00 takes precedence
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: H04R 2201/401 2D or 3D arrays of transducers
• CPC Class Codes: H04R 2201/405 Non-uniform arrays of transducers or a plurality of uniform arrays with different transducer spacing
• CPC Class Codes: H04R 2430/25 Array processing for suppression of unwanted side-lobes in directivity characteristics, e.g. a blocking matrix
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

Methods and systems are provided for receiving desired sounds. The system includes a position sensor configured to determine an occupant position of an occupant engaging in speech within a defined space and transmit the speaking occupant position. A plurality of microphones are configured to receive sound from within the defined space and transmit audio signals corresponding to the received sound. A processor, in communication with the position sensor and the microphones, is configured to receive the speaking occupant position and the audio signals, apply a beamformer to the audio signals to direct a microphone beam toward the occupant position, and generate a beamformer output signal.

Claims

1. An adaptive beamforming method for receiving desired sounds, said method comprising:

determining an occupant position of an occupant engaging in speech within a defined space in a vehicle, said determining comprising

transmitting waves in an ultrasonic range and/or a radio frequency (RF) range with a plurality of transmitters,

receiving reflected waves, and

analyzing signals corresponding to the transmitted and reflected waves to determine the occupant position;

determining when the occupant is engaging in speech by identifying motion of a mouth of the occupant engaging in speech by using a sensor in the vehicle and searching for distinguishing features in spatio-temporal and/or velocity features of signals corresponding to the reflected waves, and generating speech detection based on the determination, the speech detection including detection of when the occupant is speaking based on the detected motion of the mouth of the occupant;

receiving sound from within the defined space with a plurality of microphones;

generating a plurality of audio signals corresponding to the received sound at each of said plurality of microphones;

applying a beamformer to the audio signals to direct a microphone beam toward the determined occupant position and generate a beamformer output signal, wherein;

the applying of the beamformer comprises modifying the beamformer based simultaneously on both the determined occupant position and the speech detection, including the detection of when the occupant is speaking based on the detected motion of the mouth of the occupant; and

the applying of the beamformer comprises applying a sidelobe canceller to the audio signals to direct a microphone beam toward the determined occupant position and generate a beamformer output signal, by modifying the sidelobe canceller based on both the determined occupant position and the speech detection, including the detection of when the occupant is speaking based on the detected motion of the mouth of the occupant; and

applying a blocking matrix to the audio signals via the sidelobe canceller, and generating a plurality of blocking matrix output signals, each of which corresponds to one of the audio signals.

2. A method as set forth in claim 1 further comprising:

applying an adaptive noise cancellation process to the blocking matrix output signals to generate an adaptive noise cancellation output signal; and

summing the beamformer output signal and the adaptive noise cancellation output signal to generate a speaking occupant output signal.

10. The method of claim 1, further comprising:

summing the adaptive noise cancellation output signal with a beamformer output signal from the application of the beamformer to the audio signals, to generate a speaking output signal for a speech recognition system using the output of the beamformer.

13. The method of claim 1, wherein the beamformer changes the direction of the microphone beam as the mouth of the occupant moves, as detected via the speech detection.

16. The method of claim 1, wherein the beamformer changes the direction of the microphone beam as the lips of the occupant moves, as detected via the speech detection.

3. A system for receiving desired sounds, said system comprising:

a position sensor configured to determine an occupant position of an occupant engaging in speech within a defined space and transmit the speaking occupant position, said position sensor comprising;

a signal generator,

a plurality of transmitters electrically coupled to said signal generator and configured to generate waves in an ultrasonic range and/or an RF range,

a plurality of receivers configured to receive reflected waves and generate a plurality of received signals corresponding to the received reflected waves, and

a processing unit electrically coupled to said receivers and configured to receive the received signals, determine the occupant position, and identify motion of a mouth of the occupant engaging in speech by searching for distinguishing features in spatio-temporal and/or velocity features of signals corresponding to the reflected waves, and generating speech detection based on the determination, the speech detection including detection of when the occupant is speaking based on the detected motion of the mouth of the occupant;

a plurality of microphones configured to receive sound from within the defined space and transmit audio signals corresponding to the received sound; and

a processor in communication with said position sensor and said microphones and configured to receive the speaking occupant position and the audio signals, apply a beamformer to the audio signals to direct a microphone beam toward the determined occupant position, and generate a beamformer output signal, wherein the processor is configured to apply the beamformer by modifying the beamformer based simultaneously on both the determined occupant position and the speech detection, including the detection of when the occupant is speaking based on the detected motion of the mouth of the occupant.

4. A system as set forth in claim 3 wherein said processor is also configured to apply a blocking matrix to each of the audio signals to exclude other positions not associated with the occupant engaging in speech to generate a plurality of blocking matrix output signals corresponding to each of the audio signals, apply an adaptive noise cancellation process to the blocking matrix output signals to generate an adaptive noise cancellation output signal, and sum the beamformer output signal and the adaptive noise cancellation output signal to generate a speaking occupant output signal.

5. A system as set forth in claim 3 wherein said position sensor comprises a camera.

9. The system of claim 3, wherein the processor is configured to:

apply a sidelobe canceller to the audio signals to direct a microphone beam toward the determined occupant position and generate a beamformer output signal, by modifying the sidelobe canceller based on both the determined occupant position and the speech detection; and

apply a blocking matrix to the audio signals via the sidelobe canceller, and generate a plurality of blocking matrix output signals, each of which corresponds to one of the audio signals.

12. The system of claim 3, wherein the position sensor is disposed within a vehicle, and the position sensor is configured to determine an occupant position of an occupant in a passenger compartment of a vehicle engaging in speech within a defined space in the passenger compartment of the vehicle.

14. The system of claim 3, wherein the beamformer changes the direction of the microphone beam as the mouth of the occupant moves, as detected via the speech detection.

6. A vehicle comprising:

a passenger compartment;

an audio system comprising

a position sensor configured to determine an occupant position of an occupant engaging in speech within the passenger compartment and transmit the speaking occupant position, said position sensor comprising

a signal generator,

a plurality of transmitters electrically coupled to said signal generator and configured to generate waves in an ultrasonic range and/or an RF range,

a plurality of receivers configured to receive reflected waves and generate a plurality of received signals corresponding to the received reflected waves, and

a processing unit electrically coupled to said receivers and configured to receive the received signals, determine the occupant position, and identify motion of a mouth of the occupant engaging in speech by searching for distinguishing features in spatio-temporal and/or velocity features of signals corresponding to the reflected waves, and generating speech detection based on the determination, the speech detection including detection of when the occupant is speaking based on the detected motion of the mouth of the occupant;

a plurality of microphones configured to receive sound from within the defined space and transmit audio signals corresponding to the received sound; and

a processor in communication with said position sensor and said microphones and configured to receive the speaking occupant position and the audio signals, apply a beamformer to the audio signals to direct a microphone beam toward the occupant position, and generate a beamformer output signal, apply a blocking matrix to each of the audio signals to exclude other positions not associated with the occupant engaging in speech to generate a plurality of blocking matrix output signals corresponding to each of the audio signals, apply an adaptive noise cancellation process to the blocking matrix output signals to generate an adaptive noise cancellation output signal, and sum the beamformer output signal and the adaptive noise cancellation output signal to generate a speaking occupant output signal, wherein the processor is configured to apply the beamformer by modifying the beamformer based simultaneously on both the determined occupant position and the speech detection, including the detection of when the occupant is speaking based on the detected motion of the mouth of the occupant; and

at least one of a speech recognition system and a telephony system in communication with said audio system for receiving said speaking occupant output signal.

7. A vehicle as set forth in claim 6 wherein said plurality of transmitters comprise a plurality of ultrasonic transmitters electrically coupled to said signal generator and configured to generate sound waves in the ultrasonic range.

8. A vehicle as set forth in claim 6 wherein said plurality of transmitters comprise a plurality of RF transmitters electrically coupled to said signal generator for generating RF waves in the RF range.

11. The vehicle of claim 6, wherein the processor is configured to:

apply a sidelobe canceller to the audio signals to direct a microphone beam toward the determined occupant position and generate a beamformer output signal, by modifying the sidelobe canceller based on both the determined position and the speech detection; and

apply a blocking matrix to the audio signals via the sidelobe canceller, and generate a plurality of blocking matrix output signals, each of which corresponds to one of the audio signals.

15. The vehicle of claim 6, wherein the beamformer changes the direction of the microphone beam as the mouth of the occupant moves, as detected via the speech detection.