Apparatus and method for beamforming to obtain voice and noise signals

• US 10,229,697 B2
• Filed: 07/31/2013
• Issued: 03/12/2019
• Est. Priority Date: 03/12/2013
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Inventors: Bastyr, Kevin J., Davis, Giles T, Ivanov, Plamen A, Oliveira, Rivanaldo S, Ramabadran, Tenkasi V, Singaraju, Snehitha
• Primary Examiner(s): Opsasnick, Michael N
• Application Number:
  US13/955,723
• Publication Number:
  US 20140278394A1
• Time in Patent Office: 2,050 Days
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/20 Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech G10L21/02 takes precedence
• CPC Class Codes: G10L 2021/02165 Two microphones, one receiving mainly the noise signal and the other one mainly the speech signal
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

One method of operation includes beamforming a plurality of microphone outputs to obtain a plurality of virtual microphone audio channels. Each virtual microphone audio channel corresponds to a beamform. The virtual microphone audio channels include at least one voice channel and at least one noise channel. The method includes performing voice activity detection on the at least one voice channel and adjusting a corresponding voice beamform until voice activity detection indicates that voice is present on the at least one voice channel. Another method beamforms the plurality of microphone outputs to obtain a plurality of virtual microphone audio channels, where each virtual microphone audio channel corresponds to a beamform, and with at least one voice channel and at least one noise channel. The method performs voice recognition on the at least one voice channel and adjusts the corresponding voice beamform to improve a voice recognition confidence metric.

Claims

1. An apparatus comprising:

a first voice activity detector, the first voice activity detector configured to generate, based on a voice signal, a first voice control signal to indicate whether a voice is detected in the voice signal;

a first noise estimator, the first noise estimator configured to generate, based on a noise level of the voice signal, a first noise control signal to indicate whether to apply noise suppression for the voice signal;

a second voice activity detector, the second voice activity detector configured to generate, based on a noise signal, a second voice control signal to indicate whether another voice is detected in the noise signal;

a second noise estimator, the second noise estimator configured to generate, based on another noise level of the noise signal, a second noise control signal to indicate whether to apply noise suppression for the noise signal; and

a beamformer controller coupled to the first voice activity detector, the first noise estimator, the second voice activity detector, and the second noise estimator, the beamformer controller configured to;

adjust, based on the first voice control signal and the first noise control signal, a voice beamform pattern effective to cause the voice to be substantially present in the voice signal;

adjust, based on the second voice control signal and the second noise control signal, a noise beamform pattern effective to cause noise to be substantially present in the noise signal; and

generate, based on the first noise control signal and the second noise control signal, a noise suppression control signal to control noise suppression for the voice signal and the noise signal.

2. The apparatus of claim 1, wherein the beamformer controller is configured to generate, based on the first voice control signal and the second voice control signal, a voice recognition control signal to control voice recognition and speaker identification for the voice signal and the noise signal.

3. The apparatus of claim 2, further comprising:

a voice recognition engine coupled to the beamformer controller, the voice recognition engine configured to;

detect whether the voice in the voice signal and the other voice in the noise signal are present;

responsive to detecting the voice and the other voice, determine whether the voice in the voice signal and the other voice in the noise signal are associated with a same speaker; and

generate, based on the voice recognition control signal, at least one voice confidence metric that is associated with the detection and the determination, and wherein;

the beamformer controller is further configured to;

adjust the voice beamform pattern and the noise beamform pattern based on that at least one voice confidence metric; and

generate the noise suppression control signal based on the at least one voice confidence metric.

4. The apparatus of claim 1, further comprising:

a group of microphones having an orientation;

an orientation sensor coupled to the beamformer controller, the orientation sensor configured to determine the orientation of the group of microphones, and wherein;

the beamformer controller is further configured to adjust the voice beamform pattern and the noise beamform pattern based on the orientation of the group of microphones.

5. The apparatus of claim 4, wherein:

the orientation of the group of microphones corresponds to a tabletop orientation in which a side of the apparatus is placed on a tabletop; and

the beamformer controller is further configured to adjust the voice beamform pattern and the noise beamform pattern based on a pre-determined microphone configuration associated with the tabletop orientation.

6. The apparatus of claim 4, wherein:

the orientation of the group of microphones corresponds to a docking station orientation in which the apparatus is placed in a docking station; and

the beamformer controller is further configured to adjust the voice beamform pattern and the noise beamform pattern based on a pre-determined microphone configuration associated with the docking station orientation.

7. The apparatus of claim 1, wherein:

the adjusting of the voice beamform pattern by the beamformer controller is effective to steer a peak of the voice beamform pattern in a direction of the voice; and

the adjusting of the noise beamform pattern by the beamformer controller is effective to steer a null of the noise beamform pattern in the direction of the voice.

8. The apparatus of claim 1, wherein the noise includes a jammer voice that is not associated with the voice in the voice signal.

9. The apparatus of claim 1, wherein the noise includes environmental background noise.

10. The apparatus of claim 1, wherein the beamformer controller is configured to adjust the voice beamform pattern based on a random set of voice beamform patterns.

11. The apparatus of claim 1, wherein the beamformer controller is configured to scan the voice beamform pattern until the voice is detected in the voice signal by the first voice activity detector.

12. A method comprising:

generating at least two voice control signals that respectively indicate whether a voice is present in a voice signal and another voice is present in a noise signal;

generating at least two noise control signals that indicate whether to apply noise suppression based on a noise level of the voice signal and another noise level of the noise signal, respectively;

adjusting, based on the at least two voice control signals and the at least two noise control signals, a voice beamform pattern effective to cause the voice to be substantially present in the voice signal and a noise beamform pattern effective to cause noise to be substantially present in the noise signal; and

generating, based on the at least two noise control signals, a noise suppression control signal to activate noise suppression for the voice signal and the noise signal.

13. The method of claim 12, further comprising responsive to the at least two voice control signals indicating a presence of the voice in the voice signal and the other voice in the noise signal, determining whether the voice in the voice signal and the other voice in the noise signal are associated with a same speaker.

14. The method of claim 13, wherein adjusting the voice beamform pattern and the noise beamform pattern is effective to provide higher voice confidence metrics associated with detecting the voice and the other voice and determining whether the voice and the other voice are associated with the same speaker.

15. The method of claim 13, wherein:

the voice in the voice signal and the other voice in the noise signal are determined to be associated with the same speaker; and

adjusting the noise beamform pattern comprises adjusting the noise beamform pattern effective to cause the other voice to not be substantially present in the noise signal.

16. The method of claim 13, wherein:

the voice in the voice signal and the other voice in the noise signal are associated with different speakers;

adjusting the voice beamform pattern comprises adjusting a null of the voice beamform pattern in a direction of the other voice; and

adjusting the noise beamform pattern comprises adjusting a peak of the noise beamform pattern in the direction of the other voice.

17. The method of claim 12, further comprising:

determining an orientation of a group of microphones that produce the voice beamform pattern and the noise beamform pattern; and

adjusting the voice beamform pattern and the noise beamform pattern based on the orientation of the group of microphones.

18. A beamformer controller coupled to a first voice activity detector of a voice channel, a first noise estimator of the voice channel, a second voice activity detector of a noise channel, a second noise estimator of the noise channel, a noise suppressor, a voice recognition engine, a voice beamformer, and a noise beamformer, the beamformer controller configured to:

receive, from the first voice activity detector, a first voice control signal that indicates whether a voice is detected in a voice signal;

receive, from the first noise estimator, a first noise control signal that indicates a noise level of the voice signal;

receive, from the second voice activity detector, a second voice control signal that indicates whether another voice is detected in a noise signal;

receive, from the second noise estimator, a second noise control signal that indicates another noise level of the noise signal;

adjust, based on the first voice control signal and the first noise control signal, a voice beamform pattern for the voice signal via the voice beamformer;

adjust, based on the second voice control signal and the second noise control signal, a noise beamform pattern for the noise signal via the noise beamformer;

generate, based on the first noise control signal and the second noise control signal, a noise suppression control signal to control whether the noise suppressor is to perform noise suppression using the voice signal and the noise signal; and

generate a voice recognition control signal based on the first voice control signal and the second voice control signal to control whether the voice recognition engine is to perform voice recognition for the voice signal and the noise signal.

19. The beamformer controller of claim 18, wherein the adjusting of the voice beamform pattern, the adjusting of the noise beamform pattern, the generating of the noise suppression control signal, and the generating of the voice recognition control signal are effective to enable the beamformer controller to respond to a dynamic acoustic environment that changes based on orientation of a group of microphones, motion of the group of microphones, and environmental noise.

20. The beamformer controller of claim 18, further configured to:

receive, from a first energy estimator, a first energy control signal that indicates an energy level of the voice signal;

receive, from a second energy estimator, a second energy control signal that indicates another energy level of the noise signal;

adjust a directivity of the voice beamformer based on the energy level of the voice signal; and

adjust another directivity of the noise beamformer based on the other energy level of the noise signal.