Three-dimensional beam forming with a microphone array

• US 10,880,644 B1
• Filed: 12/16/2019
• Issued: 12/29/2020
• Est. Priority Date: 09/28/2017
• Status: Active Grant

• Current Assignee: Sonos, Inc.
• Original Assignee: Sonos, Inc.
• Inventors: Buoni, Matthew, Kadri, Romi, Oishi, Tetsuro
• Primary Examiner(s): Nguyen, Khai N.
• Application Number:
  US16/715,713
• Time in Patent Office: 379 Days
• Field of Search: 381 92, 704200, 704251, 7042701
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/28 Constructional details of speech recognition systems
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 2227/003 Digital PA systems using, e.g. LAN or internet
• CPC Class Codes: H04R 2227/005 Audio distribution systems for home, i.e. multi-room use
• CPC Class Codes: H04R 2430/21 Direction finding using differential microphone array [DMA]
• CPC Class Codes: H04R 29/005 Microphone arrays
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

Systems and methods for three-dimensional beamforming disclosed herein include, among other features (i) generating a set of received-sound beams by applying a plurality of sets of beamforming coefficients stored in a tangible memory of the network device to sound received via a microphone array of the network device, wherein each received-sound beam corresponds to a separate direction relative to the microphone array, (ii) identifying a subset of the received-sound beams comprising speech content, (iii) for each received-sound beam in the subset of the received-sound beams comprising speech content, determining whether the speech content comprises a wake word, (iv) selecting one final received-sound beam from the received-sound beams in the subset of the received-sound beams determined to comprise a wake word; and (v) causing the selected one final received-sound beam to be processed to identify a voice command.

Claims

1. A networked microphone device comprising:

one or more processors;

a microphone array; and

tangible, non-transitory computer-readable media comprising instructions encoded therein, wherein the instructions, when executed by the one or more processors, cause the networked microphone device to perform functions comprising;

continuously monitoring each received-sound beam in a set of received-sound beams generated by the networked microphone device, wherein each received-sound beam generated by the networked microphone device corresponds to a separate three-dimensional polar direction relative to the microphone array;

determining, for each received-sound beam, whether the received-sound beam comprises speech content;

ranking received-sound beams determined to comprise speech content; and

selecting one received-sound beam having the highest ranking for voice command processing.

2. The networked microphone device of claim 1, wherein determining, for each received-sound beam, whether the received-sound beam comprises speech content comprises:

calculating a correlation metric for each received-sound beam relative to every other received-sound beam;

storing each received-sound beam'"'"'s correlation metric relative to every other received-sound beam in a cross-correlation matrix; and

selecting for ranking, two or more received-sound beams having a high likelihood of speech content and a low correlation with each other based on the cross-correlation matrix.

3. The networked microphone device of claim 1, wherein ranking the received-sound beams determined to comprise speech content comprises ranking the received-sound beams according to a likelihood that the speech content comprises a wake word.

4. The networked microphone device of claim 1, wherein the functions further comprise transmitting a sound sample received via the selected one received-sound beam to a separate computing system for voice analysis.

5. The networked microphone device of claim 4, wherein the functions further comprise:

before transmitting the sound sample received via the selected one received-sound beam to the separate computing system for voice analysis, mixing an inverse of a received-sound beam orthogonal to the selected one received-sound beam with the selected one received-sound beam.

6. The networked microphone device of claim 4, wherein the functions further comprise:

after transmitting the sound sample received via the selected one received-sound beam to the separate computing system for voice analysis, receiving a command corresponding to the sound sample; and

in response to receiving the command corresponding to the sound sample, executing the command.

7. The networked microphone device of claim 6, wherein the command comprises a command to play media content, and wherein executing the command comprises causing the networked microphone device to play audio content from a playback queue of the networked microphone device via a streaming media service.

8. The networked microphone device of claim 6, wherein the command comprises a command to modify a playback queue, and wherein executing the command comprises modifying a playback queue of audio content for future playback.

9. Tangible, non-transitory computer-readable medium comprising instructions encoded therein, wherein the instructions, when executed, cause a networked microphone device to perform functions comprising:

continuously monitoring each received-sound beam in a set of received-sound beams generated by the networked microphone device, wherein each received-sound beam generated by the networked microphone device corresponds to a separate three-dimensional polar direction relative to a microphone array of the networked microphone device;

determining, for each received-sound beam, whether the received-sound beam comprises speech content;

ranking received-sound beams determined to comprise speech content; and

selecting one received-sound beam having the highest ranking for voice command processing.

10. The tangible, non-transitory computer-readable medium of claim 9, wherein determining, for each received-sound beam, whether the received-sound beam comprises speech content comprises:

calculating a correlation metric for each received-sound beam relative to every other received-sound beam;

storing each received-sound beam'"'"'s correlation metric relative to every other received-sound beam in a cross-correlation matrix; and

selecting for ranking, two or more received-sound beams having a high likelihood of speech content and a low correlation with each other based on the cross-correlation matrix.

11. The tangible, non-transitory computer-readable medium of claim 9, wherein ranking the received-sound beams determined to comprise speech content comprises ranking the received-sound beams according to a likelihood that the speech content comprises a wake word.

12. The tangible, non-transitory computer-readable medium of claim 9, wherein the functions further comprise transmitting a sound sample received via the selected one received-sound beam to a separate computing system for voice analysis.

13. The tangible, non-transitory computer-readable medium of claim 12, wherein the functions further comprise:

before transmitting the sound sample received via the selected one received-sound beam to the separate computing system for voice analysis, mixing an inverse of a received-sound beam orthogonal to the selected one received-sound beam with the selected one received-sound beam.

14. The tangible, non-transitory computer-readable medium of claim 12, wherein the functions further comprise:

after transmitting the sound sample received via the selected one received-sound beam to the separate computing system for voice analysis, receiving a command corresponding to the sound sample; and

in response to receiving the command corresponding to the sound sample, executing the command.

15. The tangible, non-transitory computer-readable medium of claim 14, wherein the command comprises a command to play media content, and wherein executing the command comprises causing the networked microphone device to play audio content from a playback queue of the networked microphone device via a streaming media service.

16. The tangible, non-transitory computer-readable medium of claim 14, wherein the command comprises a command to modify a playback queue, and wherein executing the command comprises modifying a playback queue of audio content for future playback.

17. A method performed by a networked microphone device, the method comprising:

continuously monitoring each received-sound beam in a set of received-sound beams generated by the networked microphone device, wherein each received-sound beam generated by the networked microphone device corresponds to a separate three-dimensional polar direction relative to a microphone array of the networked microphone device;

determining, for each received-sound beam, whether the received-sound beam comprises speech content;

ranking received-sound beams determined to comprise speech content; and

selecting one received-sound beam having the highest ranking for voice command processing.

18. The method of claim 17, wherein determining, for each received-sound beam, whether the received-sound beam comprises speech content comprises:

calculating a correlation metric for each received-sound beam relative to every other received-sound beam;

storing each received-sound beam'"'"'s correlation metric relative to every other received-sound beam in a cross-correlation matrix; and

selecting for ranking, two or more received-sound beams having a high likelihood of speech content and a low correlation with each other based on the cross-correlation matrix.

19. The method of claim 17, wherein ranking the received-sound beams determined to comprise speech content comprises ranking the received-sound beams according to a likelihood that the speech content comprises a wake word.

20. The method of claim 17, wherein the method further comprises transmitting a sound sample received via the selected one received-sound beam to a separate computing system for voice analysis.