Three-dimensional beam forming with a microphone array

• US 10,511,904 B2
• Filed: 08/13/2018
• Issued: 12/17/2019
• Est. Priority Date: 09/28/2017
• Status: Active Grant

• Current Assignee: Sonos, Inc.
• Original Assignee: Sonos, Inc.
• Inventors: Buoni, Matthew, Kadri, Romi, Oishi, Tetsuro
• Primary Examiner(s): Nguyen, Khai N.
• Application Number:
  US16/102,650
• Publication Number:
  US 20190098400A1
• Time in Patent Office: 491 Days
• Field of Search: 381 92, 704200, 704251, 7042701
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/28 Constructional details of speech recognition systems
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 2227/003 Digital PA systems using, e.g. LAN or internet
• CPC Class Codes: H04R 2227/005 Audio distribution systems for home, i.e. multi-room use
• CPC Class Codes: H04R 2430/21 Direction finding using differential microphone array [DMA]
• CPC Class Codes: H04R 29/005 Microphone arrays
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

Systems and methods for three-dimensional beamforming disclosed herein include, among other features (i) generating a set of received-sound beams by applying a plurality of sets of beamforming coefficients stored in a tangible memory of the network device to sound received via a microphone array of the network device, wherein each received-sound beam corresponds to a separate direction relative to the microphone array, (ii) identifying a subset of the received-sound beams comprising speech content, (iii) for each received-sound beam in the subset of the received-sound beams comprising speech content, determining whether the speech content comprises a wake word, (iv) selecting one final received-sound beam from the received-sound beams in the subset of the received-sound beams determined to comprise a wake word; and (v) causing the selected one final received-sound beam to be processed to identify a voice command.

Claims

1. A networked microphone device comprising:

one or more processors;

a microphone array; and

tangible, non-transitory computer-readable media comprising instructions encoded therein, wherein the instructions, when executed by the one or more processors, cause the networked microphone device to perform a method comprising;

identifying a subset of received-sound beams from a set of two or more received-sound beams, wherein each received-sound beam comprises speech content suspected of including a wake word received via the microphone array, and wherein each received-sound beam corresponds to a separate direction relative to the microphone array;

selecting at least one final received-sound beam from the subset of the received-sound beams, and processing the selected at least one final received-sound beam to identify a voice command; and

in response to identifying the voice command, causing the identified voice command to be implemented by the networked microphone device.

2. The networked microphone device of claim 1, wherein identifying the subset of received-sound beams comprises:

calculating a correlation metric for each received-sound beam relative to every other received-sound beam;

storing each received-sound beam'"'"'s correlation metric relative to every other received-sound beam in a cross-correlation matrix; and

selecting for further processing, two or more received-sound beams having a high likelihood of speech content and a low correlation with each other based on the cross-correlation matrix.

3. The networked microphone device of claim 2, wherein selecting for further processing, two or more received-sound beams having a high likelihood of speech content and a low correlation with each other based on the cross-correlation matrix comprises:

ranking each received-sound beam based on its likelihood of comprising speech content; and

based on the ranking, selecting a top subset of two or more least-correlated received-sound beams for further processing.

4. The networked microphone device of claim 3, wherein selecting a top subset of two or more least-correlated received-sound beams for further processing further comprises selecting at least one received-sound beam from the top subset of two or more least-correlated received-sound beams having a highest likelihood of comprising speech content for further processing.

5. The networked microphone device of claim 1, wherein identifying a subset of received-sound beams from a set of two or more received-sound beams, wherein each received-sound beam comprises speech content suspected of including a wake word received via the microphone array, comprises transmitting a speech sample of at least one of the received-sound beams to a separate computing device for voice analysis.

6. The networked microphone device of claim 1, wherein processing the selected at least one final received-sound beam to identify a voice command comprises transmitting a speech sample of the at least one final received-sound beam to a separate computing device for voice analysis.

7. The networked microphone device of claim 1, wherein processing the selected at least one received-sound beam to identify a voice command comprises:

processing a sound sample from the selected at least one received-sound beams to identify a voice command; and

transmitting the sound sample from the at least one received-sound beams to a separate computing device for voice analysis.

8. The networked microphone device of claim 1, wherein the voice command comprises a command to playback media content, and wherein causing the identified voice command to be implemented by the networked microphone device comprises causing the networked microphone device to playback audio content from a playback queue of the networked microphone device via a streaming media service.

9. The networked microphone device of claim 1, wherein the voice command comprises a command to modify a playback queue, and wherein causing the identified voice command to be implemented by the networked microphone device comprises modifying a playback queue of audio content for future playback.

10. The networked microphone device of claim 1, wherein the method further comprises:

prior to processing the selected at least one received-sound beam, mixing an inverse of a received-sound beam orthogonal to the selected at least one received-sound beam with the selected at least one received-sound beam.

11. A tangible, non-transitory computer readable medium having stored therein instructions executable by a networked microphone device to cause the networked microphone device to perform a method comprising:

identifying a subset of received-sound beams from a set of two or more received-sound beams, wherein each received-sound beam comprises speech content suspected of including a wake word received via a microphone array, and wherein each received-sound beam corresponds to a separate direction relative to the microphone array;

selecting at least one final received-sound beam from the subset of the received-sound beams, and processing the selected at least one final received-sound beam to identify a voice command; and

in response to identifying the voice command, causing the identified voice command to be implemented by the networked microphone device.

12. The tangible, non-transitory computer readable medium of claim 11, wherein identifying the subset of received-sound beams comprises:

calculating a correlation metric for each received-sound beam relative to every other received-sound beam;

storing each received-sound beam'"'"'s correlation metric relative to every other received-sound beam in a cross-correlation matrix; and

selecting for further processing, two or more received-sound beams having a high likelihood of speech content and a low correlation with each other based on the cross-correlation matrix.

13. The tangible, non-transitory computer readable medium of claim 12, wherein selecting for further processing, two or more received-sound beams having a high likelihood of speech content and a low correlation with each other based on the cross-correlation matrix comprises:

ranking each received-sound beam based on its likelihood of comprising speech content; and

based on the ranking, selecting a top subset of two or more least-correlated received-sound beams for further processing.

14. The tangible, non-transitory computer readable medium of claim 13, wherein selecting a top subset of two or more least-correlated received-sound beams for further processing further comprises selecting at least one received-sound beam from the top subset of two or more least-correlated received-sound beams having a highest likelihood of comprising speech content for further processing.

15. The tangible, non-transitory computer readable medium of claim 11, wherein identifying a subset of received-sound beams from a set of two or more received-sound beams, wherein each received-sound beam comprises speech content suspected of including a wake word received via the microphone array, comprises transmitting a speech sample of at least one of the received-sound beams to a separate computing device for voice analysis.

16. The tangible, non-transitory computer readable medium of claim 11, wherein processing the selected at least one final received-sound beam to identify a voice command comprises transmitting a speech sample of the at least one final received-sound beam to a separate computing device for voice analysis.

17. The tangible, non-transitory computer readable medium of claim 11, wherein processing the selected at least one received-sound beam to identify a voice command comprises:

processing a sound sample from the selected at least one received-sound beams to identify a voice command; and

transmitting the sound sample from the at least one received-sound beams to a separate computing device for voice analysis.

18. The tangible, non-transitory computer readable medium of claim 11, wherein the voice command comprises a command to playback media content, and wherein causing the identified voice command to be implemented by the networked microphone device comprises causing the networked microphone device to playback audio content from a playback queue of the networked microphone device via a streaming media service.

19. The tangible, non-transitory computer readable medium of claim 11, wherein the voice command comprises a command to modify a playback queue, and wherein causing the identified voice command to be implemented by the networked microphone device comprises modifying a playback queue of audio content for future playback.

20. The tangible, non-transitory computer readable medium of claim 11, wherein the method further comprises:

prior to processing the selected at least one received-sound beam, mixing an inverse of a received-sound beam orthogonal to the selected at least one received-sound beam with the selected at least one received-sound beam.