Asynchronous transfer of audio data

• US 10,297,250 B1
• Filed: 03/11/2013
• Issued: 05/21/2019
• Est. Priority Date: 03/11/2013
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Blanksteen, Scott Ian, Strom, Nikko, Velusamy, Kavitha, David, Tony, Crump, Edward Dietz
• Primary Examiner(s): Mishra, Richa
• Application Number:
  US13/792,505
• Time in Patent Office: 2,262 Days
• Field of Search: None
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 21/02 Speech enhancement, e.g. noise reduction or echo cancellation reducing echo effects in line transmission systems H04B3/20; echo suppression in hands-free telephones H04M9/08

Abstract

The systems, devices, and processes described herein may asynchronously transfer audio signals from a voice-controlled device to a remote device. The audio signals may correspond to sound that is captured by multiple microphones of the voice-controlled device, which may then process the audio signals. The audio signals may also be transferred to the remote device for processing. Moreover, a determination of whether the voice-controlled device or the remote device is to process the audio signals may be based at least in part on the bandwidth of a network communicatively coupled to the voice-controlled device. The voice-controlled device may also cache and log the audio signals, and then asynchronously stream the audio signals to the remote device after the audio signals are initially processed, which may be based on the bandwidth of the network. The remote device may utilize the unprocessed audio signals to improve subsequent processing of audio signals.

Claims

1. A system comprising:

a microphone configured to capture sound from a sound source within an environment and generate a first audio signal and a second audio signal based at least in part on data associated with the sound;

memory;

one or more processors;

one or more computer executable instructions stored in the memory and executable by the one or more processors to;

at least partially process the first audio signal, wherein partially processing the first audio signal reduces at least one of noise or echo from the data;

send, to a remote device and at a first rate, the first audio signal; and

send, to the remote device and at a second rate that is different than the first rate, the second audio signal, the second audio signal having been processed different than the first audio signal; and

the remote device being configured to;

perform speech recognition on the first audio signal based at least in part on an algorithm, the speech recognition including determining one or more words associated with the first audio signal;

perform speech recognition on the second audio signal based at least in part on an updated algorithm that is based at least in part on the algorithm; and

determine whether a first accuracy of speech recognition utilizing the updated algorithm is greater than a second accuracy of speech recognition utilizing the algorithm.

2. The system as recited in claim 1, wherein the one or more computer-executable instructions are further executable by the one or more processors to receive an indication that the algorithm has been selected, deselected, or modified or that the updated algorithm has been implemented, the algorithm or the updated algorithm being utilized to process subsequent audio signals.

3. The system as recited in claim 1, wherein the second rate is slower than the first rate.

4. The system as recited in claim 1, wherein the one or more computer-executable instructions are further executable by the one or more processors to log the at least the first portion of the first audio signal or the at least the second portion of the second audio signal, the logging including recording a time stamp, a size, or a duration of the first audio signal or the second audio signal or a location at which the first audio signal or the second audio signal was captured.

5. The system as recited in claim 1, wherein the one or more computer-executable instructions are further executable by the one or more processors to select the first audio signal or the second audio signal by determining a loudness of a sound source or a voice included within the first audio signal and the second audio signal.

6. A method comprising:

receiving, during a first time period, at a first rate, and from a device associated with a microphone, a first audio signal generated based at least in part on data received from the microphone, the first audio signal corresponding to sound that is associated with a sound source, the first audio signal having been at least partially processed by the device to reduce at least one of noise or echo from the data;

receiving, from the device, during a second time period that is subsequent to the first time period, and at a second rate that is slower than the first rate, a second audio signal, the second audio signal corresponding to the sound detected by the microphone and having been processed, by the device, different than the first audio signal;

based at least in part on receiving the second audio signal, generating an updated algorithm associated with an algorithm used to process subsequent audio signals;

performing speech recognition on the first audio signal based at least in part on the algorithm, the speech recognition including determining one or more words associated with the first audio signal;

performing speech recognition on the second audio signal based at least in part on the updated algorithm; and

determining whether a first accuracy of speech recognition utilizing the updated algorithm is greater than a second accuracy of speech recognition utilizing the algorithm.

7. The method as recited in claim 6, further comprising:

selecting one of the first audio signal or the second audio signal as a selected audio signal; and

processing the selected audio signal by reducing background noise, reverberation, or acoustic echo in the selected audio signal.

8. The method as recited in claim 6, further comprising:

determining that the first audio signal is received at the first rate; and

determining that the second audio signal is received at the second rate, the second rate being dependent upon a bandwidth of a network utilized to transfer the second audio signal.

9. The method as recited in claim 7, further comprising at least one of:

reducing, using the algorithm, noise in the selected audio signal;

or

determining, using the algorithm, one or more words that are associated with the selected audio signal.

10. The method as recited in claim 6, wherein the second audio signal is cached and logged on the device, the second audio signal being a sample of respective audio signals generated by the microphone.

11. The method as recited in claim 6, wherein updating the algorithm includes developing a new algorithm that is to be used to process the subsequent audio signals.

12. The method as recited in claim 6, wherein updating the algorithm includes at least one of modifying a value, variable, parameter, constant, or coefficient associated with the algorithm or determining a value, variable, parameter, constant, or coefficient associated with the algorithm.

13. The method as recited in claim 6, wherein updating the algorithm includes selecting or deselecting the algorithm.

24. The method as recited in claim 6, further comprising:

determining that the first accuracy of speech recognition utilizing the updated algorithm is greater than the second accuracy of speech recognition utilizing the algorithm; and

selecting the updated algorithm for subsequent processing of audio signals.

25. The method as recited in claim 24, further comprising performing, based at least in part on selecting the updated algorithm for subsequent processing of the audio signals, speech recognition on a third audio signal based at least in part on the updated algorithm.

14. A system comprising:

memory;

one or more processors; and

one or more computer-executable instructions stored in the memory and executable by the one or more processors to perform operations comprising;

receiving, during a first time period and from a device associated with a microphone, a first audio signal generated based at least in part on data received from the microphone, the audio signal corresponding to sound that is associated with a sound source, the first audio signal having been at least partially processed by the device to reduce at least one of noise or echo from the data;

receiving, during a second time period and from the device, a second audio signal, the second audio signal corresponding to the sound detected by the microphone and having been processed, by the device, different than the first audio signal;

based at least in part on receiving at least one of the first audio signal or the second audio signal, generating an algorithm used to process subsequent audio signal;

performing speech recognition on the first audio signal based at least in part on the algorithm, the speech recognition including determining one or more words associated with the first audio signal;

performing speech recognition on the second audio signal based at least in part on an updated algorithm that is based at least in part on the algorithm; and

determining whether a first accuracy of speech recognition utilizing the updated algorithm is greater than a second accuracy of speech recognition utilizing the algorithm.

15. The system as recited in claim 14, wherein the operations further comprise:

selecting one of the first audio signal or the second audio signal as a selected audio signal; and

processing the selected audio signal by reducing at least one of background noise, reverberation, or acoustic echo associated with the selected audio signal.

16. The system as recited in claim 15, wherein the operations further comprise:

determining a bandwidth of a network communicatively coupled to the device;

determining that the bandwidth at least one of meets or exceeds a bandwidth threshold; and

processing the selected audio signal.

17. The system as recited in claim 15, wherein the operations further comprise:

determining a bandwidth of a network communicatively coupled to the device;

determining that the bandwidth is less than a bandwidth threshold; and

instructing the device to process the selected audio signal.

18. The system as recited in claim 14, wherein the operations further comprise:

determining that the first audio signal is received at a first rate; and

determining that the second audio signal is received at a second rate that is slower than the first rate.

19. The system as recited in claim 18, wherein the operations further comprise determining that the second rate is dependent upon a bandwidth of a network utilized to transfer the second audio signal.

20. A method comprising:

receiving, during a first time period and from a device associated with a microphone, a first audio signal generated based at least in part on data received from the microphone, the audio signal corresponding to sound that is associated with a sound source, the first audio signal having been at least partially processed by the device to reduce at least one of noise or echo from the data;

receiving, during a second time period and from the device, a second audio signal, the second audio signal corresponding to the sound detected by the microphone and having been processed, by the device, different than the first audio signal;

based at least in part on at least one of the first audio signal or the second audio signal, determining an algorithm used to process subsequent audio signals;

generating an updated algorithm based at least in part on the algorithm;

performing speech recognition on the first audio signal based at least in part on the algorithm, the speech recognition including determining one or more words associated with the first audio signal;

performing speech recognition on the second audio signal based at least in part on the updated algorithm; and

determining whether a first accuracy of the speech recognition utilizing the updated algorithm is greater than a second accuracy of the speech recognition utilizing the algorithm.

21. The method as recited in claim 20, further comprising generating, based at least in part on an update to the algorithm, the updated algorithm.

22. The method as recited in claim 20, further comprising:

determining a bandwidth of a network communicatively coupled to the device;

determining that the bandwidth at least one of meets or exceeds a bandwidth threshold; and

processing at least one of the first audio signal or the second audio signal.

23. The method as recited in claim 20, further comprising:

determining a bandwidth of a network communicatively coupled to the device;

determining that the bandwidth is less than a bandwidth threshold; and

instructing the device to process at least one of the first audio signal or the second audio signal.