Integration of Embedded and Network Speech Recognizers

• US 20120084079A1
• Filed: 11/02/2011
• Published: 04/05/2012
• Est. Priority Date: 01/26/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Byrne, William J., Gruenstein, Alexander
• Granted Patent: US 8,412,532 B2
• Time in Patent Office: Days
• US Class Current: 704/201
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems

Abstract

A method, computer program product, and system are provided for performing a voice command on a client device. The method can include translating, using a first speech recognizer located on the client device, an audio stream of a voice command to a first machine-readable voice command and generating a first query result using the first machine-readable voice command to query a client database. In addition, the audio stream can be transmitted to a remote server device that translates the audio stream to a second machine-readable voice command using a second speech recognizer. Further, the method can include receiving a second query result from the remote server device, where the second query result is generated by the remote server device using the second machine-readable voice command and displaying the first query result and the second query result on the client device.

Claims

1. A method, comprising:

receiving, at a client device, an audio stream that defines a voice command;

defining, using a first speech recognizer module stored at the client device, a first machine-readable voice command based at least in part on the audio stream;

receiving a first query result responsive to a first query sent to a client database, the first query including the first machine-readable voice command;

transmitting the audio stream to a remote server device such that the remote server device defines a second machine-readable voice command using a second speech recognizer module, the second machine-readable voice command being based at least in part on the audio stream;

receiving a second query result from the remote server device, the second query result being responsive to the transmitted audio stream; and

displaying the first query result and the second query result at a display of the client device, wherein the displayed first query result includes at least a first selectable result item, the second query result includes at least a second selectable result item, and the first query result is displayed independent of the second query result.

2. The method of claim 1, further comprising:

storing at least a portion of the first query result and the second query result at a memory of the client device.

3. The method of claim 2, further comprising:

receiving, at the client device, a second audio stream that defines a subsequent voice command;

defining, using the first speech recognizer module, a third machine-readable voice command based at least in part on the subsequent voice command;

determining whether the third machine-readable voice command is substantially similar to the first machine-readable voice command; and

retrieving, from the memory of the client device, the stored first query result and the stored second query result when the third machine-readable voice command is substantially similar to the first machine-readable voice command.

4. The method of claim 3, further comprising:

transmitting the second audio stream associated with the subsequent voice command to the remote server device such that the remote server device defines a fourth machine-readable voice command using the second speech recognizer module, the fourth machine-readable voice command being based at least in part on the second audio stream;

receiving a third query result from the remote server device, the third query result being responsive to the transmitted second audio stream; and

displaying the third query result at the display of the client device.

5. The method of claim 2, wherein the storing at least a portion of the first query result and the second query result comprises storing at least a portion of the second query result at the memory, the method further comprising:

identifying which portion of the first query result and the second query result to store at the memory, the identifying comprising receiving a user selection of the second selectable result item included in the second query result.

6. The method of claim 1, further comprising:

transmitting the audio stream to the remote server device during a first time interval; and

sending the first query to the client database during a second time interval,

wherein at least a portion of the first time interval and a portion of the second time interval overlap in time.

7. The method of claim 1, wherein transmitting the audio stream comprises transmitting a compressed audio stream of the voice command from the client device to the remote server device.

8. The method of claim 1, wherein displaying the first query result and the second query result comprises displaying the first query result and a first subset of the second query result at a first time instance and displaying the first query result, the first subset of the second query result, and a second subset of the second query result at a second time instance.

21. The method of claim 1, wherein the displaying comprises displaying the first query result and the second query result in a single display view at the display of the client device, the first query result being displayed in a first field of the display view and the second query result being displayed in a second field of the display view.

9. A non-transitory processor-readable medium storing code representing instructions that when executed cause a processor of a client device to:

receive an audio stream that defines a voice command;

define, using a first speech recognizer module stored at the client device, a first machine-readable voice command based at least in part on the audio stream;

send a first query to a client database, the first query being based at least in part on the first machine-readable voice command;

receive a first query result responsive to the first query sent to the client database, the first query result including a list of M selectable result items, where M is a whole number;

transmit the audio stream to a remote server device such that the remote server device defines a second machine-readable voice command using a second speech recognizer module, the second machine-readable voice command being based at least in part on the audio stream;

receive a second query result from the remote server device, the second query result being responsive to the transmitted audio stream and including a list of N selectable result items, where N is a whole number;

output the first query result including the list of M selectable result items for display on the client device; and

output the second query result including the list of N selectable result items for display on the client device.

10. The non-transitory processor-readable medium of claim 9, the code representing instructions that when executed further cause the processor to:

initiate at least a portion of the first query result and the second query result to be stored at a memory of the client device.

11. The non-transitory processor-readable medium of claim 10, the code further representing instructions that when executed cause the processor to:

receive a second audio stream that defines a subsequent voice command;

define, using the first speech recognizer module, a third machine-readable voice command based at least in part on the subsequent voice command;

determine that the third machine-readable voice command is substantially similar to the first machine-readable voice command; and

retrieve, from the memory of the client device, the stored first query result and the stored second query result.

12. The non-transitory processor-readable medium of claim 11, the code representing instructions that when executed further cause the processor to:

transmit the second audio stream associated with the subsequent voice command to the remote server device such that the remote server device defines a fourth machine-readable voice command using the second speech recognizer module, the fourth machine-readable voice command being based at least in part on the second audio stream;

receive a third query result from the remote server device, the third query result being responsive to the transmitted second audio stream and including a list of P selectable result items, where P is a whole number; and

output the third query result for display at the client device.

13. The non-transitory processor-readable medium of claim 10, wherein the code representing instructions that when executed cause the processor to store at least a portion of the first query result and the second query result at the memory of the client device further causes the processor to:

identify which portion of the first query result and the second query result to store at the memory of the client device, the identification comprising receiving a user selection of a selectable result item in the list of N selectable result items included in the second query result.

14. The non-transitory processor-readable medium of claim 9, wherein the code representing instructions that when executed further causes the processor to:

transmit the audio stream to the remote server device during a first time interval; and

send the first query to the client database during a second time interval,

wherein at least a portion of the first time interval and a portion of the second time interval overlap in time.

22. The non-transitory processor-readable medium of claim 9, wherein the code representing instructions that when executed cause the processor to output the first query result and the second query result further causes the processor to:

output the first query result and the second query result in a single display view for display at the client device, the first query result being displayed in a first field of the display view and the second query result being displayed in a second field of the display view.

15. A system, comprising:

a first speech recognizer module, stored at a client device, configured to;

receive an audio stream that defines a voice command; and

define a first machine-readable voice command based at least in part on the audio stream;

a client query manager configured to;

receive a first query result including a first number of selectable result items responsive to sending a first query to a client database, the first query being based at least in part on the first machine-readable voice command;

transmit the audio stream to a remote server device such that the remote server device defines a second machine-readable voice command using a second speech recognizer module, the second machine-readable voice command being based at least in part on the audio stream; and

receive a second query result including a second number of selectable result items from the remote server device, the second query result being responsive to the transmitted audio stream; and

a display device configured to display on the client device the first number of selectable result items corresponding to the first query result and the second number of selectable result items corresponding to the second query, wherein the first number of selectable items is displayed independent of the second number of selectable items.

16. The system of claim 15, further comprising:

a microphone configured to receive the audio stream of the voice command and to provide the audio stream to the first speech recognizer module; and

a storage device configured to store at least a portion of the first query result and the second query result at a memory of the client device.

17. The system of claim 16, wherein

the first speech recognizer module is configured to:

receive a second audio stream that defines a subsequent voice command; and

define a third machine-readable voice command based at least in part on the subsequent voice command; and

the client query manager is configured to;

determine whether the third machine-readable voice command is substantially similar to the first machine-readable voice command; and

retrieve, from the memory of the client device, the stored first query result and the stored second query result from the storage device when the third machine-readable voice command is substantially similar to the first machine-readable voice command.

18. The system of claim 17, wherein

the client query manager is configured to:

transmit the second audio stream associated with the subsequent voice command to the remote server device such that the remote server device defines a fourth machine-readable voice command using the second speech recognizer module, the fourth machine-readable voice command being based at least in part on the second audio stream; and

receive a third query result including a third number of selectable result items from the remote server device, the third query result being responsive to the transmitted second audio stream; and

the display device is configured to display on the client device the third number of selectable result items corresponding to the third query result.

19. The system of claim 15, wherein the client query manager is configured to transmit the audio stream to the remote server device during a first time interval and to send the first query to the client database during a second time interval, wherein at least a portion of the first time interval and a portion of the second time interval overlap in time.

20. The system of claim 15, wherein the display device is configured to display the first query result and a first subset of the second query result at a first time instance and display the first query result, the first subset of the second query result, and a second subset of the second query result at a second time instance.

23. The system of claim 15, wherein the display device is configured to display the first query result and the second query result in a single display view, the first query result being displayed in a first field of the display view and the second query result being displayed in a second field of the display view.