COMPUTATIONAL ASSISTANT EXTENSION DEVICE

• US 20190020761A1
• Filed: 07/14/2017
• Published: 01/17/2019
• Est. Priority Date: 07/14/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Golden, Adam Bradley, Donnelly, William Gerard, Zunger, Yonatan, Kusumo, Leonardo
• Granted Patent: US 11,005,993 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/435 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: H04M 2201/40 using speech recognition speech recognition per se G10L15/00
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00
• CPC Class Codes: H04M 3/4936 Speech interaction details speech recognition per se G10L15/00
• CPC Class Codes: H04M 3/4938 comprising a voice browser which renders and interprets, e.g. VoiceXML

Abstract

An example method includes receiving, by a computational assistant executing at one or more processors of a mobile computing device and via a wireless link between the mobile computing device and an external device, a representation of audio data generated by a microphone of the external device, the audio data representing a spoken utterance detected by the external device; determining, by the computational assistant and based on the audio data, a response to the spoken utterance; and sending, by the mobile computing device, to the external device, and via the wireless link between the mobile computing device and the external device, a command to output, for playback by one or more speakers connected to the external device via a hardwired analog removable connector of the external device or a wireless link between the external device and the one or more speakers, audio data representing the response to the spoken utterance.

Claims

1. A method comprising:

receiving, by a computational assistant executing at one or more processors of a mobile computing device and via a wireless link between the mobile computing device and an external assistant extension device, a representation of audio data generated by one or more microphones of the external assistant extension device, the audio data representing a spoken utterance detected by the external assistant extension device,

wherein the external assistant extension device is connected to a first separate external speaker and is connected to a second speaker,

wherein the external assistant extension device includes a hardwired removable connector, and

wherein the external assistant extension device is connected to the first separate external speaker via a cable that is connected to the hardwired removable connector;

determining, by the computational assistant and based on the audio data, a response to the spoken utterance; and

sending, by the mobile computing device, to the external assistant extension device, and via the wireless link between the mobile computing device and the external assistant extension device, a command to output, for playback, audio data representing the response to the spoken utterance, wherein the command specifies which, of the first separate external speaker and the second external speaker, are to be used by the assistant extension device for playback of the audio data.

2. The method of claim 1, wherein the hardwired removable connector comprises a 3.5 millimeter audio jack.

3. The method of claim 1, further comprising:

transmitting, via the Internet and by the computational assistant, the representation of the audio data to a remote general-purpose assistant system;

receiving, by the computational assistant, data from the remote general-purpose assistant, the date received via the Internet and in response to the transmitting; and

determining the response to the spoken utterance based on the data received from the general-purpose assistant system via the Internet.

4. The method of claim 3, wherein the external assistant extension device does not have an Internet connection.

6. The method of claim 1, wherein the computational assistant is personalized to the user and wherein the response is personalized to the user.

21. The method of claim 1, wherein the assistant extension device lacks an internal speaker.

22. The method of claim 1, wherein the external assistant extension device is connected to the second speaker via a wireless link between the external assistant extension device and the second speaker.

5. (canceled)

7. A method comprising:

determining, by an assistant extension device, that an activation action has occurred;

generating, by one or more microphones of the assistant extension device, audio data representing a spoken utterance detected by the assistant extension device after the activation action occurred;

outputting, by the assistant extension device and via a wireless link between the assistant extension device and an external computing device, a representation of the audio data, wherein outputting the representation of the audio data is in response to the audio data being captured after the activation action occurred;

receiving, by the assistant extension device and via the wireless link between the assistant extension device and the external computing device, a command for the assistant extension device to output, for playback by one or more external speakers connected to the assistant extension device via a hardwired analog removable connector of the assistant extension device or a wireless link between the assistant extension device and the one or more external speakers, audio data representing a response to the spoken utterance; and

outputting, by the assistant extension device and via the hardwired removable connector or the wireless link between the assistant extension device and the one or more external speakers, the audio data representing the response to the spoken utterance.

8. The method of claim 7, wherein the assistant extension device does not have an Internet connection.

9. The method of claim 7, wherein the one or more microphones of the assistant extension device comprise at least one far-field microphone.

10. The method of claim 7, wherein the response to the spoken utterance is generated by a computational assistant executing at least in part at one or more processors of the external computing device.

12. The method of claim 7, wherein determining, by the assistant extension device, that the activation action has occurred, comprises:

generating, by the one or more microphones of the assistant extension device, preceding audio data detected by the assistant extension device before the spoken utterance;

determining whether the preceding audio data includes a trigger phrase; and

determining that the activation action has occurred in response to determining that the preceding audio data includes the trigger phrase.

23. The method of claim 7, wherein the external assistant extension device lacks an internal speaker.

24. The method of claim 7, wherein determining, by the assistant extension device, that the activation action has occurred, comprises:

determining that a speech reception button of the assistant extension device has been activated.

25. The method of claim 7, wherein the one or more external speakers connected to the assistant extension device include both a first external speaker connected via the hardwired removable connector and one or more external speakers connected via the wireless link.

11. (canceled)

13. A mobile computing device comprising:

one or more communication units configured to establish a wireless link between the mobile computing device and an external assistant extension device;

one or more microphones;

one or more speakers; and

one or more processors configured to execute a computational assistant configured to;

receive, via the wireless link between the mobile computing device and the external assistant extension device, a representation of audio data generated by one or more microphones of the external assistant extension device, the one or more microphones of the external assistant extension device including one or more far-field microphones, and the audio data representing a spoken utterance detected by the external assistant extension device,

wherein the external assistant extension device includes a hardwired analog removable connector, and

wherein the external assistant extension device is connected to a separate external speaker via a cable that is connected to the hardwired removable connector;

determine, based on the received audio data, a response to the spoken utterance; and

output, via the wireless link between the mobile computing device and the external assistant extension device, a command to output, for playback by the separate external speaker connected to the external assistant extension device via the hardwired analog removable connector of the external assistant extension device, audio data representing the response to the spoken utterance.

14. The mobile computing device of claim 13, wherein the one or more microphones of the mobile computing device do not include a far-field microphone.

15. The mobile computing device of claim 13, wherein, to determine the response to the spoken utterance, the computational assistant is configured to:

transmit, via connection to the Internet provided by one or more of the communication units, the representation of the audio data to a remote general-purpose assistant system; and

receive, responsive to the transmitting and via connection to the Internet provided by one or more of the communication units, data from the remote general-purpose assistant system.

16. (canceled)

17. An assistant extension device comprising:

one or more communication units configured to establish a wireless link between the assistant extension device and an external computing device at which a computational assistant is executing;

one or more microphones, including at least one far-field microphone;

a hardwired analog removable audio connector; and

one or more processors configured to;

determine that an activation action has occurred;

receive, from the one or more microphones, audio data representing a spoken utterance detected by the assistant extension device after the activation action occurred;

output, via the wireless link between the assistant extension device and the external computing device, a representation of the audio data, wherein outputting the representation of the audio data is in response to the audio data being captured after the activation action occurred;

receive, via the wireless link between the assistant extension device and the external computing device, a command for the assistant extension device to output, for playback by one or more external speakers connected to the device via the hardwired analog removable audio connector of the device, audio data representing a response to the spoken utterance; and

output, via the hardwired removable audio connector, the audio data representing the response to the spoken utterance.

18. The device of claim 17, wherein the hardwired analog removable audio connector comprises a 3.5 millimeter audio jack.

19. (canceled)

20. (canceled)