Implementations for voice assistant on devices

• US 10,535,343 B2
• Filed: 05/10/2017
• Issued: 01/14/2020
• Est. Priority Date: 05/10/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Mixter, Kenneth
• Primary Examiner(s): Roberts, Shaun
• Application Number:
  US15/592,137
• Publication Number:
  US 20170329573A1
• Time in Patent Office: 979 Days
• Field of Search: 704231, 704270, 704275
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/28 Constructional details of speech recognition systems
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04M 1/271 controlled by voice recognition
• CPC Class Codes: H04M 3/493 Interactive information services, e.g. directory enquiries ; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals
• CPC Class Codes: H04N 21/20 Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
• CPC Class Codes: H04N 21/2393 involving handling client requests scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/40 Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
• CPC Class Codes: H04N 21/4104 Peripherals receiving signals from specially adapted client devices
• CPC Class Codes: H04N 21/4112 having fewer capabilities than the client, e.g. thin client having less processing power or no tuning capabilities
• CPC Class Codes: H04N 21/4147 PVR [Personal Video Recorder] H04N5/76 takes precedence ; arrangements for broadcast with accumulation-type receivers H04H20/40
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/42206 characterized by hardware details
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/4751 for defining user accounts, e.g. accounts for children

Abstract

A method at an electronic device with an audio input system includes: receiving a verbal input at the device; processing the verbal input; transmitting a request to a remote system, the request including information determined based on the verbal input; receiving a response to the request, wherein the response is generated by the remote system in accordance with the information based on the verbal input; and performing an operation in accordance with the response, where one or more of the receiving, processing, transmitting, receiving, and performing are performed by one or more voice processing modules of a voice assistant library executing on the electronic device, the voice processing modules providing a plurality of voice processing operations that are accessible to one or more application programs and/or operating software executing or executable on the electronic device.

Claims

1. A method, comprising:

at an electronic device having a first device type and comprising an audio input system, one or more processors, and memory storing one or more programs for execution by the one or more processors;

downloading a device-agnostic voice assistant library configured to execute across a plurality of different electronic device types, including the first device type, wherein the voice-assistant library includes a plurality of voice processing modules, each of the voice processing modules providing one or more voice processing operations that are accessible to application programs executing or executable on the different electronic device types;

configuring the device-agnostic voice assistant library to execute on the electronic device based on the electronic device having the first device type, including;

selecting an implementation for the voice assistant library based on the electronic device having the first device type, wherein the implementation for the voice assistant library is selected from a group consisting of;

in an application installed on the electronic device, in an operating system of the electronic device, and in firmware of the electronic device;

after the configuring, receiving, via a microphone of the audio input system, a verbal input from a user;

extracting request information from the verbal input by processing the verbal input using the device-agnostic voice assistant library executing on the electronic device;

transmitting a request to a remote system, the request including the extracted request information;

receiving a response to the request, wherein the response is generated by the remote system in accordance with the extracted request information; and

performing an operation in accordance with the response by one or more voice processing modules of the configured voice assistant library.

2. The method of claim 1, wherein at least some voice processing operations associated with the voice processing modules are performed on the remote system, which is interconnected with the electronic device via a wide area network.

3. The method of claim 1, wherein processing the verbal input comprises performing speech processing on the verbal input, and the speech processing is performed by a module of the voice processing modules of the voice assistant library.

4. The method of claim 1, wherein processing the verbal input comprises performing audio input processing on audio data of the verbal input, and the audio input processing is performed by a module of the voice processing modules of the voice assistant library.

5. The method of claim 1, wherein performing an operation in accordance with the response comprises decoding audio, and the audio decoding is performed by a module of the voice processing modules of the voice assistant library.

6. The method of claim 1, further comprising:

at a second electronic device having a second device type, distinct from the first device type;

downloading the device-agnostic voice assistant library;

configuring the device-agnostic voice assistant library to execute on the second electronic device based on the second electronic device having the second device type;

after the configuring the device-agnostic voice assistant library, receiving a second verbal input from a second user; and

performing a second operation at the second electronic device in response to the second verbal input by one or more voice processing modules of the configured voice assistant library.

7. The method of claim 1, wherein performing the operation comprises outputting an audible response to the user via the audio input system.

8. The method of claim 1, wherein configuring the device-agnostic voice assistant library includes enabling a voice assistant functionality on the electronic device.

9. A device-agnostic voice assistant library for electronic devices that include respective audio input systems, comprising:

one or more implementation modules configured to implement the voice assistant library across each of a plurality of different electronic devices based on a corresponding device type, wherein the implementation for the voice assistant library is selected from a group consisting of;

in an application installed on the electronic device, in an operating system of the electronic device, and in firmware of the electronic device;

a plurality of voice processing modules, each of the voice processing modules providing one or more voice processing operations that are accessible to application programs executing or executable on the different electronic device types; and

one or more application programming interfaces (APIs) configured to provide interfaces between the plurality of voice processing operations and hardware and/or software of the electronic devices;

whereby the one or more voice processing modules and APIs enable portability across the plurality of different electronic device types of voice-enabled applications configured to interact with one or more of the voice processing operations.

10. The voice assistant library of claim 9, wherein at least some voice processing operations associated with the voice processing modules are performed on a backend server interconnected with the electronic devices via a wide area network.

11. The voice assistant library of claim 10, wherein the voice processing operations include device-specific operations configured to control devices coupled with the electronic devices.

12. The voice assistant library of claim 10, wherein the voice processing operations include information and media request operations configured to provide requested information and/or media content to users of the respective electronic devices, or on devices coupled with the respective electronic devices.

13. The voice assistant library of claim 9, wherein the plurality of voice processing operations comprises hotword detection.

14. The voice assistant library of claim 9, wherein the plurality of voice processing operations comprises speech processing.

15. The voice assistant library of claim 9, wherein the plurality of voice processing operations comprises audio input processing.

16. An electronic device having a first device type, comprising:

an audio input system;

one or more processors; and

memory storing one or more programs to be executed by the one or more processors, the one or more programs comprising instructions for;

downloading a device-agnostic voice assistant library configured to execute across a plurality of different electronic device types, including the first device type, wherein the voice-assistant library includes a plurality of voice processing modules, each of the voice processing modules providing one or more voice processing operations that are accessible to application programs executing or executable on the different electronic device types;

configuring the device-agnostic voice assistant library to execute on the electronic device based on the electronic device having the first device type of the plurality of different electronic device types, including;

selecting an implementation for the voice assistant library based on the electronic device having the first device type, wherein the implementation for the voice assistant library is selected from a group consisting of;

in an application installed on the electronic device, in an operating system of the electronic device, and in firmware of the electronic device;

after the configuring, receiving, via a microphone of the audio input system, a verbal input from a user;

extracting request information from the verbal input by processing the verbal input using the device-agnostic voice assistant library executing on the electronic device;

transmitting a request to a remote system, the request including the extracted request information;

receiving a response to the request, wherein the response is generated by the remote system in accordance with the extracted request information; and

performing an operation in accordance with the response by one or more voice processing modules of the configured voice assistant library.

17. The device of claim 16, comprising instructions for performing hotword detection on the verbal input, wherein the instructions for performing hotword detection on the verbal input are included in a module of the voice processing modules of the voice assistant library.

18. The device of claim 16, comprising instructions for performing speech processing on the verbal input, wherein the instructions for performing speech processing on the verbal input are included in a module of the voice processing modules of the voice assistant library.

19. The device of claim 16, comprising instructions for performing audio input processing on audio data of the verbal input, wherein the instructions for performing audio input processing on audio data of the verbal input are included in a module of the voice processing modules of the voice assistant library.

20. The device of claim 16, wherein the plurality of voice processing modules includes a ducking module, the ducking module providing a ducking operation that is accessible to application programs executing or executable on the different electronic device types, the ducking operation including:

while the electronic device is producing an audible output, receiving an activation input to the electronic device indicating that a user is about to submit verbal input to the electronic device; and

in response to the activation input, adjusting by the electronic device the audible output from the first volume level to a second volume level, less than the first volume level.