Voice actions on computing devices

• US 9,239,603 B2
• Filed: 10/28/2010
• Issued: 01/19/2016
• Est. Priority Date: 10/28/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: LeBeau, Michael J., Jitkoff, John Nicholas
• Primary Examiner(s): JACKSON, JAKIEDA R
• Application Number:
  US12/914,965
• Publication Number:
  US 20110106534A1
• Time in Patent Office: 1,909 Days
• Field of Search: 704/275, 704/235, 701/202
• US Class Current: 1/1
• CPC Class Codes: G01C 21/265 constructional aspects of navigation devices, e.g. housings, mountings, displays G01C21/3688 takes precedence
• CPC Class Codes: G01C 21/3608 using speech input, e.g. using speech recognition
• CPC Class Codes: G01C 21/362 received from an external device or application, e.g. PDA, mobile phone or calendar application
• CPC Class Codes: G01C 21/3661 Guidance output on an external device, e.g. car radio
• CPC Class Codes: G01C 21/367 Details, e.g. road map scale, orientation, zooming, illumination, level of detail, scrolling of road map or positioning of current position marker
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1632 External expansion units, e.g. docking stations
• CPC Class Codes: G06F 1/266 Arrangements to supply power to external peripherals either directly from the computer or under computer control, e.g. supply of power through the communication port, computer controlled power-strips
• CPC Class Codes: G06F 1/3265 Power saving in display device
• CPC Class Codes: G06F 1/329 by task scheduling
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06F 16/638 Presentation of query results
• CPC Class Codes: G06F 2200/1614 Image rotation following screen orientation, e.g. switching from landscape to portrait mode
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: H04M 1/04 Supports for telephone transmitters or receivers
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces
• CPC Class Codes: H04M 1/72415 for remote control of appliances
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 1/72457 according to geographic location
• CPC Class Codes: H04M 2250/02 including a Bluetooth interface
• CPC Class Codes: H04M 2250/74 : with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00
• CPC Class Codes: Y02D 10/00 : Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A computer-implemented method includes receiving spoken input at a computing device from a user of the computing device, the spoken input including a carrier phrase and a subject to which the carrier phrase is directed, providing at least a portion of the spoken input to a server system in audio form for speech-to-text conversion by the server system, the portion including the subject to which the carrier phrase is directed, receiving from the server system instructions for automatically performing an operation on the computing device, the operation including an action defined by the carrier phrase using parameters defined by the subject, and automatically performing the operation on the computing device.

Claims

1. A computer-implemented method, comprising:

receiving, by a computing device, spoken input from a user of the computing device, the spoken input including a first portion that indicates an action to be performed with the computing device and a second portion that indicates a subject of the action;

analyzing, by the computing device, audio data that characterizes the spoken input to identify the action that is indicated by the first portion of the spoken input;

selecting first data that is stored on the computing device and that is determined to be usable to generate a parameter for performing the identified action in a manner that is customized to the user;

providing, by the computing device and over a network to a computer system that is separate and remote from the computing device, the first data and at least a portion of the audio data, the portion of the audio data characterizing the second portion of the spoken input that indicates the subject of the action;

receiving, by the computing device and from the computer system, one or more values of the parameter for performing the identified action in a manner that is customized to the user, the one or more values having been generated by the computer system based on the first data and the at least the portion of the audio data that characterizes the second portion of the spoken input; and

in response to receiving the one or more values of the parameter, using the one or more values to perform the identified action on the computing device.

2. The method of claim 1, wherein the identified action comprises an instruction to navigate to a destination using a navigation application, and the subject of the action describes a geographical landmark that is located at the destination.

3. The method of claim 2, wherein the subject of the action describes the geographical landmark without naming the geographical landmark.

4. The method of claim 2, wherein:

the one or more values of the parameter comprise data for generating turn-by-turn navigation directions from a current location of the computing device to the destination, and

using the one or more values of the parameter to perform the identified action comprises displaying the turn-by-turn navigation directions with the navigation application.

5. The method of claim 1, wherein the identified action comprises an instruction to execute an application that plays a media file, and the subject of the action describes the media file to be played.

6. The method of claim 5, wherein the subject of the action includes a name of content in the media file.

7. The method of claim 1, wherein the subject comprises a search query and the identified action comprises an instruction to navigate to a destination using a navigation application, the destination determined based on a first search result that is responsive to the search query.

8. The method of claim 7, wherein:

the one or more values of the parameter comprise location data for the first search result, and

the location data is generated by the computer system based on information obtained from a landing page corresponding to the first search result.

9. The method of claim 8, wherein the information obtained from the landing page corresponding to the first search result comprises an address or a geographic location of an entity that is identified in the first search result.

10. The method of claim 1, wherein the first data comprises (i) data that is obtained from an account of the user or (ii) location information that indicates a current location of the computing device and that is obtained from global positioning system (GPS) data or network location data.

11. The method of claim 1, wherein the one or more values of the parameter for performing the identified action in the manner that is customized to the user comprises data other than a transcription of the portion of the audio data that characterizes the second portion of the spoken input.

12. The method of claim 1, wherein the first data is of a type of data that is mapped to the identified action, wherein the first data is selected based on the identified action from data of a plurality of types that are respectively mapped to different ones of a plurality of actions.

13. The method of claim 1, wherein the one or more values of the parameter are received as having been generated by the computer system based at least on the first data and a speech recognition result of the at least the portion of the audio data that characterizes the second portion of the spoken input.

14. The method of claim 1, wherein at least a portion of the analyzed audio data that characterizes the spoken input comprises hashed audio features.

15. The method of claim 1, wherein:

the first portion of the spoken input comprises a command to perform the action of providing turn-by-turn navigation directions to a destination;

the second portion of the spoken input includes one or more spoken words that describe the destination;

selecting the first data that is stored on the computing device and that is determined to be usable to generate the parameter for performing the identified action in the manner that is customized to the user comprises selecting location information that indicates a current location of the computing device and that is obtained from global positioning system (GPS) data or from network location data;

providing, by the computing device and to the computer system, the first data and the at least the portion of the audio data comprises transmitting to the computer system (i) the location information that indicates the current location of the computing device, and (ii) the at least the portion of the audio data that characterizes the second portion of the spoken input that includes the one or more spoken words that describe the destination; and

the one or more values of the parameter for performing the identified action comprise data that is arranged for use by the computing device to present turn-by-turn navigation directions from the current location of the computing device to the destination.

16. One or more non-transitory computer-readable media having instructions stored thereon that, when executed by one or more processors, cause performance of operations comprising:

receiving, by a computing device, spoken input from a user of the computing device, the spoken input including a first portion that indicates an action to be performed with the computing device and a second portion that indicates a subject of the action;

analyzing, by the computing device, audio data that characterizes the spoken input to identify the action that is indicated by the first portion of the spoken input;

selecting first data that is stored on the computing device and that is determined to be usable to generate a parameter for performing the identified action in a manner that is customized to the user;

providing, by the computing device and over a network to a computer system that is separate and remote from the computing device, the first data and at least a portion of the audio data, the portion of the audio data characterizing the second portion of the spoken input that indicates the subject of the action;

receiving, by the computing device and from the computer system, one or more values of the parameter for performing the identified action in a manner that is customized to the user, the one or more values having been generated by the computer system based on the first data and the at least the portion of the audio data that characterizes the second portion of the spoken input; and

in response to receiving the one or more values of the parameter, using the one or more values to perform the identified action on the computing device.

17. The one or more non-transitory computer-readable media of claim 16, wherein the identified action comprises an instruction to navigate to a destination using a navigation application, and the subject of the action describes a geographical landmark that is located at the destination.

18. The one or more non-transitory computer-readable media of claim 17, wherein the subject of the action describes the geographical landmark without naming the geographical landmark.

19. The one or more non-transitory computer-readable media of claim 17, wherein:

the one or more values of the parameter comprise data for generating turn-by-turn navigation directions from a current location of the computing device to the destination, and

using the one or more values of the parameter to perform the identified action comprises displaying the turn-by-turn navigation directions with the navigation application.

20. The one or more non-transitory computer-readable media of claim 16, wherein the identified action comprises an instruction to execute an application that plays a media file, and the subject of the action describes the media file to be played.

21. The one or more non-transitory computer-readable media of claim 20, wherein the subject of the action includes a name of content in the media file.

22. The one or more non-transitory computer-readable media of claim 16, wherein the subject comprises a search query and the identified action comprises an instruction to navigate to a destination using a navigation application, the destination determined based on a first search result that is responsive to the search query.

23. The one or more non-transitory computer-readable media of claim 22, wherein:

the one or more values of the parameter comprise location data for the first search result, and

the location data is generated by the computer system based on information obtained from a landing page corresponding to the first search result.

24. The one or more non-transitory computer-readable media of claim 23, wherein the information obtained from the landing page corresponding to the first search result comprises a geographic location of an entity that is described by the first search result.

25. The one or more non-transitory computer-readable media of claim 16, wherein the first data comprises (i) data that is obtained from an account of the user or (ii) location information that indicates a current location of the computing device and that is obtained from global positioning system (GPS) data or network location data.

26. The one or more non-transitory computer-readable media of claim 16, wherein the one or more values of the parameter for performing the identified action in the manner that is customized to the user comprises data other than a transcription of the portion of the audio data that characterizes the second portion of the spoken input.

27. The one or more non-transitory computer-readable media of claim 16, wherein the first data is of a type of data that is mapped to the identified action, wherein the first data is selected based on the identified action from data of a plurality of types that are respectively mapped to different ones of a plurality of actions.

28. The one or more non-transitory computer-readable media of claim 16, wherein the one or more values of the parameter are received as having been generated by the computer system based at least on the first data and a speech recognition result of the at least the portion of the audio data that characterizes the second portion of the spoken input.

29. A computer-implemented method, comprising:

receiving, at a computer system and from a computing device that is separate and remote from the computer system, audio data that characterizes spoken input provided by a user of the remote computing device;

receiving, at the computer system and from the computing device, a request to determine one or more values of a parameter for performing a particular action at the computing device in a manner that is customized to the user, the request including second data that (i) identifies the articular action to be performed at the computing device, and that (ii) was generated by performing speech recognition at the computing device on the audio data that characterizes the spoken input provided by the user of the computing device;

performing speech recognition, at the computer system, on the audio data that characterizes the spoken input provided by the user of the computing device, so as to identify a subject of the particular action;

obtaining, at the computer system, user data that is determined to be usable to generate the one or more values of the parameter for performing the particular action at the computing device in the manner that is customized to the user;

determining, by the computer system, the one or more values of the parameter for performing the particular action in the manner that is customized to the user, the one or more values of the parameter determined based on (i) the second data that identifies the particular action to be performed at the computing device, (ii) the identified subject of the particular action, and (iii) the user data; and

transmitting, from the computer system and to the computing device, the one or more values of the parameter for performing the particular action in the manner that is customized to the user.

30. The method of claim 29, wherein the particular action is to navigate to a destination using a navigation application installed on the remote computing device, wherein the subject of the particular action describes a geographical landmark that is located at the destination.