Multi-level voice menu

• US 9,721,567 B2
• Filed: 04/15/2016
• Issued: 08/01/2017
• Est. Priority Date: 01/30/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: LeBeau, Michael J., Nass, Clifford Ivar
• Primary Examiner(s): PULLIAS, JESSE SCOTT
• Application Number:
  US15/130,827
• Publication Number:
  US 20160232896A1
• Time in Patent Office: 473 Days
• Field of Search: 704231-257, 704270-275
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 21/10 Transforming into visible information
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: H04R 1/08 Mouthpieces; Microphones; Attachments therefor

Abstract

Methods, apparatus, and computer-readable media are described herein related to a user interface (UI) that can be implemented on a head-mountable device (HMD). The UI can include a voice-navigable UI. The voice-navigable UI can include a voice navigable menu that includes one or more menu items. The voice-navigable UI can also present a first visible menu that includes at least a portion of the voice navigable menu. In response to a first utterance comprising one of the one or more menu items, the voice-navigable UI can modify the first visible menu to display one or more commands associated with the first menu item. In response to a second utterance comprising a first command, the voice-navigable UI can invoke the first command. In some embodiments, the voice-navigable UI can display a second visible menu, where the first command can be displayed above other menu items in the second visible menu.

Claims

1. A method, comprising:

establishing, by a computing device, a communication session with a remote device in order to provide a voice-navigable user interface, wherein the remote device comprises a display that initially has a first visible menu comprising at least a portion of a voice navigable menu displayed thereon, wherein the voice navigable menu includes one or more menu items, and wherein each menu item comprises a text version of a corresponding speech command from a plurality of speech commands;

receiving a first utterance at the computing device, the first utterance comprising the speech command corresponding to a first menu item from the one or more menu items, wherein the first utterance is captured by at least one microphone at the computing device;

in response to receiving the first utterance corresponding to first menu item, sending an instruction to the remote device to modify the first visible menu to display, on the display of the remote device, one or more commands associated with the first menu item, wherein each displayed command comprises a text version of a corresponding speech command from the plurality of speech commands;

receiving a second utterance at the computing device, the second utterance comprising the speech command corresponding to a first command from the one or more commands, wherein the second utterance is captured by at least one microphone at the computing device; and

in response to receiving the second utterance, initiating a process to invoke the first command.

2. The method of claim 1, wherein initiating the process to invoke the first command comprises sending an instruction to the remote device to display at least a portion of a second visible menu, wherein the second visible menu includes the first command and at least one menu item of the one or more menu items, wherein each displayed menu item comprises a text version of a corresponding speech command from a plurality of speech commands, and wherein the second visible menu displays the first command above the at least one menu item.

3. The method of claim 2, further comprising:

receiving a third utterance at the computing device, the third utterance comprising the speech command corresponding to a second command from the one or more commands, the second command differing from the first command;

after receiving the third utterance, initiating a process to invoke the second command; and

sending an instruction to the remote device to display at least a portion of a third visible menu, wherein the third visible menu includes the second command and at least one menu item of the one or more menu items, and wherein the third visible menu displays the second command above the at least one menu item.

4. The method of claim 3,

wherein the third visible menu further includes the first command;

and

wherein an order in which the first command and the second command are displayed in the third visible menu is determined based on at least a frequency of use for the first command and a frequency of use for the second command.

5. The method of claim 3,

wherein the third visible menu further includes the first command;

and

wherein an order in which the first command and the second command are displayed in the third visible menu is determined based on at least a recency of use for the first command and a recency of use for the second command.

6. The method of claim 3,

wherein the third visible menu further includes the first command;

and

wherein an order in which the first command and the second command are displayed in the third visible menu is determined based on at least a combination of a frequency of use and a recency of use for the first command and a combination of a frequency of use and a recency of use for the second command.

7. The method of claim 1, further comprising, after establishing the communication session and before the remote device display the first visible menu:

initially receiving a hotword utterance comprising a hotword;

in response to receiving the hotword utterance, sending an instruction to the remote device to display at least a portion of the first visible menu.

8. The method of claim 1, wherein the first menu item comprises an identification of a category.

9. The method of claim 1, wherein at least one menu item of the one or more menu items comprises an identification of a category.

10. The method of claim 1, wherein the instruction to modify the first visible menu causes an expansion of the first visible menu to display the one or more commands in-line with the first menu item.

11. The method of claim 1, wherein the computing device comprises a wearable computing device.

12. A wearable computing device comprising:

at least one microphone arranged on the wearable computing device;

a processor; and

a non-transitory computer-readable medium configured to store at least program instructions that, when executed by the processor, cause the wearable to carry out functions comprising;

establishing a communication session between the wearable computing device and a remote device in order to provide a voice-navigable user interface, wherein the remote device comprises a display that initially has a first visible menu comprising at least a portion of a voice navigable menu displayed thereon, wherein the voice navigable menu includes one or more menu items, and wherein each menu item comprises a text version of a corresponding speech command from a plurality of speech commands;

receiving a first utterance via the at least one microphone, the first utterance comprising the speech command corresponding to a first menu item from the one or more menu items;

in response to receiving the first utterance corresponding to first menu item, sending an instruction to the remote device to modify the first visible menu to display, on the display of the remote device, one or more commands associated with the first menu item, wherein each displayed command comprises a text version of a corresponding speech command from the plurality of speech commands;

receiving a second utterance via the at least one microphone, the second utterance comprising the speech command corresponding to a first command from the one or more commands; and

in response to receiving the second utterance, initiating a process to invoke the first command.

13. A method, comprising:

establishing, by a remote device, a communication session with a client computing device to provide a voice-navigable user interface via at least one microphone of the client computing device, wherein the remote device comprises a display;

initially displaying, by the remote device, a first visible menu comprising at least a portion of a voice navigable menu, wherein the voice navigable menu includes one or more menu items, and wherein each menu item comprises a text version of a corresponding speech command from a plurality of speech commands;

receiving, by the remote device an indication of a first utterance captured by at least one microphone of the client computing device, the first utterance comprising the speech command corresponding to a first menu item from the one or more menu items;

in response to receiving the indication of the first utterance, modifying the first visible menu to display one or more commands associated with the first menu item, wherein each displayed command comprises a text version of a corresponding speech command from the plurality of speech commands;

receiving, by the remote device, an indication of a second utterance captured by the at least one microphone of the client computing device, the second utterance comprising the speech command corresponding to a first command from the one or more commands; and

in response to receiving the second utterance, invoking the first command.

14. The method of claim 13, further comprising:

displaying at least a portion of a second visible menu, wherein the second visible menu includes the first command and at least one menu item of the one or more menu items, wherein each menu item comprises a text version of a corresponding speech command from a plurality of speech commands; and

wherein the second visible menu displays the first command above the at least one menu item.

15. The method of claim 14, further comprising:

receiving, by the remote device, an indication of a third utterance captured by the at one microphone of the client computing device, the third utterance comprising the speech command corresponding to a second command from the one or more commands, the second command differing from the first command;

after receiving the third utterance, invoking the second command; and

displaying at least a portion of a third visible menu, wherein the third visible menu includes the second command and at least one menu item of the one or more menu items, and wherein the third visible menu displays the second command above the at least one menu item.

16. The method of claim 15,

wherein the third visible menu further includes the first command;

and

wherein an order in which the first command and the second command are displayed in the third visible menu is determined based on at least a frequency of use for the first command and a frequency of use for the second command.

17. The method of claim 13, further comprising, before displaying the first visible menu:

receiving an indication of a hotword utterance captured by the at least one microphone of the client computing device, wherein the hotword utterance comprises a hotword; and

initiating the displaying of at least a portion of the first visible menu in response to receipt of the indication of the hotword utterance.

18. The method of claim 13, wherein modifying the first visible menu to display the one or more commands associated with the first menu item comprises expanding the first visible menu to display the one or more commands in-line with the first menu item.

19. A method, comprising:

visually displaying, on a display of a computing device having a voice-navigable user interface, at least a portion of a voice navigable menu, wherein the voice navigable menu includes one or more menu items, and wherein each menu item comprises a text version of a corresponding speech command from a plurality of speech commands;

receiving, by the computing device, a first utterance comprising the speech command corresponding to a first menu item from the one or more menu items, wherein the first utterance is captured by a microphone of the computing device;

in response to receiving the first utterance, modifying, by the computing device, displaying one or more commands associated with the first menu item, wherein each displayed command comprises a text version of a corresponding speech command from the plurality of speech commands;

receiving a second utterance at the computing device, the second utterance comprising the speech command corresponding to a first command from the one or more commands, wherein the first utterance is captured by a microphone of the computing device; and

in response to receiving the second utterance, invoking the first command.

20. The method of claim 19, wherein the computing device is a wearable computing device.