Multi-modal command display

• US 9,383,827 B1
• Filed: 04/07/2014
• Issued: 07/05/2016
• Est. Priority Date: 04/07/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Karlo, Thomas Ramon, Faaborg, Alexander, Shiplacoff, Daniel Marc Gatan
• Primary Examiner(s): Polo, Gustavo
• Application Number:
  US14/246,458
• Time in Patent Office: 820 Days
• Field of Search: 345/168
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/023 Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G11B 27/00 Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel

Abstract

A computing device includes at least one processor and at least one module, operable by the at least one processor to receive indications of user input comprising a selection of at least one key of a keyboard and at least one other user input, determine, based at least in part on the indications of user input, that the user input collectively indicates a command for execution of at least one operation, and, responsive to determining that the user input collectively indicates the command, determine that the command is alternatively indicated by a particular audio input. The at least one module may be further operable, responsive to determining that the command is alternatively indicated by the particular audio input, to output, for display, a visual representation of the particular audio input, and execute, based at least in part on the command, the at least one operation.

Claims

1. A method comprising:

receiving, by a computing device, a plurality of indications of user input comprising a selection of at least one key of a keyboard and at least one other user input;

after receiving the plurality of indications of user input, determining, by the computing device and based at least in part on the plurality of indications of user input, that the selection of the at least one key of the keyboard and the at least one other user input collectively indicate a command for execution of at least one operation;

responsive to determining that the selection of the at least one key and the at least one other user input collectively indicate the command, determining, by the computing device, that the command is alternatively indicated by a particular audio input;

responsive to determining that the command is alternatively indicated by the audio input, outputting, by the computing device and for display, a visual representation of the particular audio input; and

executing, by the computing device and based at least in part on the command, the at least one operation.

2. The method of claim 1, further comprising:

responsive to determining that the at least one selection of the key and the at least one other user input collectively indicate the command, determining a context of the computing device;

determining, based at least in part on the context of the computing device, at least one related command, the at least one related command indicated by a particular related audio input, wherein the at least one related command is related to the command based on the context; and

responsive to determining the at least one related command, outputting, by the computing device and for display, a visual representation of the particular related audio input.

3. The method of claim 2, wherein determining the context of the computing device comprises determining at least one of:

an application module installed at the computing device, an application module currently executing at the computing device, a currently focused application module, a previous action performed by the computing device, data that indicates a user of the computing device, data that indicates a current focus of the user of the computing device, or a configuration option of the computing device.

4. The method of claim 1, further comprising:

responsive to determining that the command is alternatively indicated by the particular audio input, outputting, by the computing device, an audio representation of the particular audio input.

5. The method of claim 1, wherein the plurality of indications of user input comprises a plurality of indications of first user input, the method further comprising:

responsive to outputting the visual representation of the particular audio input, receiving, by the computing device, an indication of second user input comprising the particular audio input;

responsive to receiving the indication of second user input, receiving, by the computing device, a plurality of indications of third user input comprising the selection of the at least one key of the keyboard and the at least one other user input; and

responsive to receiving the plurality of indications of third user input, determining, based at least in part on the indication of second user input, to refrain from outputting the visual representation of the particular audio input.

6. The method of claim 1, further comprising

outputting, by the computing device and for display, a graphical user interface comprising a plurality of partitions, each respective partition of the plurality of partitions comprising a non-overlapping region of the graphical user interface,

wherein outputting the visual representation of the particular audio input comprises:

determining a currently active partition from the plurality of partitions; and

outputting the visual representation of the particular audio input at a partition from the plurality of partitions other than the currently active partition.

7. The method of claim 1, wherein determining that the selection of the at least one key of the keyboard and the at least one other user input collectively indicate the command for execution of the at least one operation comprises at least one of:

determining, by an operating system of the computing device, that the selection of the at least one key of the keyboard and the at least one other user input collectively indicate a system level command for execution of at least one operation of the operating system;

or

determining, by an application module, that the selection of the at least one key of the keyboard and the at least one other user input collectively indicate an application level command for execution of at least one operation of the application module.

8. The method of claim 1, wherein the at least one other user input comprises a selection of a second key of the keyboard.

9. A computing device comprising:

at least one processor; and

at least one module operable by the at least one processor to;

receive a plurality of indications of user input comprising a selection of at least one key of a keyboard and at least one other user input;

after receiving the plurality of indications of user input, determine, based at least in part on the plurality of indications of user input, that the selection of the at least one key of the keyboard and the at least one other user input collectively indicate a command for execution of at least one operation;

responsive to determining that the selection of the at least one key and the at least one other user input collectively indicate the command, determine that the command is alternatively indicated by a particular audio input;

responsive to determining that the command is alternatively indicated by the particular audio input, output, for display, a visual representation of the particular audio input; and

execute, based at least in part on the command, the at least one operation.

10. The computing device of claim 9, wherein the at least one module is further operable by the at least one processor to:

responsive to determining that the at least one selection of the key and the at least one other user input collectively indicate the command, determine a context of the computing device;

determine, based at least in part on the context of the computing device, at least one related command, the at least one related command indicated by a particular related audio input, wherein the at least one related command is related to the command based on the context; and

responsive to determining the at least one related command, output, for display, a visual representation of the particular related audio input.

11. The computing device of claim 10, wherein the at least one module operable to determine the context of the computing device is operable to determine at least one of:

an application module installed at the computing device, an application module currently executing at the computing device, a currently focused application module, a previous action performed by the computing device, data that indicates a user of the computing device, data that indicates a current focus of the user of the computing device, or a configuration option of the computing device.

12. The computing device of claim 9, wherein the at least one module is further operable by the at least one processor to:

responsive to determining that the command is alternatively indicated by the particular audio input, output an audio representation of the particular audio input.

13. The computing device of claim 9, wherein the plurality of indications of user input comprises a plurality of indications of first user input, and wherein the at least one module is further operable by the at least one processor to:

responsive to outputting the visual representation of the particular audio input, receive an indication of second user input comprising the particular audio input;

responsive to receiving the indication of second user input, receive a plurality of indications of third user input comprising the selection of the at least one key of the keyboard and the at least one other user input; and

responsive to receiving the plurality of indications of third user input, determine, based at least in part on the indication of second user input, to refrain from outputting the visual representation of the particular audio input.

14. The computing device of claim 9, wherein the at least one module is further operable by the at least one processor to output, for display, a graphical user interface comprising a plurality of partitions, each respective partition of the plurality of partitions comprising a non-overlapping region of the graphical user interface, and wherein the at least one module operable to output the visual representation of the particular audio input is operable to:

determine a currently active partition from the plurality of partitions; and

output the visual representation of the particular audio input at a partition from the plurality of partitions other than the currently active partition.

15. The computing device of claim 9, wherein the at least one module operable to determine that the selection of the at least one key of the keyboard and the at least one other user input collectively indicate the command for execution of the at least one operation comprises at least one of:

an operating system of the computing device, wherein the operating system is operable to determine that the selection of the at least one key of the keyboard and the at least one other user input collectively indicate a system level command for execution of at least one operation of the operating system;

or

an application module of the computing device, wherein the application module is operable to determine that the selection of the at least one key of the keyboard and the at least one other user input collectively indicate an application level command for execution of at least one operation of the application module.

16. The computing device of claim 9, wherein the at least one other user input comprises a selection of a second key of the keyboard.

17. A non-transitory computer-readable storage medium encoded with instructions that, when executed, cause at least one processor to:

receive a plurality of indications of user input comprising a selection of at least one key of a keyboard and at least one other user input;

after receiving the plurality of indications of user input, determine, based at least in part on the plurality of indications of user input, that the selection of the at least one key of the keyboard and the at least one other user input collectively indicate a command for execution of at least one operation;

responsive to determining that the selection of the at least one key and the at least one other user input collectively indicate the command, determine that the command is alternatively indicated by a particular audio input;

responsive to determining that the command is alternatively indicated by the particular audio input, output, for display, a visual representation of the particular audio input; and

execute, based at least in part on the command, the at least one operation.

18. The non-transitory computer-readable storage medium of claim 17, wherein the at least one processor comprises at least one processor of a computing device, the computer-readable storage medium being further encoded with instructions that, when executed, cause the at least one processor to:

responsive to determining that the at least one selection of the key and the at least one other user input collectively indicate the command, determine a context of the computing device;

determine, based at least in part on the context of the computing device, at least one related command, the at least one related command indicated by a particular related audio input, wherein the at least one related command is related to the command based on the context; and

responsive to determining the at least one related command, output, by the computing device and for display, a visual representation of the particular related audio input.

19. The non-transitory computer-readable storage medium of claim 18, wherein the instructions that cause the at least one processor to determine the context of the computing device comprise instructions that cause the at least one processor to determine at least one of:

an application module installed at the computing device, an application module currently executing at the computing device, a currently focused application module, a previous action performed by the computing device, data that indicates a user of the computing device, data that indicates a current focus of the user of the computing device, or a configuration option of the computing device.

20. The non-transitory computer-readable storage medium of claim 18, wherein the plurality of indications of user input comprises a plurality of indications of first user input, the computer-readable storage medium being further encoded with instructions that, when executed, cause the at least one processor to:

responsive to outputting the visual representation of the particular audio input, receive an indication of second user input comprising the particular audio input;

responsive to receiving the indication of second user input, receive a plurality of indications of third user input comprising the selection of the at least one key of the keyboard and the at least one other user input; and

responsive to receiving the plurality of indications of third user input, determine, based at least in part on the indication of second user input, to refrain from outputting the visual representation of the particular audio input.