Layered mobile application user interfaces

• US 9,135,914 B1
• Filed: 09/15/2012
• Issued: 09/15/2015
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Bringert, Bjorn Erik, Pietryka, Pawel, Hodgson, Peter John, Tickner, Simon, Penha, Henrique, Cohen, Richard Zarek, Zanolin, Luca
• Primary Examiner(s): Dorvil, Richemond
• Assistant Examiner(s): Ortiz Sanchez, Michael
• Application Number:
  US13/621,094
• Time in Patent Office: 1,095 Days
• Field of Search: 715/716, 715/727
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

Disclosed are systems, methods, and devices for providing a layered user interface for one or more applications. A user-interface layer for a voice user interface is generated. The user-interface layer can be based on a markup-language-structured user-interface description for an application configured to execute on a computing device. The user-interface layer can include a command display of one or more voice-accessible commands for the application. The computing device can display at least the user-interface layer of the voice user interface. The computing device can receive an input utterance, obtain input text based upon speech recognition performed upon the input utterance, and determine that the input text corresponds to a voice-accessible command displayed as part of the command display. The computing device can execute the application to perform the command.

Claims

1. A method, comprising:

displaying, by a computing device, a display for a voice user interface to an application, the display for the voice user interface comprising a command display of one or more voice accessible commands for the application, a display for a non-voice user interface to the application, and an action bar, wherein the action bar is configured to separate the display for the voice user interface and the display for the non-voice user interface;

receiving, by the computing device, an input utterance;

obtaining, by the computing device, input text based upon speech recognition performed upon the input utterance;

determining, by the computing device, that the input text corresponds to a voice-accessible command displayed as part of the command display;

executing, by the computing device, the application to perform the command;

receiving, by the computing device, input associated with a motion on a touch screen of the computing device, where the motion starts at a region of the touch screen that corresponds to the action bar displayed on the display; and

in response to the input, switching between the voice user interface and the non-voice user interface by the computing device.

2. The method of claim 1, wherein the application is configured to execute a set of commands that are accessible via user input, and wherein the one or more voice-accessible commands displayed as part of the command display are a subset of the set of commands.

3. The method of claim 1, wherein the voice user interface comprises at least one visual feature different from the non-voice user interface, and wherein the at least one visual feature comprises a font, a color, a recognition-feedback region, or command display.

4. The method of claim 1, wherein the application is configured to use the voice user interface, wherein the motion on the touch screen that starts at a region of the touch screen that corresponds to the action bar displayed on the display comprises:

a motion on the touch screen starting at the region of the touch screen that corresponds to the action bar in a direction associated with the non-voice user interface; and

wherein switching between the voice user interface and the non-voice user interface comprises;

in response to the motion on the touch screen in the direction associated with the non-voice user interface, switching to use the non-voice user interface.

5. The method of claim 1, wherein the application is configured to use the non-voice user interface, wherein the motion on the touch screen that starts at a region of the touch screen that corresponds to the action bar displayed on the display comprises:

a motion on the touch screen starting at the region of the touch screen that corresponds to the action bar in a direction associated with the voice interface; and

wherein switching between the voice user interface and the non-voice user interface comprises;

in response to the motion on the touch screen in the direction associated with the voice interface, switching to use the voice user interface.

6. A computing device, comprising:

a display;

a touch screen;

at least one input device;

a processor; and

memory having one or more instructions that, upon execution by the processor, cause the computing device to perform functions, the functions comprising;

displaying, on the display, a display for a voice user interface to an application, the display for the voice user interface comprising a command display of one or more voice accessible commands for the application, a display for a non-voice user interface to the application, and an action bar, wherein the action bar is configured to separate the display for the voice user interface and the display for the non-voice user interface;

receiving an input utterance via the at least one input device;

obtaining input text based upon speech recognition performed upon the input utterance;

determining that the input text corresponds to a voice-accessible command displayed as part of the command display;

executing the application to perform the command;

receiving input associated with a motion on the touch screen that starts at a region of the touch screen that corresponds to the action bar displayed on the display; and

in response to the input, switching between the voice user interface and the non-voice user interface.

7. The computing device of claim 6, wherein the application is configured to execute a set of commands that are accessible via user input, and wherein the one or more voice-accessible commands displayed as part of the command display are a subset of the set of commands.

8. The computing device of claim 6, wherein the voice user interface comprises at least one visual feature different from the non-voice user interface, and wherein the at least one visual feature comprises a font, a color, a recognition-feedback region, or command display.

9. The computing device of claim 6, wherein the application is configured to receive input using the voice user interface, wherein the motion on the touch screen that starts at a region of the touch screen that corresponds to the action bar displayed on the display comprises:

a motion on the touch screen in a direction associated with the non-voice user interface starting at the region of the touch screen that corresponds to the action bar; and

wherein switching between the voice user interface and the non-voice user interface comprises;

in response to the motion on the touch screen in the direction associated with the non-voice user interface, switching to receive input using the non-voice user interface.

10. The computing device of claim 6, wherein the application is configured to receive input using the non-voice user interface, wherein the motion on the touch screen that starts at a region of the touch screen that corresponds to the action bar displayed on the display comprises:

a motion on the touch screen in a direction associated with the voice interface starting at the region of the touch screen that corresponds to the action bar; and

wherein switching between the voice user interface and the non-voice user interface comprises;

in response to the motion on the touch screen in the direction associated with the voice interface, switching to receive input using the voice user interface.

11. An article of manufacture including a non-transitory computer-readable storage medium having instructions stored thereon that, upon execution by a computing device comprising a display, a touch screen, and at least one input device, cause the computing device to perform functions, the functions comprising:

displaying, on the display, a display for a voice user interface to an application, the display for the voice user interface comprising a command display of one or more voice accessible commands for the application, a display for a non-voice user interface to the application, and an action bar, wherein the action bar is configured to separate the display for the voice user interface and the display for the non-voice user interface, and wherein the application is configured to use both the voice user interface and the non-voice user interface;

receiving an input utterance via the at least one input device;

obtaining input text based upon speech recognition performed upon the input utterance;

determining that the input text corresponds to a voice-accessible command displayed as part of the command display;

executing the application to perform the command;

receiving input associated with a motion on the touch screen that starts at a region of the touch screen that corresponds to the action bar displayed on the display; and

in response to the input, switching between the voice user interface and the non-voice user interface.

12. The article of manufacture of claim 11, wherein the application is configured to execute a set of commands that are accessible via user input, and wherein the one or more voice-accessible commands displayed as part of the command display are a subset of the set of commands.

13. The article of manufacture of claim 11, wherein the voice user interface comprises at least one visual feature different from the non-voice user interface, and wherein the at least one visual feature comprises a font, a color, a recognition-feedback region, or command display.

14. The article of manufacture of claim 11, wherein the application is configured to receive input using the voice user interface, wherein the motion on the touch screen that starts at a region of the touch screen that corresponds to the action bar displayed on the display comprises:

a motion on the touch screen in a direction associated with the non-voice user interface starting at the region of the touch screen that corresponds to the action bar; and

wherein switching between the voice user interface and the non-voice user interface comprises;

in response to the motion on the touch screen in the direction associated with the non-voice user interface, switching to receive input using the non-voice user interface.

15. The article of manufacture of claim 11, wherein the application is configured to receive input using the non-voice user interface, wherein the motion on the touch screen that starts at a region of the touch screen that corresponds to the action bar displayed on the display comprises:

a motion on the touch screen in a direction associated with the voice interface starting at the region of the touch screen that corresponds to the action bar; and

wherein switching between the voice user interface and the non-voice user interface comprises;

in response to the motion on the touch screen in the direction associated with the voice interface, switching to receive input using the voice user interface.