System and method for speech-based navigation and interaction with a device's visible screen elements using a corresponding view hierarchy

• US 10,528,320 B2
• Filed: 02/03/2017
• Issued: 01/07/2020
• Est. Priority Date: 11/21/2013
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Inventors: P. V., Sanjeev Kumar, Agrawal, Amit K., Rout, Satyabrata, Patil, Vishal S.
• Primary Examiner(s): Opsasnick, Michael N
• Application Number:
  US15/424,324
• Publication Number:
  US 20170206060A1
• Time in Patent Office: 1,068 Days
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for enabling screen-specific user interfacing with elements of viewable screens presented by an electronic device are disclosed. In one aspect, a method includes the actions of identifying a character sequence representing a first input that is received while displaying a viewable screen having at least one selectable viewable element. The actions further include accessing an electronic file that provides a text representation of one or more of the at least one selectable viewable element. The actions further include comparing the character sequence to the text representation. The actions further include selecting, within the viewable screen, a selectable viewable element whose text representation matches the character sequence. The actions further include triggering any action linked to the selecting the selectable viewable element.

Claims

1. A computer-implemented method comprising:

providing, for display, a first user interface that includes a first user interface element that corresponds to a text string and a second, different user interface element that corresponds to the text string;

receiving audio data of an utterance that includes the text string;

providing, for display, a second user interface that indicates that both the first user interface element and the second, different user interface element correspond to the utterance;

receiving data indicating a selection of a particular one of the first user interface element or the second, different user interface element; and

in response to receiving the data indicating the selection of the particular one of the first user interface element or the second, different user interface element;

performing an action corresponding to the particular one of the first user interface element or the second, different user interface element; and

storing data indicating to perform the action corresponding to the particular one of the first user interface element or the second, different user interface element (i) in response to receiving additional audio data of an additional utterance that includes the text string, (ii) without providing, for display, the second user interface, and (iii) without receiving data indicating an additional selection of the particular one of the first user interface element or the second, different user interface element.

2. The method of claim 1, wherein receiving data indicating a selection of the particular one of the first user interface element or the second, different user interface element comprises:

receiving touch input indicating the selection of the particular one of the first user interface element or the second, different user interface element.

3. The method of claim 1, wherein receiving data indicating a selection of the particular one of the first user interface element or the second, different user interface element comprises:

receiving other audio data corresponding to an additional utterance that indicates the selection of the particular one of the first user interface element or the second, different user interface element.

4. The method of claim 1, wherein the utterance only includes the text string.

5. The method of claim 1, comprising:

providing, for display, a third user interface that includes the first user interface element and the second, different user interface element;

receiving other audio data corresponding to another utterance that includes the text string; and

in response to receiving the other audio data, performing the action corresponding to the particular one of the first user interface element or the second, different user interface element.

6. The method of claim 5, wherein the other utterance only includes the text string.

7. The method of claim 1, comprising:

providing, for display, a third user interface that includes the first user interface element and the second, different user interface element;

receiving other audio data corresponding to another utterance that includes the text string;

providing, for display, a fourth user interface that indicates that both the first user interface element and the second, different user interface element correspond to the additional utterance; and

after a particular amount of time has elapsed;

receiving additional data indicating a selection of the first user interface element or the second, different user interface element; and

providing, for display, the third user interface.

8. The method of claim 1, wherein the first user interface element is an icon that corresponds to an action of an application and the second, different user interface element is user inputted text.

9. The method of claim 1, wherein the first user interface element and the second, different user interface element are auto completion entries in a drop-down auto completion menu.

10. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

providing, for display, a first user interface that includes a first user interface element that corresponds to a text string and a second, different user interface element that corresponds to the text string;

receiving audio data of an utterance that includes the text string;

providing, for display, a second user interface that indicates that both the first user interface element and the second, different user interface element correspond to the utterance;

receiving data indicating a selection of a particular one of the first user interface element or the second, different user interface element; and

in response to receiving the data indicating the selection of the particular one of the first user interface element or the second, different user interface element;

performing an action corresponding to the particular one of the first user interface element or the second, different user interface element; and

storing data indicating to perform the action corresponding to the particular one of the first user interface element or the second, different user interface element (i) in response to receiving additional audio data of an additional utterance that includes the text string, (ii) without providing, for display, the second user interface, and (iii) without receiving data indicating an additional selection of the particular one of the first user interface element or the second, different user interface element.

11. The system of claim 10, wherein receiving data indicating a selection of the particular one of the first user interface element or the second, different user interface element comprises:

receiving touch input indicating the selection of the particular one of the first user interface element or the second, different user interface element.

12. The system of claim 10, wherein receiving data indicating a selection of the particular one of the first user interface element or the second, different user interface element comprises:

receiving other audio data corresponding to an additional utterance that indicates the selection of the particular one of the first user interface element or the second, different user interface element.

13. The system of claim 10, wherein the utterance only includes the text string.

14. The system of claim 10, wherein the operations further comprise:

providing, for display, a third user interface that includes the first user interface element and the second, different user interface element;

receiving other audio data corresponding to another utterance that includes the text string; and

in response to receiving the other audio data, performing the action corresponding to the particular one of the first user interface element or the second, different user interface element.

15. The system of claim 14, wherein the other utterance only includes the text string.

16. The system of claim 10, wherein the operations further comprise:

providing, for display, a third user interface that includes the first user interface element and the second, different user interface element;

receiving other audio data corresponding to another utterance that includes the text string;

providing, for display, a fourth user interface that indicates that both the first user interface element and the second, different user interface element correspond to the additional utterance; and

after a particular amount of time has elapsed;

receiving additional data indicating a selection of the first user interface element or the second, different user interface element; and

providing, for display, the third user interface.

17. The system of claim 10, wherein the first user interface element is an icon that corresponds to an action of an application and the second, different user interface element is user inputted text.

18. The system of claim 10, wherein the first user interface element and the second, different user interface element are auto completion entries in a drop-down auto completion menu.

19. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

providing, for display, a first user interface that includes a first user interface element that corresponds to a text string and a second, different user interface element that corresponds to the text string;

receiving audio data of an utterance that includes the text string;

providing, for display, a second user interface that indicates that both the first user interface element and the second, different user interface element correspond to the utterance;

receiving data indicating a selection of a particular one of the first user interface element or the second, different user interface element; and

in response to receiving the data indicating the selection of the particular one of the first user interface element or the second, different user interface element;

performing an action corresponding to the particular one of the first user interface element or the second, different user interface element; and

storing data indicating to perform the action corresponding to the particular one of the first user interface element or the second, different user interface element (i) in response to receiving additional audio data of an additional utterance that includes the text string, (ii) without providing, for display, the second user interface, and (iii) without receiving data indicating an additional selection of the particular one of the first user interface element or the second, different user interface element.

20. The medium of claim 19, wherein the first user interface element is an icon that corresponds to an action of an application and the second, different user interface element is user inputted text.