System and method for speech-based navigation and interaction with a device's visible screen elements using a corresponding view hierarchy

• US 9,600,227 B2
• Filed: 11/21/2013
• Issued: 03/21/2017
• Est. Priority Date: 11/21/2013
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Inventors: P.V., Sanjeev Kumar, Agrawal, Amit Kumar, Rout, Satyabrata, Patil, Vishal Shashikant
• Primary Examiner(s): Opsasnick, Michael N
• Application Number:
  US14/086,454
• Publication Number:
  US 20150142435A1
• Time in Patent Office: 1,216 Days
• Field of Search: 704/270, 704/276
• US Class Current: 1/1
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for enabling screen-specific user interfacing with elements of viewable screens presented by an electronic device are disclosed. In one aspect, a method includes the actions of identifying a character sequence representing a first input that is received while displaying a viewable screen having at least one selectable viewable element. The actions further include accessing an electronic file that provides a text representation of one or more of the at least one selectable viewable element. The actions further include comparing the character sequence to the text representation. The actions further include selecting, within the viewable screen, a selectable viewable element whose text representation matches the character sequence. The actions further include triggering any action linked to the selecting the selectable viewable element.

Claims

1. A computer-implemented method comprising:

obtaining (i) a view hierarchy that is of a user interface and that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of a particular viewable element in the user interface, and (ii) a transcription of a speech input that was spoken while the user interface was displayed;

determining that the transcription of the speech input matches text that is associated with the particular viewable element (i) that is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken; and

in response to determining that the transcription of the speech input matches text that is associated with the particular viewable element (i) that is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken, initiating an action that is associated with the particular viewable element (i) that is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken.

2. The method of claim 1, wherein the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface further includes data indicating an action of each selectable element of the particular viewable element and the text-based and non-text-based viewable elements in the user interface.

3. The method of claim 1, comprising:

determining that the that the transcription of the speech input matches text that is associated with another viewable element that (i) is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken; and

determining, based on the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface, that the particular viewable element has a higher priority than the other viewable element,

wherein the action is initiated based on determining that the particular viewable element has a higher priority than the other viewable element.

4. The method of claim 1, wherein the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface further includes priority assignments for each of the text-based and non-text-based viewable elements and the particular viewable element.

5. The method of claim 1, wherein the text-based and non-text-based viewable elements include multi-level interconnected viewable elements.

6. The method of claim 1, wherein the location data includes a pixel location on the user interface of each text-based and non-text-based viewable element.

7. The method of claim 1, comprising:

determining that the that the transcription of the speech input matches text that is associated with another viewable element that (i) is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken;

determining, based on the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface, that the particular viewable element has an equal priority to the other viewable element;

providing data indicating that the transcription of the speech input matches text that is associated with the particular viewable element and the other viewable element;

obtaining an additional transcription of an additional speech input; and

determining that the additional transcription of the additional speech input matches the text that is associated with the particular viewable element,

wherein the action is initiated based on determining that the additional transcription of the additional speech input matches the text that is associated with the particular viewable element.

8. The method of claim 7, wherein the data indicating that the transcription of the speech input matches text that is associated with the particular viewable element and the other viewable element comprises:

providing, for display, text that uniquely identifies the particular viewable element while highlighting the particular viewable element; and

providing, for display, additional text that uniquely identifies the other viewable element while highlighting the other viewable element.

9. The method of claim 7, wherein the data indicating that the transcription of the speech input matches text that is associated with the particular viewable element and the other viewable element comprises:

playing audio data of text that uniquely identifies the particular viewable element while highlighting the particular viewable element; and

playing audio data of additional text that uniquely identifies the other viewable element while highlighting the other viewable element.

10. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

obtaining (i) a view hierarchy that is of a user interface and that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of a particular viewable element in the user interface, and (ii) a transcription of a speech input that was spoken while the user interface was displayed;

determining that the transcription of the speech input matches text that is associated with the particular viewable element (i) that is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken; and

in response to determining that the transcription of the speech input matches text that is associated with the particular viewable element (i) that is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken, initiating an action that is associated with the particular viewable element (i) that is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken.

11. The system of claim 10, wherein the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface further includes data indicating an action of each selectable element of the particular viewable element and the text-based and non-text-based viewable elements in the user interface.

12. The system of claim 10, wherein the operations further comprise:

determining that the that the transcription of the speech input matches text that is associated with another viewable element that (i) is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken; and

determining, based on the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface, that the particular viewable element has a higher priority than the other viewable element,

wherein the action is initiated based on determining that the particular viewable element has a higher priority than the other viewable element.

13. The system of claim 10, wherein the operations further comprise:

determining that the that the transcription of the speech input matches text that is associated with another viewable element that (i) is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken;

determining, based on the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface, that the particular viewable element has an equal priority to the other viewable element;

providing data indicating that the transcription of the speech input matches text that is associated with the particular viewable element and the other viewable element;

obtaining an additional transcription of an additional speech input; and

determining that the additional transcription of the additional speech input matches the text that is associated with the particular viewable element,

wherein the action is initiated based on determining that the additional transcription of the additional speech input matches the text that is associated with the particular viewable element.

14. The system of claim 13, wherein the data indicating that the transcription of the speech input matches text that is associated with the particular viewable element and the other viewable element comprises:

providing, for display, text that uniquely identifies the particular viewable element while highlighting the particular viewable element; and

providing, for display, additional text that uniquely identifies the other viewable element while highlighting the other viewable element.

15. The system of claim 13, wherein the data indicating that the transcription of the speech input matches text that is associated with the particular viewable element and the other viewable element comprises:

playing audio data of text that uniquely identifies the particular viewable element while highlighting the particular viewable element; and

playing audio data of additional text that uniquely identifies the other viewable element while highlighting the other viewable element.

16. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

obtaining (i) a view hierarchy that is of a user interface and that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of a particular viewable element in the user interface, and (ii) a transcription of a speech input that was spoken while the user interface was displayed;

determining that the transcription of the speech input matches text that is associated with the particular viewable element (i) that is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken; and

in response to determining that the transcription of the speech input matches text that is associated with the particular viewable element (i) that is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken, initiating an action that is associated with the particular viewable element (i) that is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken.

17. The medium of claim 16, wherein the operations further comprise:

determining that the that the transcription of the speech input matches text that is associated with another viewable element that (i) is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken; and

determining, based on the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface, that the particular viewable element has a higher priority than the other viewable element,

wherein the action is initiated based on determining that the particular viewable element has a higher priority than the other viewable element.

18. The medium of claim 16, wherein the operations further comprise:

determining that the that the transcription of the speech input matches text that is associated with another viewable element that (i) is included in the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface and (ii) that was displayed when the speech input was spoken;

determining, based on the view hierarchy that includes textual representations, location data, and properties of text-based and non-text-based viewable elements in the user interface and of the particular viewable element in the user interface, that the particular viewable element has an equal priority to the other viewable element;

providing data indicating that the transcription of the speech input matches text that is associated with the particular viewable element and the other viewable element;

obtaining an additional transcription of an additional speech input; and

determining that the additional transcription of the additional speech input matches the text that is associated with the particular viewable element,

wherein the action is initiated based on determining that the additional transcription of the additional speech input matches the text that is associated with the particular viewable element.

19. The medium of claim 18, wherein the data indicating that the transcription of the speech input matches text that is associated with the particular viewable element and the other viewable element comprises:

providing, for display, text that uniquely identifies the particular viewable element while highlighting the particular viewable element; and

providing, for display, additional text that uniquely identifies the other viewable element while highlighting the other viewable element.