System and Method for Speech-Based Navigation and Interaction with a Device's Visible Screen Elements Using a Corresponding View Hierarchy

• US 20150142435A1
• Filed: 11/21/2013
• Published: 05/21/2015
• Est. Priority Date: 11/21/2013
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Inventors: Agrawal, Amit Kumar, Kumar P.V., Sanjeev, Rout, Satyabrata, Patil, Vishal Shashikant
• Granted Patent: US 9,600,227 B2
• Time in Patent Office: Days
• US Class Current: 704/235
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

A method 900 includes: identifying 903, within an electronic device 100, a character sequence representing a first speech input that is received 902 while displaying a viewable screen having at least one selectable viewable element; accessing 904 a view hierarchy that provides a text representation of one or more of the at least one selectable viewable element; comparing 906 the character sequence to the text representation of each of the selectable elements associated with the viewable screen; and in response to detecting 908 a match of the character sequence with at least one text representation, a processor of the electronic device 100 (i) selecting 910, within the viewable screen, a corresponding selectable viewable element whose text representation matches the character sequence and (ii) triggering 918 any action linked to selecting the corresponding selectable viewable element.

Claims

1. A method for enabling screen-specific user interfacing with elements of viewable screens presented by an electronic device, the method comprising:

identifying a character sequence representing a first input that is received while displaying a viewable screen having at least one selectable viewable element;

accessing an electronic file that provides a text representation of one or more of the at least one selectable viewable element;

comparing the character sequence to the text representation; and

in response to detecting a match of the character sequence with at least a portion of the text representation, a processor of the electronic device

(i) selecting, within the viewable screen, a selectable viewable element whose text representation matches the character sequence, and

(ii) triggering any action linked to the selecting the selectable viewable element.

2. The method of claim 1, wherein:

the at least one selectable viewable element comprises one or more of (i) a navigational function associated with the viewable screen, (ii) a functional command associated with at least one of the electronic device, the viewable screen, and content within the viewable screen, (iii) a selectable affordance presented on the viewable screen, (iv) a text sequence, word, or sequence of words presented on the viewable screen having alphanumeric characters; and

(v) graphical symbols presented on the viewable screen representing a secondary function applicable to the viewable screen; and

the selecting includes;

highlighting the selectable viewable element on the viewable screen to provide visual feedback.

3. The method of claim 1, wherein the first input is a speech input converted to a derived text sequence, the method further comprising:

receiving the speech input from a speech input component of the electronic device;

activating a speech recognition engine to decipher the speech input; and

converting the deciphered speech input, via a speech-to-text converter, to generate the character sequence.

4. The method of claim 3, wherein converting the deciphered speech input further comprises:

extracting one or more natural language dictionary (NLD) words;

in response to extracting the one or more NLD words, identifying at least one keyword from the one or more NLD words to utilize as the character sequence.

5. The method of claim 4, further comprising:

in response to failing to extract any NLD words, determining and assigning a text sequence that has a similar acoustic sound to the speech input to utilize as the character sequence.

6. The method of claim 4, further comprising:

in response to not detecting a match of the character sequence with at least a portion of the text representation;

retrieving, from an alternate word source, at least one alternate word related to the at least one keyword; and

performing a next comparison with the text representation using the at least one alternate word as the character sequence.

7. The method of claim 4, further comprising:

accessing an alternate word source;

retrieving at least one alternate word that has a relation to the at least one keyword;

expanding the character sequence to include both the at least one keyword and the at least one alternate word; and

wherein the comparing the character sequence to the text representation uses both the at least one keyword and the at least one alternate word.

8. The method of claim 1, further comprising:

in response to not detecting a match of the character sequence,

identifying a closest match candidate to the character sequence from the text representation;

presenting the closest match candidate for user confirmation;

wherein, upon receiving the user confirmation, the selectable viewable element has a text representation that matches the closest match candidate.

9. The method of claim 1, further comprising:

in response to not detecting a match of the character sequence, prompting for entry of a next input.

10. The method of claim 1, wherein the electronic file is a view hierarchy of the viewable screen and wherein accessing the electronic file further comprises:

extracting, from the view hierarchy, the text representation of each selectable viewable element, sub-element, and associated properties.

11. The method of claim 1, further comprising:

in response to a plurality of selectable viewable elements having text representations matching the character sequence;

ranking each of the plurality of selectable viewable elements based on an established priority scheme; and

in response to identifying a single selectable viewable element having a higher priority, selecting the single selectable viewable element from among the plurality of selectable viewable elements.

12. The method of claim 11, further comprising:

in response to not identifying a single selectable viewable element having a higher priority,

displaying a message indicating a conflict among the plurality of selectable viewable elements based on the first input, and

prompting for additional user input to resolve the conflict; and

in response to receiving the additional user input selecting a specific one of the plurality of selectable viewable elements, performing the selecting the selectable viewable element using the specific one of the plurality of selectable viewable elements.

13. The method of claim 1, further comprising:

in response to the character sequence including text related to multiple interrelated view elements having at least two levels of hierarchy;

identifying a primary view element among the multiple interrelated view elements;

wherein the triggering;

applies any actions associated with the primary view element; and

subsequently performs a selection of a secondary view element from among the multiple interrelated view elements.

14. The method of claim 13, wherein in response to the secondary view element being one of a first visible selection and a second visible selection for the primary view element, the identifying the primary view element further comprises:

highlighting the primary view element;

identifying a current visible selection of the secondary view element; and

automatically toggling the secondary view element to a next visible selection from the current visible selection that is displayed on the viewable screen in response to the character sequence including text related to one of the first visible selection and the second visible selection.

15. An electronic device comprising:

at least one processor;

a display coupled to the at least one processor for presenting at least one viewable screen having at least one selectable viewable element;

at least one input component communicatively coupled to the at least one processor for enabling entry of a first input while a viewable screen is presented on the display;

a plurality of executable modules that execute on the at least one processor and configure the electronic device to;

in response to receipt of the first input while displaying the viewable screen, access an electronic file that provides a text representation of one or more selectable elements associated with the viewable screen;

convert the first input to a character sequence;

compare the character sequence to the text representation; and

in response to detecting a match of the character sequence with at least a portion of the text representation,

(i) select, within the viewable screen, a selectable element whose text representation matches the character sequence, and

(ii) trigger any action linked to the selection of the selectable element.

16. The electronic device of claim 15, wherein:

the electronic file is a view hierarchy of the viewable screen;

the plurality of executable modules comprises a viewable screen content interfacing (VSCI) utility that provides;

a view item extractor, which extracts the text representation from the view hierarchy of each selectable element, sub-elements, and associated properties; and

a text comparator that compares the character sequence to the text representation of each selectable element, sub-element, and associated properties to determine whether the character sequence has an approximate match to the text representation of at least one of the selectable elements, sub-elements, or associated properties.

17. The electronic device of claim 16, wherein the plurality of executable modules further comprises:

a speech recognition engine that receives first inputs that are speech inputs;

a speech-to-text converter that converts the speech inputs into corresponding text;

a natural language processing engine that extracts one or more words from the corresponding text using a natural language dictionary (NLD); and

a keyword extractor that identifies at least one keyword from the one or more words to utilize as the character sequence.

18. The electronic device of claim 17, further comprising:

an alternate word source; and

wherein the VSCI utility, in response to not detecting a match of the character sequence with at least a portion of the text representation;

retrieves, from the alternate word source, at least one alternate word related to the at least one keyword; and

wherein the text comparator performs a next comparison using the at least one alternate word as the character sequence.

19. The electronic device of claim 16, wherein the VSCI utility further comprises:

a conflict resolution module that, in response to detecting a match of the character sequence with at least a portion of the text representation that matches a plurality of selectable elements;

ranks each of the plurality of selectable elements based on an established priority scheme;

in response to identifying a single selectable element as having a higher priority, selects the single selectable element from among the plurality of selectable elements;

in response to not identifying a single selectable element as having a higher priority,

displays a message indicating a conflict among the plurality of selectable elements based on the first input, and

prompts for additional user input to resolve the conflict; and

in response to receiving the additional user input selecting a specific one selectable element from the plurality of selectable elements, the plurality of executable modules configure the electronic device to select the specific one selectable element.

20. The electronic device of claim 16, wherein the VSCI utility further comprises:

a multi-level selection (MLS) module that;

identifies when the character sequence matches text related to multiple interrelated view elements having at least two levels of hierarchy; and

in response to the character sequence matching text related to multiple interconnected view elements;

identifies a primary view element from among the multiple interconnected view elements;

selects the primary view element;

applies any actions associated with the primary view element; and

subsequently performs a selection of a secondary view element from among the multiple interconnected view elements.

21. The electronic device of claim 20, wherein in response to the secondary view element being one of a first visible selection and a second visible selection for the primary view element, the MLS module further:

highlights the primary view element;

identifies a current visible selection of the secondary view element; and

automatically toggles the secondary view element to a next visible selection from the current visible selection that is displayed on the viewable screen in response to the character sequence matching text related to one of the first visible selection and the second visible selection.