Dynamic inference of voice command for software operation from help information

• US 9,583,097 B2
• Filed: 01/30/2015
• Issued: 02/28/2017
• Est. Priority Date: 01/30/2015
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Agrawal, Amit Kumar, Essick, Raymond B. IV, Rout, Satyabrata
• Primary Examiner(s): PULLIAS, JESSE SCOTT
• Application Number:
  US14/609,973
• Publication Number:
  US 20160225371A1
• Time in Patent Office: 760 Days
• Field of Search: 704/275
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/075 supervised, i.e. under machine guidance
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence

Abstract

In an electronic device, a method includes analyzing help information associated with a software application to identify a sequence of manipulations of viewable elements associated with an instance of an operation by the software application. The method further includes generating a voice command set based on the sequence of manipulations of viewable elements and storing the voice command set. The method further includes receiving voice input from a user, determining the voice input represents a voice command of the voice command set, and performing an emulated manipulation sequence of viewable elements based on the voice command to actuate an instance of the operation by the software application, the emulated manipulation sequence based on the sequence of manipulations of viewable elements.

Claims

1. In an electronic device, a method comprising:

analyzing help information associated with a software application to identify a sequence of manipulations of viewable elements associated with an instance of an operation by the software application, wherein the help information is to assist a user in interacting with the software application to perform the operation and comprises at least one of;

a textual tutorial with one or more screen shots of the software application;

a video tutorial for performing the operation; and

an audio tutorial for performing the operation;

generating a voice command set based on the sequence of manipulations of viewable elements; and

storing the voice command set.

2. The method of claim 1, wherein analyzing the help information includes:

identifying a visual reference to an action of the operation to be actuated by the software application responsive to manipulation of an identified viewable element associated with the software application; and

identifying a term associated with the action; and

wherein generating the voice command set includes;

generating at least one voice command for the voice command set based on the identified term.

3. The method of claim 2, wherein identifying the term associated with the action comprises:

accessing a set of one or more descriptive attributes associated with the identified viewable element; and

identifying the term associated with the action based on the set of one or more descriptive attributes.

4. The method of claim 3, wherein accessing the set of one or more descriptive attributes associated with the viewable element comprises:

accessing the set of one or more descriptive attributes from a view hierarchy having a text representation of elements within one or more view screens.

5. The method of claim 4, wherein identifying the term associated with the action comprises:

identifying the term from at least one of;

the view hierarchy;

a natural language processor; and

an alternative term database.

6. The method of claim 2, wherein identifying the term associated with the action comprises:

identifying the term associated with the action based on a text description in the help information that references the action.

7. The method of claim 1, further comprising:

receiving voice input from a user;

determining the voice input represents a voice command of the voice command set; and

performing an emulated manipulation sequence of viewable elements based on the voice command to actuate an instance of the operation by the software application, the emulated manipulation sequence based on the sequence of manipulations of viewable elements.

8. The method of claim 1, wherein analyzing the help information comprises:

matching a first screenshot of the software application presented in the help information with a first view screen of the software application;

matching a first graphical element of the first screenshot that is referenced by the help information with a first viewable element of the first view screen; and

identifying the first viewable element of the first view screen as a viewable element manipulated in the sequence of manipulations of viewable elements.

9. The method of claim 8, wherein analyzing the help information further comprises:

matching a second screenshot of the software application presented in the help information with a second view screen of the software application;

matching a second graphical element of the second screenshot that is referenced by the help information with a second viewable element of the second view screen;

identifying the second viewable element of the second view screen as a viewable element manipulated in the sequence of manipulations of viewable elements; and

identifying an order between the manipulation of the first viewable element and the manipulation of the second viewable element in the sequence of manipulations of viewable elements based on an order of reference between the first graphical element and the second graphical element in the help information.

10. The method of claim 8, wherein analyzing the help information comprises:

matching a text description of a user manipulation action in the help information with a second viewable element of a second view screen of the software application; and

identifying the second viewable element of the second view screen as a viewable element manipulated in the sequence of manipulations of viewable elements; and

identifying an order between the first viewable element and the second viewable element in the sequence of manipulations of viewable elements based on an order of reference between the first graphical element and the text description in the help information.

11. The method of claim 1, wherein analyzing the help information comprises:

matching a text description of a first user manipulation action in the help information with a first viewable element of a first view screen of the software application; and

identifying the first viewable element of the first view screen as a viewable element manipulated in the sequence of manipulations of viewable elements.

12. The method of claim 1, wherein analyzing the help information associated with the software application to identify the sequence of manipulations of viewable elements comprises:

identifying a manipulation of a viewable element as representing an action referenced in the help information; and

programmatically verifying that the manipulation of the viewable element results in the action referenced in the help information.

13. The method of claim 1, further comprising:

accessing the help information from one or more of;

a remote networked system; and

a help datastore locally stored in association with the software application.

14. The method of claim 1, further comprising:

providing a representation of the voice command set and the sequence of manipulations of viewable elements to a remote networked system for distribution to one or more other electronic devices.

15. In an electronic device, a method comprising:

identifying a first viewable element of a view screen of a software application as being associated with a first action of an operation performed by the software application based on help information associated with the software application, wherein the help information is to assist a user in interacting with the software application to perform the operation and comprises at least one of;

a textual tutorial with one or more screen shots of the software application;

a video tutorial for performing the operation; and

an audio tutorial for performing the operation;

identifying a second viewable element of a view screen of the software application as being associated with a second action of the operation based on the help information associated with the software application;

generating a voice command set representing the operation based on the first viewable element and the second viewable element; and

storing the voice command set.

16. The method of claim 15, wherein generating the voice command set comprises:

determining a first term associated with the first action;

determining a second term associated with the second action;

determining an order between the first action and the second action within the operation; and

generating a voice command of the voice command set based on the first term, the second term, and the order between the first action and the second action.

17. The method of claim 16, wherein:

determining the first term associated with the first action includes determining the first term based on a text description in the help information that is associated with the first action.

18. The method of claim 16, wherein:

determining the first term associated with the first action includes determining the first term based on at least one descriptive attribute associated with the first viewable element.

19. The method of claim 15, further comprising:

receiving a voice input from a user;

determining the voice input represents a voice command of the voice command set; and

performing an emulation of a manipulation sequence of viewable elements based on the voice command to actuate an instance of the operation by the software application.

20. The method of claim 19, further comprising:

identifying a manipulation of the first viewable element that actuates the first action;

identifying a manipulation of the second viewable element that actuates the second action;

determining an order between the first action and the second action within the operation; and

generating the manipulation sequence of viewable elements based on the manipulation of the first viewable element, the manipulation of the second viewable element, and the order between the first action and the second action.

21. The method of claim 15, wherein identifying the first viewable element as being associated with the first action comprises:

matching a screenshot in the help information that is associated with the first action with a view screen of the software application that contains the first viewable element.

22. The method of claim 15, wherein identifying the first viewable element as being associated with the first action comprises:

matching the first viewable element with text description in the help information that is associated with the first action.

23. An electronic device comprising:

a help information analysis module to analyze help information associated with a software application to identify a sequence of manipulations of viewable elements associated with an instance of an operation by the software application, wherein the help information is to assist a user in interacting with the software application to perform the operation and comprises at least one of;

a textual tutorial with one or more screen shots of the software application;

a video tutorial for performing the operation; and

an audio tutorial for performing the operation;

a voice command generator module, coupled to the help information analysis module, to generate a voice command set based on the sequence of manipulations of viewable elements; and

a datastore to store the voice command set.

24. The electronic device of claim 23, wherein:

the help information analysis module also identifies a visual reference to an action of the operation to be actuated by the software application responsive to manipulation of an identified viewable element associated with the software application; and

the voice command generator module generates the voice command set by identifying a term associated with the action and by generating at least one voice command of the voice command set based on the identified term.

25. The electronic device of claim 24, wherein the voice command generator module is to identify the term associated with the action from at least one of:

a set of one or more descriptive attributes associated with the identified viewable element; and

a text description in the help information that references the action.

26. The electronic device of claim 25, further comprising:

a view manager, coupled to the voice command generator module, to store a view hierarchy having a text representation of viewable elements within one or more view screens; and

wherein the voice command generator module is to determine the set of one or more descriptive attributes from the view hierarchy.

27. The electronic device of claim 23, further comprising:

a command mapper module, coupled to the datastore, to determine a received voice input represents a voice command of the voice command set; and

a manipulation emulation module, coupled to the command mapper module, to perform an emulated manipulation sequence of viewable elements based on the voice command to actuate an instance of the operation by the software application, the emulated manipulation sequence based on the sequence of manipulations of viewable elements.

28. The electronic device of claim 23, wherein:

the help information analysis module is to correlate a first screenshot of the software application presented in the help information with a first view screen of the software application and to correlate a first graphical element of the first screenshot that is referenced by the help information with a first viewable element of the first view screen; and

the help information analysis module further is to identify the first viewable element of the first view screen as a viewable element manipulated in the sequence of manipulations of viewable elements.