Automated voice enablement of applications

• US 10,481,865 B2
• Filed: 10/23/2017
• Issued: 11/19/2019
• Est. Priority Date: 10/23/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dolph, Blaine H., Lubensky, David M., Pattiarachi, Mal, Pistoia, Marco, Rajput, Nitendra, Weisz, Justin
• Primary Examiner(s): Pham, Linh K
• Application Number:
  US15/790,162
• Publication Number:
  US 20190121609A1
• Time in Patent Office: 757 Days
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/247 Thesauruses; Synonyms
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/56 Natural language generation
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

Techniques are disclosed for generating a voice user interface (VUI) modality within an application that includes graphical user interface (GUI) screens. A GUI screen parser analyzes the GUI screens to determine the various navigational GUI screen paths that are associated with edge objects within multiple GUI screens. Some edge objects are identified as select objects or prompt objects. A natural language processing system generates a select object synonym data structure and a prompt object data structure that may be utilized by a VUI generator to generate VUI data structures that give the application VUI modality.

Claims

1. A method, comprising:

receiving application data of an application that is under development from an application developer computing system;

aggregating static images of a graphical user interface of the application (GUI screens) that are contained within the application data;

analyzing each GUI screen to determine GUI screen navigational flow paths (flow paths);

generating a flow path data structure comprising a list of the flow paths ranked by the number of edge objects comprised within the flow paths and comprising a list of edge objects within the flow paths ranked by the number of dependent edge objects;

identifying a first edge object within the flow path data structure as a select object;

determining a word or phrase associated with the select object from characters that are displayed by the select object with a natural language processing system;

determining synonyms to the word or phrase with the natural language processing system;

generating a select object synonym data structure comprising the word or phrase and the synonyms to the word or phrase associated with the select object;

creating a voice user interface (VUI) select object data structure from the select object synonym data structure;

sending the VUI select object data structure to the application developer computing system to create an application with VW modality; and

wherein a first GUI screen navigational flow path is a first navigational sequence of GUI screens associated with a first set of user interactions with the GUI screens and a second GUI screen navigational flow path is a second navigational sequence of GUI screens associated with a second set of user interactions with the GUI screens.

3. The method of claim 1, wherein the GUI screens are aggregated screenshots of the application under development that is evoked by the application developer computing system.

4. The method of claim 1, wherein the GUI screens are aggregated from an executable file of the application under development.

6. The method of claim 1, wherein the GUI screens are aggregated from storyboard images of the application under development.

7. The method of claim 1, wherein the flow path data structure further comprises an edge object identified as a fork object that belongs to multiple flow paths.

8. The method of claim 1, wherein the first set of user interactions are a first order of simulated user manipulation sequences of a first collection of edge objects within the GUI screens, wherein the second set of user interactions are a second order of simulated user manipulation sequences of a second collection of edge objects within the GUI screens, and wherein a same edge object is comprised within the first collection of edge objects and within the second collection of edge objects.

2. The method of claim wherein the select object is a GUI object that begins or continues an associated GUI screen flow path upon the selection of the GUI object.

5. The method of claim wherein the GUI screens are aggregated from mockup images of the application under development.

9. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions readable by a service computing system to cause the service computing system to:

receive application data of an application that is under development from an application developer computing system;

aggregate static images of a graphical user interface of the application (GUI screens) that are contained within the application data;

analyze each GUI screen to determine GUI screen navigational flow paths (flow paths);

generate a flow path data structure comprising a list of the flow paths ranked by the number of edge objects comprised within the flow paths and comprising a list of edge objects within the flow paths ranked by the number of dependent edge objects;

identify a first edge object within the flow path data structure as a select object;

determine a word or phrase associated with the select object from characters that are displayed by the select object with a natural language processing system;

determine synonyms to the word or phrase with the natural language processing system;

generate a select object synonym data structure comprising the word or phrase and the synonyms to the word or phrase associated with the select object;

create a voice user interface (VUI) select object data structure from the select object synonym data structure;

send the VUI select object data structure to the application developer computing system to create an application with VUI modality; and

wherein a first GUI screen navigational flow path is a first navigational sequence of GUI screens associated with a first set of user interactions with the GUI screens and a second GUI screen navigational flow path is a second navigational sequence of GUI screens associated with a second set of user interactions with the GUI screens.

10. The computer program product of claim 9, wherein the select object is a GUI object that begins or continues an associated GUI screen flow path upon the selection of the GUI object.

11. The computer program product of claim 9, wherein the GUI screens are aggregated screenshots of the application under development that is evoked by the application developer computing system.

12. The computer program product of claim 9, wherein the GUI screens are aggregated from an executable file of the application under development.

13. The computer program product of claim 9, wherein the GUI screens are aggregated from mockup images of the application under development.

15. The computer program product of claim 9, wherein the flow path data structure further comprises an edge object identified as a fork object that belongs to multiple flow paths.

16. The computer program product of claim 9, wherein the first set of user interactions are a first order of simulated user manipulation sequences of a first collection of edge objects within the GUI screens, wherein the second set of user interactions are a second order of simulated user manipulation sequences of a second collection of edge objects within the GUI screens, and wherein a same edge object is comprised within the first collection of edge objects and within the second collection of edge objects.

14. The computer program product of claim wherein the GUI screens are aggregated from storyboard images of the application under development.