Graphical user interface language model

• US 10,261,753 B2
• Filed: 05/16/2017
• Issued: 04/16/2019
• Est. Priority Date: 05/16/2017
• Status: Active Grant

• Current Assignee: Bank of America Corp.
• Original Assignee: Bank of America Corp.
• Inventors: Chakraborty, Pinak, Lamba, Anupreet K.
• Primary Examiner(s): Ell, Matthew
• Assistant Examiner(s): Shalu, Zelalem W
• Application Number:
  US15/596,187
• Publication Number:
  US 20180336006A1
• Time in Patent Office: 700 Days
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G09B 21/005 Details of specially-adapted software to access information, e.g. to browse through hyperlinked information
• CPC Class Codes: G09B 21/008 using visual presentation of the information for the partially sighted
• CPC Class Codes: G10L 13/027 Concept to speech synthesisers; Generation of natural phrases from machine-based concepts generation of parameters for speech synthesis out of text G10L13/08

Abstract

Method and apparatus for allowing visually impaired users to easily interact with GUI applications is provided. The method and apparatus may utilize a directed graph of the GUI and a language model to describe the GUI in a brief but concise and descriptive manner.

Claims

1. A method for generating an audio description of a Graphical User Interface (GUI) operating on a computer system, the method comprising:

activating a computer program, the computer program displaying a GUI on a monitor, the GUI comprising a plurality of objects, one of the plurality of objects being primed to receive input;

preparing a directed graph representation of the GUI, the directed graph representation defining a data structure including a node associated with each of the plurality of objects, an action verb associated with each node, an action object associated with each action verb, a transition associated with each action verb, and a consequence associated with each action verb;

parsing at least one string to contain words describing the primed object, the number of parsed strings being equal to a number of action verbs associated with the one of the nodes associated with the primed object, the parsing comprising;

concatenating and conjugating according to an order and rules of a human language grammar, the node associated with the primed object, the action verbs associated with the node associated with the primed object, and each of the action object, transition, and consequence associated with the action verbs; and

generating a text-to-voice formatted audio signal corresponding to a sequential expression of each parsed string;

wherein;

each string comprises a unique action verb;

each action verb comprises words of a human language description of an activity which may be performed by the corresponding primed object according to execution code logic of the computer program;

each action object comprises words of a human language description of a user input activity for the primed object addressed by the execution code logic;

each consequence comprises words of a human language description of a generated result in response to executing the action verb; and

each transition comprises words of a human language description of an object or GUI primed in response to executing the action verb.

2. The method of claim 1 wherein the user input activity includes striking keys on a keyboard.

3. The method of claim 1 further comprising receiving an input from a user in response to the generated audio signal.

4. The method of claim 1 further comprising:

receiving from a user, via a microphone, an idiosyncratic expression in response to the generated audio signal;

associating the idiosyncratic expression with an action verb; and

executing the generated result associated with the action verb;

wherein;

the idiosyncratic expression defines a response different from the action verbs and action objects explicitly listed in the directed graph but human understandable to correspond to with an action verb or action object explicitly listed in the directed graph.

5. The method of claim 4 wherein:

the microphone transmits the expression to speech-to-text software; and

the speech-to-text software converts the expression into a data format which can be processed by the execution code logic.

6. The method of claim 1 wherein the GUI comprises at least one object selected from the group consisting of checkboxes, radio buttons, drop down lists, combo boxes, list boxes, tree views, hotkeys, toggles, text fields, vertical slider bars, horizontal slider bars, progress bars, menu bars, tab sequences, group boxes and any combination thereof.

7. The method of claim 3 wherein in response to the receipt of the user input, changing the human language of the generated audio signal.

8. The method of claim 1 wherein the directed graph representation of the GUI is prepared prior to activating the computer program.

9. The method of claim 1 wherein the data structure is selected from the group consisting of list, array, associative array, multimap, set, bag, multiset, stack, queue, double ended queue, priority queue, tree, graph, and any combination thereof.

10. The method of claim 1 wherein the audio signal comprises a sound attribute selected from the group consisting of alternating tone, alternating tempo, musical notes, pauses between words, sound effects, and words absent from all of the nodes, action verbs, action objects, transitions and consequences.