Usability improvements for visual interfaces

• US 10,452,231 B2
• Filed: 06/26/2015
• Issued: 10/22/2019
• Est. Priority Date: 06/26/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Akiner, Veli, Confino, Benjamin A., Jiang, Fenghui, Ross, Martin A., Whitehouse, Bradley G.
• Primary Examiner(s): Chuang, Jung-Mu T
• Application Number:
  US14/751,914
• Publication Number:
  US 20160378274A1
• Time in Patent Office: 1,579 Days
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

One or more aspects relate to a screen reader. Markup code for a visual interface containing at least one visual control is located. A screen reader menu for the visual interface is created. A visual control name and screen location, both corresponding to the at least one visual control, are extracted from the markup code. A menu item is generated in the screen reader menu corresponding to the at least one visual control. The menu item is set to simulate an action at the screen location corresponding to the at least one visual control on selection of the menu item whereby on user selection of the menu item the action is simulated at the screen location corresponding to the at least one visual control.

Claims

1. A screen reader system comprising:

a memory;

one or more processors in communication with the memory; and

program instructions executable by the one or more processors via the memory to perform a method, the method comprising;

locating, by the one or more processors, markup code for a visual interface containing an input node, wherein the visual interface comprises a proprietary application which is a message flow application that is not operating system based, and wherein the visual interface comprises a visual integrated development environment;

creating, by the one or more processors, a screen reader menu for the visual interface to view in the visual interface of the screen reader system;

extracting, by the one or more processors, properties from the markup code, the properties including a visual control name and a screen location, both corresponding to the input node, wherein the markup code comprises properties of the input node;

generating, by the one or more processors, based on the properties from the markup code of the proprietary application, a menu item in the screen reader menu corresponding to the input node, wherein a location of the generated menu item is different from a screen location corresponding to the input node in the properties of the markup code, wherein the generating provides accessibility to a user utilizing the screen reader system to the input node and to an underlying action associated with selecting the input node, at the screen location corresponding to the input node, through the generated menu item, wherein at the input node and the underlying action associated with selecting the input node, at the screen location corresponding to the input node-is inaccessible by the user utilizing the screen reader system, through the visual interface of the screen reader system, prior to the generating, based on the underlying action comprising the proprietary application, because the visual interface is not operating system based, wherein the underlying action is selected from the group consisting of 1;

a left mouse click, a right mouse click, and a hover, and wherein the generating enables the user of the screen reader system to achieve a focus on the input node via the generated menu item within the visual interface of the screen reader system;

simulating, by the one or more processors, a selection of the input node at the screen location corresponding to the input node, based on obtaining a selection the generated menu item in the screen reader menu by the user, wherein the selection of the generated menu item comprises triggering the underlying action associated with the input node, at the screen location corresponding to the input node; and

rendering, by the one or more processors, the generated menu item in the screen reader system such that the user can access the input node via the generated menu item to make inputs.

2. The screen reader system according to claim 1, the method further comprising:

locating at least one dependent visual control depending from the input node; and

extracting, from the markup code, a dependent visual control name and screen location, both corresponding to the at least one dependent visual control.

3. The screen reader system according to claim 2, the method further comprising:

extracting, from the markup code, further descriptive information relating to the at least one dependent a visual control; and

making such further descriptive information available from the generated menu item.

4. The screen reader system according to claim 2, the method further comprising:

locating at least one doubly dependent visual control depending from the at least one dependent visual control; and

extracting from the markup code a doubly dependent visual control name and screen location, both corresponding to the at least one doubly dependent visual control.

5. The screen reader system according to claim 4, the method further comprising:

extracting, from the markup code, further descriptive information relating to the doubly dependent visual control; and

making such further descriptive information available from the generated menu item.

6. The screen reader system according to claim 1, the method further comprising:

extracting, from the markup code, further descriptive information relating to the input node; and

making such further descriptive information available from the generated menu item.

7. A computer-implemented method for a screen reader comprising:

locating markup code for a visual interface containing an input node, wherein the visual interface comprises a proprietary application which is a message flow application that is not operating system based, and wherein the visual interface comprises a visual integrated development environment;

creating a screen reader menu for the visual interface to view in the visual interface of the screen reader;

extracting properties from the markup code, the properties including a visual control name and a screen location, both corresponding to the input node, wherein the markup code comprises properties of the input node;

generating, based on the properties from the markup code of the proprietary application, a menu item in the screen reader menu corresponding to an input node, wherein a location of the generated menu item is different from a screen location corresponding to the input node in the properties of the markup code, wherein the generating provides accessibility to a user utilizing the screen reader to the input node and to an underlying action associated with selecting the input node, at the screen location corresponding to the input node, through the generated menu item, wherein at the input node and the underlying action associated with selecting the input node, at the screen location corresponding to the input node is inaccessible by the user utilizing the screen reader, through the visual interface of the screen reader, prior to the generating, based on the underlying action comprising the proprietary application, because the visual interface is not operating system based, wherein the underlying action is selected from the group consisting of;

a left mouse click, a right mouse click, and a hover, and wherein the generating enables the user of the screen reader to achieve a focus on the input node via the generated menu item within the visual interface of the screen reader;

simulating a selection of the input node at the screen location corresponding to the input node, based on obtaining a selection of the generated menu item in the screen reader menu by the user, wherein the selection of the generated menu item comprises triggering the underlying action associated with the input node, at the screen location corresponding to the input node; and

rendering the generated menu item in the screen reader such that the user can access the input node via the generated menu item to make inputs.

8. The computer-implemented method according to claim 7, further comprising locating at least one dependent visual control depending from the input node, and extracting from the markup code the at least one dependent visual control name and screen location, both corresponding to the input node.

9. The computer-implemented method according to claim 8, further comprising extracting from the markup code further descriptive information relating to the at least one visual control and making such further descriptive information available from the generated menu item.

10. The computer-implemented method according to claim 8, further comprising locating at least one doubly dependent visual control depending from the at least one dependent visual control, and extracting from the markup code a doubly dependent visual control name and screen location, both corresponding to the at least one doubly dependent visual control.

11. The computer-implemented method according to claim 7, further comprising extracting from the markup code further descriptive information relating to the input node and making such further descriptive information available from the generated menu item.

12. A computer program product for a screen reader, the computer program product comprising a computer readable storage medium having program instructions embodied therein, the program instructions executable by a processor to cause the processor to:

locate markup code for a visual interface containing an input node, wherein the visual interface comprises a proprietary application which is a message flow application that is not operating system based, and wherein the visual interface comprises a visual integrated development environment;

create a screen reader menu for the visual interface to view in the visual interface of the screen reader;

extract properties from the markup code, the properties including a visual control name and a screen location, both corresponding to the input node, wherein the markup code comprises properties of the input node;

generate, based on the properties from the markup code of the proprietary application, a menu item in the screen reader menu corresponding to an input node, wherein a location of the generated menu item is different from a screen location corresponding to the input node in the properties of the markup code, wherein the generating provides accessibility to a user utilizing the screen reader to the input node and to an underlying action associated with selecting the input node, at the screen location corresponding to the input node, through the generated menu item, wherein at the input node and the underlying action associated with selecting the input node, at the screen location corresponding to the input node is inaccessible by the user utilizing the screen reader, through the visual interface of the screen reader, prior to the generating, based on the underlying action comprising the proprietary application, because the visual interface is not operating system based, wherein the underlying action is selected from the group consisting of;

a left mouse click, a right mouse click, and a hover, and wherein the generating enables the user of the screen reader to achieve a focus on the input node via the generated menu item within the visual interface of the screen reader;

simulate a selection of the input node at the screen location corresponding to the input node, based on obtaining a selection of the generated menu item in the screen reader menu by the user, wherein the selection of the generated menu item comprises triggering the underlying action associated with the input node, at the screen location corresponding to the input node; and

render the generated menu item in the screen reader such that the user can access the input node via the generated menu item to make inputs.

13. The computer program product according to claim 12, wherein the processor is to locate at least one dependent visual control depending from the input node, and extract from the markup code the at least one dependent visual control name and screen location, both corresponding to the input node.

14. The computer program product according to claim 13, wherein the processor is further to locate at least one doubly dependent visual control depending from the at least one dependent visual control, and extract from the markup code a doubly dependent visual control name and screen location, both corresponding to the at least one doubly dependent visual control.

15. The computer program product according to claim 12, wherein the processor is further to extract from the markup code further descriptive information relating to the input node and make such further descriptive information available from the generated menu item.