Screen reader improvements

• US 10,394,421 B2
• Filed: 06/26/2015
• Issued: 08/27/2019
• Est. Priority Date: 06/26/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Akiner, Veli, Confino, Benjamin A., Jiang, Fenghui, Ross, Martin A., Whitehouse, Bradley G.
• Primary Examiner(s): Silverman, Seth A
• Application Number:
  US14/751,984
• Publication Number:
  US 20160378275A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 8/76 Adapting program code to run in a different environment; Porting
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables

Abstract

One or more aspects relate to providing a user interface menu in a screen reader reading an application. A graphical user interface (GUI) is screen scraped to determine GUI components and a user option menu is created including user options corresponding to the determined GUI components. A corresponding GUI component is activated when a user option is selected.

Claims

1. A method of providing a temporary user interface menu in a screen reader reading an application within a computing environment, the method comprising:

detecting by the screen reader when the application is started, the application including a graphical user interface (GUI) of a display screen of the computing environment, and determining by the screen reader that an existing user option menu exists for the application;

screen scraping the GUI of the application using visual analysis of the GUI of the display screen to determine GUI components of the GUI, the GUI components including menu known GUI components and one or more menu unknown GUI components, the menu known GUI components being accessible from the existing user option menu, and the one or more menu unknown GUI components being unknown from and inaccessible from the existing user option menu and not corresponding to any existing user options in the existing user option menu determined for the application, and the screen scraping of the GUI using visual analysis comprising digital image processing of the GUI, the screen scraping including;

determining which GUI components of the GUI are the menu known GUI components, accessible from the existing user options in the existing user option menu; and

determining which GUI component(s) of the GUI are the one or more menu unknown GUI components, unknown and inaccessible from the existing user options in the existing user option menu;

determining x-y coordinates of the display screen for the one or more menu unknown GUI components identified based on the screen scraping;

creating an updated user option menu, from the existing user option menu, for the application in the screen reader comprising user options corresponding to each of the determined GUI components, including the menu known GUI components and the one or more menu unknown GUI components, the creating including;

creating a new user option in the existing user option menu that corresponds to a menu unknown GUI component of the one or more menu unknown GUI components identified based on the screen scraping, the new user option, when selected, simulating a mouse click at target coordinates of the display screen to initiate the menu unknown GUI component, the target coordinates being determined using the determined x-y coordinates for that GUI component;

activating a corresponding GUI component when a user option is selected in the updated user option menu in the screen reader; and

based on detecting closing of the application, deleting the updated user option menu from the computing environment.

2. The method according to claim 1, wherein the screen scraping of the GUI comprises performing optical character recognition on a bit map of the GUI in order to identify GUI controls and labels.

3. The method according to claim 1, wherein the screen scraping of the GUI comprises performing edge detection on a bit map of the GUI in order to identify GUI controls and labels.

4. The method according to claim 1, wherein activating the corresponding GUI component comprises:

selecting the corresponding GUI component;

simulating left or right mouse clicks on the corresponding GUI component;

or hovering a cursor over the corresponding GUI component.

5. The method according to claim 1, wherein the screen scraping of the GUI comprises performing complementary edge detection and optical character recognition on a bit map of the GUI in order to identify GUI controls and labels.

6. A computer program product for providing a temporary user interface menu in a screen reader reading an application within a computing environment, the computer program product comprising:

a computer readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising;

detecting by the screen reader when the application is started, the application including a graphical user interface (GUI) of a display screen of the computing environment, and determining by the screen reader that an existing user option menu exists for the application;

screen scraping the GUI of the application using visual analysis of the GUI of the display screen to determine GUI components of the GUI, the GUI components including menu known GUI components and one or more menu unknown GUI components, the menu known GUI components being accessible from the existing user option menu, and the one or more menu unknown GUI components being unknown from and inaccessible from the existing user option menu and not corresponding to any existing user options in the existing user option menu determined for the application, and the screen scraping of the GUI using visual analysis comprising digital image processing of the GUI, the screen scraping including;

determining which GUI components of the GUI are the menu known GUI components, accessible from the existing user options in the existing user option menu; and

determining which GUI component(s) of the GUI are the one or more menu unknown GUI components, unknown and inaccessible from the existing user options in the existing user option menu;

determining x-y coordinates of the display screen for the one or more menu unknown GUI components identified based on the screen scraping;

creating an updated user option menu, from the existing user option menu, for the application in the screen reader comprising user options corresponding to each of the determined GUI components, including the menu known GUI components and the one or more menu unknown GUI components, the creating including;

creating a new user option in the existing user option menu that corresponds to a menu unknown GUI component of the one or more menu unknown GUI components identified based on the screen scraping, the new user option, when selected, simulating a mouse click at target coordinates of the display screen to initiate the menu unknown GUI component, the target coordinates being determined using the determined x-y coordinates for that GUI component;

activating a corresponding GUI component when a user option is selected in the updated user option menu in the screen reader; and

based on detecting closing of the application, deleting the updated user option menu from the computing environment.

7. The computer program product according to claim 6, wherein the screen scraping of the GUI comprises performing optical character recognition on a bit map of the GUI in order to identify GUI controls and labels.

8. The computer program product according to claim 6, wherein the screen scraping of the GUI comprises performing edge detection on a bit map of the GUI in order to identify GUI controls and labels.

9. The computer program product according to claim 6, wherein activating the corresponding GUI component comprises:

selecting the corresponding GUI component;

simulating left or right mouse clicks on the corresponding GUI component;

or

hovering a cursor over the corresponding GUI component.

10. The computer program product according to claim 6, wherein the screen scraping of the GUI comprises performing complementary edge detection and optical character recognition on a bit map of the GUI in order to identify GUI controls and labels.

11. A system for providing a temporary user interface menu for an application with a graphical user interface (GUI) within a computing environment, the system comprising:

a memory; and

a processing circuit communicatively coupled with the memory, wherein the system performs a method comprising;

detecting by the screen reader when the application is started, the application including a graphical user interface (GUI) of a display screen of the computing environment, and determining by the screen reader that an existing user option menu exists for the application;

screen scraping the GUI of the application using visual analysis of the GUI of the display screen to determine GUI components of the GUI, the GUI components including menu known GUI components and one or more menu unknown GUI components, the menu known GUI components being accessible from the existing user option menu, and the one or more menu unknown GUI components being unknown from and inaccessible from the existing user option menu and not corresponding to any existing user options in the existing user option menu determined for the application, and the screen scraping of the GUI using visual analysis comprising digital image processing of the GUI, the screen scraping including;

determining which GUI components of the GUI are the menu known GUI components, accessible from the existing user options in the existing user option menu; and

determining which GUI component(s) of the GUI are the one or more menu unknown GUI components, unknown and inaccessible from the existing user options in the existing user option menu;

determining x-y coordinates of the display screen for the one or more menu unknown GUI components identified based on the screen scraping;

creating an updated user option menu, from the existing user option menu, for the application in the screen reader comprising user options corresponding to each of the determined GUI components, including the menu known GUI components and the one or more menu unknown GUI components, the creating including;

creating a new user option in the existing user option menu that corresponds to a menu unknown GUI component of the one or more menu unknown GUI components identified based on the screen scraping, the new user option, when selected, simulating a mouse click at target coordinates of the display screen to initiate the menu unknown GUI component, the target coordinates being determined using the determined x-y coordinates for that GUI component;

activating a corresponding GUI component when a user option is selected in the updated user option menu in the screen reader; and

based on detecting closing of the application, deleting the updated user option menu from the computing environment.

12. The system according to claim 11, wherein the screen scraping of the GUI comprises performing optical character recognition on a bit map of the GUI in order to identify GUI controls and labels.

13. The system of claim 11, wherein the screen scraping of the GUI comprises performing edge detection on a bit map of the GUI in order to identify GUI controls and labels.

14. The system of claim 11, wherein activating the corresponding GUI component comprises:

selecting the corresponding GUI component;

simulating left or right mouse clicks on the corresponding GUI component;

or hovering a cursor over the corresponding GUI component.

15. The system of claim 11, wherein the screen scraping of the GUI comprises performing complementary edge detection and optical character recognition on a bit map of the GUI in order to identify GUI controls and labels.