Display-independent recognition of graphical user interface control

• US 9,703,462 B2
• Filed: 11/30/2014
• Issued: 07/11/2017
• Est. Priority Date: 08/24/2009
• Status: Active Grant

• Current Assignee: Kryon Systems Ltd.
• Original Assignee: Kryon Systems Ltd.
• Inventors: Butin, Emma, Keinan, Oren, Rabinovich, Alon, Keinan, Erez
• Primary Examiner(s): Titcomb, William
• Application Number:
  US14/556,158
• Publication Number:
  US 20150089363A1
• Time in Patent Office: 954 Days
• Field of Search: 715704
• CPC Class Codes: G06F 11/3692 for test results analysis
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04855 Interaction with scrollbars
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/453 Help systems

Abstract

A method for display-independent recognition of a graphical user interface (GUI) control, the method comprising capturing a screenshot of a GUI of a computerized application, determining a cursor position in the screenshot, analyzing an area surrounding the cursor position, to identify a GUI control, and storing an image of the GUI control.

Claims

1. A method for automatic extraction of an image of a control in a graphical user interface (GUI), the method comprising:

capturing a first screenshot of a GUI of a computerized application, and automatically analyzing said first screenshot to identify a scroll bar by extracting at least one parameter of the scroll bar from within the first screenshot;

if a scrollbar is identified;

(1) receiving from a user coordinates of at least one position in said first screenshot;

analyzing an area surrounding said at least one position, to identify at least one GUI control; and

(2) storing an image of said at least one GUI control substantially in conjunction with at least one parameter of the scroll bar; and

if a scrollbar is not identified;

(1) receiving coordinates of at least one position from a user in said first screenshot;

analyzing an area surrounding said at least one position, to identify at least one GUI control; and

(2) storing an image of said at least one GUI control separate from the first screenshot of the GUI from which the image of the GUI control is taken.

2. The method according to claim 1, further comprising:

creating an operation sequence of the computerized application; and

associating the at least one GUI control, with a step of an operation sequence, to produce a computerized guidance script adapted to provide guidance to a user including moving a cursor associated with an input device, to imitate user-controlled movement of said cursor via said input device.

3. The method according to claim 2, wherein the at least one GUI control is selected from a group consisting of:

an icon, a text, a button, an input box, a menu, a drop-down list, a slider, another scroll bar and a bar.

4. The method according to claim 2, wherein the receiving of the coordinates comprises detecting where a user cursor click is performed.

5. The method according to claim 2, further comprising converting at least one of said first screenshot and said second screenshot to a 1-bit version.

6. The method according to claim 2, further comprising converting at least one of said first screenshot and said second screenshot to an inverted 1-bit version.

7. The method according to claim 2, wherein the storing further comprises storing an image of surroundings of the at least one GUI control.

8. The method according to claim 2, wherein the analyzing of the area surrounding each of said at least one position comprises employing optical character recognition (OCR).

9. A non-transitory computer-readable medium having stored therein instructions for automatic extraction of an image of at least one GUI control in a graphical user interface (GUI), wherein the instructions, when executed by the computer, cause the computer to:

capture a first image of a GUI of a computerized application, and automatically analyze said first image, to identify a scroll bar;

if a scroll bar is identified;

(1) if stored scrollbar parameters indicate that scrolling is required, automatically operate said scroll bar to reveal a previously-invisible region of said GUI, and capture a second image of said previously-invisible region;

(2) analyze said first or second image to identify at least one GUI control; and

if a scroll bar is not identified, analyze in said first image to identify at least one GUI control.

10. The non-transitory computer readable medium according to claim 9, wherein the instructions further cause the computer to:

create an operation sequence of the computerized application; and

associate the at least one GUI control, with a step of an operation sequence, to produce a computerized guidance script adapted to provide guidance to a user including moving a cursor associated with an input device, to imitate user-controlled movement of said cursor via said input device.

11. The non-transitory computer-readable medium according to claim 10, wherein said at least one GUI control is selected from a group consisting of:

an icon, a text, a button, an input box, a menu, a drop-down list, a slider, another scroll bar and a bar.

12. The non-transitory computer-readable medium according to claim 10, wherein the instructions, when executed by the computer, further cause the computer to convert at least one of said first image and said second image to a 1-bit version.

13. The non-transitory computer-readable medium according to claim 10, wherein the instructions, when executed by the computer, further cause the computer to convert at least one of said first image and said second image to an inverted 1-bit version.

14. The non-transitory computer-readable medium according to claim 10, wherein the instructions, when executed by the computer, further cause the computer to identify a cursor click on said at least one GUI control.

15. The non-transitory computer-readable medium according to claim 10, wherein the instructions, when executed by the computer, further cause the computer to identify keyboard input into said at least one GUI control.

16. The non-transitory computer-readable medium according to claim 10, wherein analyzing said area surrounding said position comprises employing an OCR engine.

17. A non-transitory computer-readable medium having stored therein instructions for recording a display-independent computerized guidance script, wherein the instructions, when executed by a computer, cause the computer to:

capture a screenshot of a GUI of a computerized application running on the computer;

analyze said screenshot to receive from a user coordinates of at least one position in said screenshot and analyzing an area surrounding said at least one position, to identify at least one GUI control;

if a scrollbar is identified, store an image of said at least one GUI control substantially in conjunction with at least one parameter of the scroll bar;

and

if a scrollbar is not identified, store an image of said at least one GUI control separate from the screenshot of the GUI from which the image of the GUI control is taken.

18. The non-transitory computer-readable medium according to claim 17, wherein said instructions, when executed by the computer, further cause the computer to automatically analyze said screenshot to identify a scroll bar by extracting at least one parameter of the scroll bar from within the screenshot, wherein the extracted parameter comprises at least one of an island, a triangle, or an arrow.