Display-independent computerized guidance

• US 9,405,558 B2
• Filed: 08/24/2009
• Issued: 08/02/2016
• Est. Priority Date: 08/24/2009
• Status: Active Grant

• Current Assignee: Kryon Systems Ltd.
• Original Assignee: Kryon Systems Ltd.
• Inventors: Butin, Emma, Keinan, Oren
• Primary Examiner(s): LOWENBERGER, AARON J
• Application Number:
  US12/546,042
• Publication Number:
  US 20110047462A1
• Time in Patent Office: 2,535 Days
• Field of Search: 715/708, 715/704
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 3/04895 Guidance during keyboard input operation, e.g. prompting
• CPC Class Codes: G06F 9/453 Help systems

Abstract

A method for display-independent execution of a computerized guidance script, the method comprising capturing a screenshot of a graphical user interface (GUI) of a computerized application, analyzing the screenshot for matching a control of the screenshot with a control of the guidance script, and executing the guidance script, based on the matching, to operate the computerized application. Additionally, a display-independent computerized guidance engine, comprising a screenshot analysis engine and a guidance script execution engine.

Claims

1. A method for generating a display-independent computerized guidance script, the method comprising:

generating an application specific guidance script adapted to guide a user of a specific application through an operation sequence comprising a series of steps of operation of the specific application, said guidance script including actions pertaining to controls of the application and a control actuation sequence, wherein said generating an application specific guidance script comprises;

i. recording actions executed in the application during a performance of the operation sequence;

ii. recording first screenshots of a user interface of the specific application during the performance of the operation sequence;

iii. identifying within the first screenshots an individual control of the specific application associated with each of one or more of the recorded actions;

iv. extracting a separate image of each identified individual control from the recorded screenshots;

v. assigning each separate extracted image of each of the identified individual controls to a respective step of the operation sequence in the guidance script; and

vi. storing each separate extracted image of each of the identified individual controls together with the respective step of the operation sequence in the guidance script;

and

providing guidance script playback code adapted to cause a computer processor to;

capture second screenshots of a client user interface on a computing platform including an instance of the specific application;

identify, within the captured second screenshots, control elements of the instance of the specific application correlated to the identified individual controls stored in the guidance script so as to determine a screen location, within the client user interface, of each of the identified control elements; and

use the determined screen locations to demonstrate operation of the operation sequence within the specific application, wherein to demonstrate includes displaying a visual description, or imitating, activation of a control element.

2. The method according to claim 1, wherein the guidance script playback code is adapted to cause the computer processor to identify control elements according to a pre-defined accuracy level.

3. The method according to claim 1, wherein the guidance script playback code is adapted to cause the computer processor to match surroundings of the identified individual controls with surroundings of the control elements of the instance of the specific application.

4. The method according to claim 1, wherein to demonstrate further includes displaying help text associated with the control element in a guidance bubble.

5. The method according to claim 1, further comprising converting the first screenshots to inverted 1-bit versions.

6. The method according to claim 1, wherein to demonstrate operation of the operation sequence within the specific application further comprises activating a control of the specific application.

7. The method according to claim 6, wherein the activating of the control comprises imitating a cursor click.

8. The method according to claim 6, wherein the activating of the control comprises imitating keyboard input.

9. The method according to claim 1, wherein the control element is selected from a group consisting of:

an icon, a text, a button, an input box, a menu, a drop-down list, a slider, a scroll bar and a bar.

10. The method according to claim 1, the guidance script playback code is further adapted to cause the computer processor to activate a scroll bar for scrolling to a previously-invisible region of the specific application.

11. The method according to claim 1, further comprising converting the screenshot into a converted screenshot, wherein the converted screenshot is selected from the group consisting of:

a resized screenshot;

a portion of a screenshot;

a grey-scaled screenshot;

a 1-bit black-and-white screenshot;

an inverted version of a screenshot;

an inverted version of a grey-scaled screenshot;

or an inverted version of a black-and-white version of a screenshot.

12. A computerized system for generating a display-independent computerized guidance script, said system comprising:

first processing circuitry adapted to record;

(i) actions executed in a specific application, during a performance of an operation sequence comprising a series of steps of operation of the application;

(ii) first screenshots of a user interface of the specific application during the performance of the operation sequence;

screenshot analysis circuitry adapted to process the first screenshots and;

(i) identify within the first screenshots an individual control of the specific application associated with each of one or more of the recorded actions;

(ii) extract a separate image of each identified individual control from the recorded screenshots

script generation circuitry adapted to generate an application specific guidance script adapted to guide a user of the specific application through the operation sequence, said guidance script including actions pertaining to controls of the application and a control actuation sequence, wherein said generating an application specific guidance script comprises

i. assigning each separate extracted image of each of the identified individual controls to a respective step of the operation sequence in the guidance script; and

ii. storing each separate extracted image of each of the identified individual controls together with the respective step of the operation sequence in the guidance script;

and

a tangible medium containing guidance script playback code adapted to cause a computer processor to;

capture second screenshots of a client user interface on a computing platform including an instance of the specific application;

identify, within the captured second screenshots, control elements of the instance of the specific application correlated to the identified individual controls stored in the guidance script so as to determine a screen location, within the client user interface, of each of the identified control elements; and

use the determined screen locations to demonstrate operation of the operation sequence within the specific application, wherein to demonstrate includes displaying a visual description, or imitating, activation of a control element.

13. The system according to claim 12, wherein the guidance script playback code is adapted to cause the computer processor to identify control elements according to a pre-defined accuracy level.

14. The system according to claim 12, wherein the guidance script playback code is adapted to cause the computer processor to match surroundings of the identified individual controls with surroundings of the control elements of the instance of the specific application.

15. The system according to claim 12, wherein to demonstrate the operation sequence within the specific application further includes displaying help text associated with the control element in a guidance bubble.

16. The system according to claim 12, wherein to demonstrate operation of the operation sequence within the specific application further comprises activating a control of the specific application.

17. The system according to claim 16, wherein the activating of the control comprises imitating a cursor click.

18. The system according to claim 16, wherein the activating of the control comprises imitating keyboard input.

19. The system according to claim 12, wherein the guidance script playback code is further adapted to cause the computer processor to activate a scroll bar for scrolling to a previously-invisible region of the specific application.

20. A method for display-independent execution of a computerized guidance script, the method comprising:

retrieving from a guidance script repository an application specific guidance script adapted to guide a user of a specific application through an operation sequence comprising a series of steps of operation of the specific application, said guidance script including actions pertaining to controls of the application and a control actuation sequence, wherein said application specific guidance script includes;

i. actions recorded during execution of the operation sequence on a first computing device;

ii. separate images of each of a set of individual controls of the specific application, extracted from first screenshots recorded during the execution of the operation sequence on the first computing device,

wherein each of the separate images is assigned to a respective step of the operation sequence in the guidance script and stored together with the respective step in the guidance script;

capturing second screenshots of a client user interface on a computing platform including an instance of the specific application;

identifying, within the captured second screenshots, control elements of the instance of the specific application correlated to the identified individual controls stored in the guidance script so as to determine a screen location, within the client user interface, of each of the identified control elements; and

using the determined screen locations to demonstrate operation of the operation sequence within the specific application, wherein to demonstrate includes displaying a visual description, or imitating, activation of a control element.