Software robots for programmatically controlling computer programs to perform tasks

• US 10,310,701 B2
• Filed: 04/02/2018
• Issued: 06/04/2019
• Est. Priority Date: 03/03/2015
• Status: Active Grant

• Current Assignee: Soroco Private Limited
• Original Assignee: Soroco Private Limited
• Inventors: Nychis, George Peter, Murty, Rohan Narayan
• Primary Examiner(s): Aponte, Francisco J
• Assistant Examiner(s): Gooray, Mark A
• Application Number:
  US15/942,930
• Publication Number:
  US 20180225010A1
• Time in Patent Office: 428 Days
• Field of Search: None
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/32 with visual or acoustical indication of the functioning of the machine
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/36 Preventing errors by testing or debugging software
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/44 Arrangements for executing specific programs
• CPC Class Codes: G06F 9/4488 Object-oriented
• CPC Class Codes: G06F 9/451 : Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/455 : Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
• CPC Class Codes: G06F 9/45508 : Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation
• CPC Class Codes: G06F 9/45512 : Command shells
• CPC Class Codes: G06F 9/45533 : Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/45558 : Hypervisor-specific management and integration aspects

Abstract

A system comprising at least one computer hardware processor configured to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program; controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions.

Claims

1. A system for controlling at least one application program for performing a task comprising a sequence of actions, the at least one application program including a first application program and a second application program, the first and second application programs configured to execute at least in part by using different graphical user interface (GUI) application libraries, the system comprising:

at least one computer hardware processor; and

at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform;

generating an object hierarchy comprising a plurality of objects corresponding to active GUI elements of the at least one application program, the generating comprising;

identifying a first plurality of active GUI elements of the first application program and a second plurality of active GUI elements of the second application program, wherein the first application program is configured to execute by using a first GUI application library and the second application program is configured to execute by using a second GUI application library that is different than the first GUI application library;

generating a first portion of the object hierarchy comprising a first plurality of objects corresponding to the first plurality of active GUI elements of the first application program and a second portion of the object hierarchy comprising a second plurality of objects corresponding to the second plurality of active GUI elements of the second application program,

controlling the first application program to perform a first action in the sequence of actions using the first portion of the object hierarchy at least in part by;

using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform the first action, the first action programmatically emulating a user'"'"'s interaction with the first active GUI element; and

after the first application program performs the first action, refreshing the object hierarchy to obtain a first refreshed object hierarchy comprising a third plurality of objects corresponding to a third plurality of active GUI elements reflecting changes in a user interface of the first application program; and

controlling the second application program to perform a second action in the sequence of actions using the second portion of the object hierarchy at least in part by;

using a second object of the second plurality of objects to cause, via a second active GUI element of the second plurality of active GUI elements, the second application program to at least partially perform the second action, the second action programmatically emulating a user'"'"'s interaction with the second active GUI element; and

after the second application program performs the second action, refreshing the object hierarchy to obtain a second refreshed object hierarchy comprising a fourth plurality of objects corresponding to a fourth plurality of active GUI elements reflecting changes in a user interface of the second application program.

2. The system of claim 1, wherein refreshing the object hierarchy comprises:

refreshing at least one object in the object hierarchy, adding at least one object to the object hierarchy, and/or removing at least one object from the object hierarchy.

3. The system of claim 1, wherein refreshing the object hierarchy comprises:

refreshing only a subset of objects in the object hierarchy.

4. The system of claim 1, wherein controlling the first application program to perform the first action in the sequence of actions comprises:

accessing the first object in the object hierarchy corresponding to the first active GUI element of the first application program.

5. The system of claim 1, wherein refreshing the object hierarchy comprises:

refreshing only one or more parent objects of the first object in the object hierarchy, one or more descendant objects of the first object in the object hierarchy, and/or the first object.

6. The system of claim 1, wherein refreshing the object hierarchy comprises:

adding a new object to the object hierarchy, wherein the new object is associated with an active GUI element of the first application program.

7. The system of claim 1, wherein the first active GUI element of the first application program is a visible GUI element of the first application program.

8. The system of claim 1, further comprising a display, wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:

displaying a GUI of the first application program on the display during the performance of at least some of the sequence of actions.

9. The system of claim 1, wherein generating the object hierarchy comprises:

obtaining a first hierarchical representation of the first plurality of active GUI elements of the first application program from an operating system; and

identifying the first plurality of active GUI elements of the first application program using the first hierarchical representation.

10. The system of claim 9, wherein obtaining the first hierarchical representation is performed by using the WINDOWS Automation API.

11. The system of claim 1, wherein the processor-executable instructions, when executed by the at least one computer hardware processor, further cause the at least one computer hardware processor to perform:

controlling the first application program to perform a third action in the sequence of actions using the first refreshed object hierarchy at least in part by;

using a third object of the third plurality of objects to cause, via a third active GUI element of the third plurality of active GUI elements, the first application program to at least partially perform the third action, the third action programmatically emulating a user'"'"'s interaction with the third active GUI element.

12. The system of claim 1, wherein the processor-executable instructions, when executed by the at least one computer hardware processor, further cause the at least one computer hardware processor to perform:

controlling the second application program to perform a fourth action in the sequence of actions using the second refreshed object hierarchy at least in part by;

using a fourth object of the fourth plurality of objects to cause, via a fourth active GUI element of the fourth plurality of active GUI elements, the second application program to at least partially perform the fourth action, the fourth action programmatically emulating a user'"'"'s interaction with the fourth active GUI element.

13. The system of claim 1, wherein the first and second application programs are configured to execute using different operating systems.

14. The system of claim 1, wherein the first application program is a WINDOWS application program configured to execute on a WINDOWS operating system and the second application program is a web-based application program configured to execute at least in part by using a web-browser application program.

15. The system of claim 1, wherein the first application program is a WINDOWS application program and the second application program is a JAVA application program.

16. The system of claim 1, wherein the first and second application programs are configured to execute at least in part by using different software platforms.

17. The system of claim 16, wherein the different software platforms include a first software platform selected from the group consisting of a WINDOWS operating system, a LINUX operating system, a JAVA virtual machine, a web-browser, an operating system configured to execute on a mobile device, a SAP platform, and an ORACLE platform.

18. A method for controlling at least one application program for performing a task comprising a sequence of actions, the at least one application program including a first application program and a second application program, the first and second application programs configured to execute at least in part by using different graphical user interface (GUI) application libraries, the method comprising:

using at least one computer hardware processor to perform;

generating an object hierarchy comprising a plurality of objects corresponding to active GUI elements of the at least one application program, the generating comprising;

identifying a first plurality of active GUI elements of the first application program and a second plurality of active GUI elements of the second application program, wherein the first application program is configured to execute by using a first GUI application library and the second application program is configured to execute by using a second GUI application library that is different than the first GUI application library;

generating a first portion of the object hierarchy comprising a first plurality of objects corresponding to the first plurality of active GUI elements of the first application program and a second portion of the object hierarchy comprising a second plurality of objects corresponding to the second plurality of active GUI elements of the second application program;

controlling the first application program to perform a first action in the sequence of actions using the first portion of the object hierarchy at least in part by;

using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform the first action, the first action programmatically emulating a user'"'"'s interaction with the first active GUI element; and

after the first application program performs the first action, refreshing the object hierarchy to obtain a first refreshed object hierarchy comprising a third plurality of objects corresponding to a third plurality of active GUI elements reflecting changes in a user interface of the first application program; and

controlling the second application program to perform a second action in the sequence of actions using the second portion of the object hierarchy at least in part by;

using a second object of the second plurality of objects to cause, via a second active GUI element of the second plurality of active GUI elements, the second application program to at least partially perform the second action, the second action programmatically emulating a user'"'"'s interaction with the second active GUI element; and

after the second application program performs the second action, refreshing the object hierarchy to obtain a second refreshed object hierarchy comprising a fourth plurality of objects corresponding to a fourth plurality of active GUI elements reflecting changes in a user interface of the second application program.

19. The method of claim 18, wherein refreshing the object hierarchy comprises:

refreshing at least one object in the object hierarchy, adding at least one object to the object hierarchy, and/or removing at least one object from the object hierarchy.

20. The method of claim 18, wherein refreshing the object hierarchy comprises:

refreshing only a subset of objects in the object hierarchy.

21. The method of claim 18, wherein controlling the first application program to perform the first action in the sequence of actions comprises:

accessing the first object in the object hierarchy corresponding to the first active GUI element of the first application program.

22. The method of claim 21, wherein refreshing the object hierarchy comprises:

refreshing only one or more parent objects of the first object in the object hierarchy, one or more descendant objects of the first object in the object hierarchy, and/or the first object.

23. The method of claim 18, wherein generating the object hierarchy comprises:

obtaining a first hierarchical representation of the first plurality of active GUI elements of the first application program from an operating system; and

identifying the first plurality of active GUI elements of the first application program using the first hierarchical representation.

24. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, causes the at least one computer hardware processor to perform a method for controlling at least one application program for performing a task comprising a sequence of actions, the at least one application program including a first application program and a second application program, the first and second application programs configured to execute at least in part by using different graphical user interface (GUI) application libraries, the method comprising:

generating an object hierarchy comprising a plurality of objects corresponding to active GUI elements of the at least one application program, the generating comprising;

identifying a first plurality of active GUI elements of the first application program and a second plurality of active GUI elements of the second application program, wherein the first application program is configured to execute by using a first GUI application library and the second application program is configured to execute by using a second GUI application library that is different than the first GUI application library;

generating a first portion of the object hierarchy comprising a first plurality of objects corresponding to the first plurality of active GUI elements of the first application program and a second portion of the object hierarchy comprising a second plurality of objects corresponding to the second plurality of active GUI elements of the second application program;

controlling the first application program to perform a first action in the sequence of actions using the first portion of the object hierarchy at least in part by;

using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform the first action, the first action programmatically emulating a user'"'"'s interaction with the first active GUI element; and

after the first application program performs the first action, refreshing the object hierarchy to obtain a first refreshed object hierarchy comprising a third plurality of objects corresponding to a third plurality of active GUI elements reflecting changes in a user interface of the first application program;

controlling the second application program to perform a second action in the sequence of actions using the second portion of the object hierarchy at least in part by;

using a second object of the second plurality of objects to cause, via a second active GUI element of the second plurality of active GUI elements, the second application program to at least partially perform the second action, the second action programmatically emulating a user'"'"'s interaction with the second active GUI element; and

after the second application program performs the second action, refreshing the object hierarchy to obtain a second refreshed object hierarchy comprising a fourth plurality of objects corresponding to a fourth plurality of active GUI elements reflecting changes in a user interface of the second application program.

25. The at least one non-transitory computer-readable storage medium of claim 24, wherein refreshing the object hierarchy comprises refreshing only a subset of objects in the object hierarchy.