Software robots for programmatically controlling computer programs to perform tasks

• US 10,671,235 B2
• Filed: 03/03/2016
• Issued: 06/02/2020
• Est. Priority Date: 03/03/2015
• Status: Active Grant

• Current Assignee: Soroco Private Limited
• Original Assignee: Soroco Private Limited
• Inventors: Nychis, George Peter, Murty, Rohan Narayan
• Primary Examiner(s): Hong, Stephen S
• Assistant Examiner(s): Sheffield, Hope C
• Application Number:
  US15/059,770
• Publication Number:
  US 20160259653A1
• Time in Patent Office: 1,552 Days
• Field of Search: 715704
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/32 with visual or acoustical indication of the functioning of the machine
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/36 Preventing errors by testing or debugging software
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/44 Arrangements for executing specific programs
• CPC Class Codes: G06F 9/4488 Object-oriented
• CPC Class Codes: G06F 9/451 : Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/455 : Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
• CPC Class Codes: G06F 9/45508 : Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation
• CPC Class Codes: G06F 9/45512 : Command shells
• CPC Class Codes: G06F 9/45533 : Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/45558 : Hypervisor-specific management and integration aspects

Abstract

A system comprising at least one computer hardware processor; and at least one computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of a first application program and a second application program; controlling the first application program to perform a first sub-task of a task at least in part by using the object hierarchy; and controlling the second application program to perform a second sub-task of the task at least in part by using the object hierarchy, wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.

Claims

1. A system, comprising:

at least one computer hardware processor; and

at least one computer-readable storage medium storing;

a computer program for controlling multiple application programs to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and

processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform;

generating an object hierarchy comprising;

a first plurality of objects corresponding to a first plurality of active graphical user interface (GUI) elements of the first application program, wherein;

 

the first application program is configured to execute by using a first GUI application library, and

 

the first plurality of active GUI elements comprise at least one active GUI element contained within an outer container of a first graphical user interface through which user input for controlling the first application program can be provided, and

a second plurality of objects corresponding to a second plurality of active GUI elements of the second application program, wherein;

 

the second application program is configured to execute by using a second GUI application library that is different than the first GUI application library, and

 

the second plurality of active GUI elements comprise at least one active GUI element within an outer container of a second graphical user interface through which user input for controlling the second application program can be provided;

controlling the first application program to perform the first sub-task using the computer program and the object hierarchy at least in part by;

using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform a first action in furtherance of the first sub-task, the first action programmatically emulating a user'"'"'s interaction with the first active GUI element, wherein programmatically emulating the user'"'"'s interaction with the first active GUI element comprises invoking one or more functions of the first application program or an operating system managing execution of the first application program to cause the first application program to perform the first action using the first active GUI element; and

controlling the second application program to perform the second sub-task using the computer program and the object hierarchy at least in part by;

using a second object of the second plurality of objects to cause, via a second active GUI element of the second plurality of active GUI elements, the second application program to at least partially perform a second action in furtherance of the second sub-task, the second action programmatically emulating the user'"'"'s interaction with the second active GUI element, wherein programmatically emulating the user'"'"'s interaction with the second active GUI element comprises invoking one or more functions of the second application program or an operating system managing execution of the second application program to cause the second application program to perform the second action using the second active GUI element.

2. The system of claim 1, wherein, when the controlling of the second application is performed, the object hierarchy comprises the first plurality of objects corresponding to the first plurality of active GUI elements of the first application program and the second plurality of objects corresponding to the second plurality of active GUI elements of the second application program.

3. The system of claim 1, wherein controlling the first application program to perform the first sub-task comprises:

automatically accessing the first object in the object hierarchy corresponding to the first active GUI element of the first application program; and

automatically using the first object to cause the first application to perform the first action in furtherance of the first sub-task.

4. The system of claim 3, wherein controlling the second application program to perform the second sub-task comprises:

automatically accessing the second object in the object hierarchy corresponding to the second active GUI element of the second application program; and

automatically using the second object to cause the second application to perform the second action in furtherance of the second sub-task.

5. The system of claim 4, wherein:

automatically accessing the first object comprises searching for the first object in the object hierarchy using a search grammar; and

automatically accessing the second object comprises searching for the second object in the object hierarchy using the search grammar.

6. The system of claim 1, wherein the first and second application programs are configured to execute using different operating systems.

7. The system of claim 1, wherein the first application program is a WINDOWS application program configured to execute on a WINDOWS operating system and the second application program is a web-based application program configured to execute at least in part by using a web-browser application program.

8. The system of claim 1, wherein the first application program is a WINDOWS application program and the second application program is a JAVA application program.

9. The system of claim 1, wherein the first application program is an application program selected from the group consisting of an application program configured to execute on a mobile device, an application program configured to execute on a server, a WINDOWS application program, a LINUX GTK application program, an application program having component object model (COM) access, a JAVA application program, a FLASH application program, a SILVERLIGHT application program, a SAP application program, an ANDROID application program, an ORACLE application program, a web-based application program, an HTML application program, a JAVASCRIPT application program, an AJAX application program, a WINDOWS MOBILE application program, an IOS application program, a SOLARIS application program, an IBM AIX application program, and a proxy application program that is used as a proxy for controlling another application program.

10. The system of claim 1, wherein the first and second application programs are configured to execute at least in part by using different software platforms.

11. The system of claim 10, wherein the different software platforms include a first software platform selected from the group consisting of a WINDOWS operating system, a LINUX operating system, a JAVA virtual machine, a web-browser, an operating system configured to execute on a mobile device, a SAP platform, and an ORACLE platform.

12. A system, comprising:

at least one computer hardware processor; and

at least one computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform;

generating an object hierarchy comprising;

a first plurality of objects corresponding to a first plurality of active graphical user interface (GUI) elements of a first application program, wherein;

the first application program is configured to execute by using a first GUI application library, and

the first plurality of active GUI elements comprise at least one active GUI element contained within an outer container of a first graphical user interface through which user input for controlling the first application program can be provided, and

a second plurality of objects corresponding to a second plurality of active GUI elements of a second application program, wherein;

the second application program is configured to execute by using a second GUI application library that is different than the first GUI application library, and

the second plurality of active GUI elements comprise at least one active GUI element within an outer container of a second graphical user interface through which user input for controlling the second application program can be provided;

controlling the first application program to perform a first sub-task of a task using the object hierarchy at least in part by;

using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform a first action in furtherance of the first sub-task, the first action programmatically emulating a user'"'"'s interaction with the first active GUI element, wherein programmatically emulating the user'"'"'s interaction with the first active GUI element comprises invoking one or more functions of the first application program or an operating system managing execution of the first application program to cause the first application program to perform the first action using the first active GUI element; and

controlling the second application program to perform a second sub-task of the task using the object hierarchy at least in part by;

using a second object of the second plurality of objects to cause, via a second active GUI element of the second plurality of active GUI elements, the second application program to at least partially perform a second action in furtherance of the second sub-task, the second action programmatically emulating the user'"'"'s interaction with the second active GUI element, wherein programmatically emulating the user'"'"'s interaction with the second active GUI element comprises invoking one or more functions of the second application program or an operating system managing execution of the second application program to cause the second application program to perform the second action using the second active GUI element.

13. The system of claim 12, wherein the first and second application programs are configured to execute using different operating systems.

14. The system of claim 12, wherein the first application program is a WINDOWS application program configured to execute on a WINDOWS operating system and the second application program is a web-based application program configured to execute at least in part by using a web-browser application program.

15. The system of claim 12, wherein the first application program is a WINDOWS application program and the second application program is a JAVA application program.

16. The system of claim 12, wherein the first application program is an application program selected from the group consisting of an application program configured to execute on a mobile device, an application program configured to execute on a server, a WINDOWS application program, a LINUX GTK application program, an application program having component object model (COM) access, a JAVA application program, a FLASH application program, a SILVERLIGHT application program, a SAP application program, an ANDROID application program, an ORACLE application program, a web-based application program, an HTML application program, a JAVASCRIPT application program, an AJAX application program, a WINDOWS MOBILE application program, an IOS application program, a SOLARIS application program, an IBM AIX application program, and a proxy application program that is used as a proxy for controlling another application program.

17. The system of claim 12, wherein the first and second application programs are configured to execute at least in part by using different software platforms.

18. The system of claim 17, wherein the different software platforms include a first software platform selected from the group consisting of a WINDOWS operating system, a LINUX operating system, a JAVA virtual machine, a web-browser, an operating system configured to execute on a mobile device, a SAP platform, and an ORACLE platform.

19. The system of claim 1, wherein the first object comprises information indicating a displayed location of the first active GUI element.

20. The system of claim 19, wherein the second object comprises information indicating a displayed location of the second active GUI element.

21. A method, comprising:

using at least one computer hardware processor to perform;

generating an object hierarchy comprising;

a first plurality of objects corresponding to a first plurality of active graphical user interface (GUI) elements of a first application program, wherein;

the first application program is configured to execute by using a first GUI application library, and

the first plurality of active GUI elements comprise at least one active GUI element contained within an outer container of a first graphical user interface through which user input for controlling the first application program can be provided, and

a second plurality of objects corresponding to a second plurality of active GUI elements of a second application program, wherein;

the second application program is configured to execute by using a second GUI application library that is different than the first GUI application library, and

the second plurality of active GUI elements comprise at least one active GUI element within an outer container of a second graphical user interface through which user input for controlling the second application program can be provided;

controlling the first application program to perform a first sub-task of a task using the object hierarchy at least in part by;

using a first object of the first plurality of objects to cause, via a first active GUI element of the first active GUI elements, the first application program to at least partially perform a first action in furtherance of the first sub-task, the first action programmatically emulating a user'"'"'s interaction with the first active GUI element wherein programmatically emulating the user'"'"'s interaction with the first active GUI element comprises invoking one or more functions of the first application program or an operating system managing execution of the first application program to cause the first application program to perform the first action using the first active GUI element; and

controlling the second application program to perform a second sub-task of the task using the object hierarchy at least in part by;

using a second object of the second plurality of objects to cause, via a second active GUI element of the second active GUI elements, the second application program to at least partially perform a second action in furtherance of the second sub-task, the second action programmatically emulating the user'"'"'s interaction with the second active GUI element, wherein programmatically emulating the user'"'"'s interaction with the second active GUI element comprises invoking one or more functions of the second application program or an operating system managing execution of the second application program to cause the second application program to perform the second action using the second active GUI element.