Software robots for programmatically controlling computer programs to perform tasks

• US 10,585,548 B2
• Filed: 03/03/2016
• Issued: 03/10/2020
• Est. Priority Date: 03/03/2015
• Status: Active Grant

• Current Assignee: Soroco Private Limited
• Original Assignee: Soroco Private Limited
• Inventors: Nychis, George Peter, Murty, Rohan Narayan
• Primary Examiner(s): Hong, Stephen S
• Assistant Examiner(s): Sheffield, Hope C
• Application Number:
  US15/059,883
• Publication Number:
  US 20160259654A1
• Time in Patent Office: 1,468 Days
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/32 with visual or acoustical indication of the functioning of the machine
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/36 Preventing errors by testing or debugging software
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/44 Arrangements for executing specific programs
• CPC Class Codes: G06F 9/4488 Object-oriented
• CPC Class Codes: G06F 9/451 : Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/455 : Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
• CPC Class Codes: G06F 9/45508 : Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation
• CPC Class Codes: G06F 9/45512 : Command shells
• CPC Class Codes: G06F 9/45533 : Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/45558 : Hypervisor-specific management and integration aspects

Abstract

A system for remotely controlling multiple application programs executing on multiple respective physical computing devices, the system comprising a first computing device, comprising a first processor, and configured to execute a first application program; a second computing device, comprising a second processor, and configured to execute a second application program; and a controller configured to perform: generating, based on first information obtained from the first computing device and second information obtained from the second computing device, a global object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program and the second application program; controlling the first application program to perform the first sub-task at least in part by using the global object hierarchy; and controlling the second application program to perform the second sub-task at least in part by using the global object hierarchy.

Claims

1. A system for remotely controlling multiple application programs executing on multiple respective virtual machines to perform a task, the system comprising:

a computing device, comprising a first processor, and configured to execute;

a first virtual machine configured to execute a first application program; and

a second virtual machine configured to execute a second application program; and

a controller communicatively coupled to the computing device and comprising a second processor, the controller configured to perform;

generating, based on first information obtained from the first virtual machine and second information obtained from the second virtual machine, a global object hierarchy comprising;

a first plurality of objects corresponding to a first plurality of active graphical user interface (GUI) elements of the first application program executing on the first virtual machine, wherein the first plurality of active GUI elements comprises at least one active GUI element contained within an outer container of a first graphical user interface through which user input for controlling the first application program can be provided, and

a second plurality of objects corresponding to a second plurality of active GUI elements of the second application program executing on the second virtual machine, wherein the second plurality of active GUI elements comprises at least one active GUI element contained within an outer container of a second graphical user interface through which user input for controlling the second application program can be provided;

controlling the first application program to perform a first sub-task of the task using the global object hierarchy at least in part by;

using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform a first action in furtherance of the first sub-task, the first action programmatically emulating a user'"'"'s interaction with the first active GUI element, wherein programmatically emulating the user'"'"'s interaction with the first active GUI element comprises invoking one or more functions of the first application program or an operating system managing execution of the first application program to cause the first application program to perform the first action using the first active GUI element; and

controlling the second application program to perform a second sub-task of the task using the global object hierarchy at least in part by;

using a second object of the second plurality of objects to cause, via a second active GUI element of the second plurality of active GUI elements, the second application program to at least partially perform a second action in furtherance of the second sub-task, the second action programmatically emulating the user'"'"'s interaction with the second active GUI element, wherein programmatically emulating the user'"'"'s interaction with the second active GUI element comprises invoking one or more functions of the second application program or an operating system managing execution of the second application program to cause the second application program to perform the second action using the second active GUI element.

2. The system of claim 1,

wherein the first information includes information generated using the first virtual machine about the first plurality of active GUI elements of the first application program;

wherein the second information includes information generated using the second virtual machine about the second plurality of active GUI elements of the second application program; and

wherein generating the global object hierarchy comprises combining a first local object hierarchy and a second local object hierarchy to obtain the global object hierarchy.

3. The system of claim 1, wherein controlling the first application program comprises:

accessing, in the global object hierarchy, the first object corresponding to the first active GUI element of the first application program.

4. The system of claim 3, wherein the first virtual machine maintains a local object hierarchy including a first local object corresponding to the first object in the global object hierarchy, and wherein using the first object comprises providing an indication to the first virtual machine to use the first local object to cause the first application program to at least partially perform the first action.

5. The system of claim 1, wherein the computing device is configured to execute a hypervisor to manage execution of the first virtual machine and the second virtual machine.

6. A system for remotely controlling one or more application programs executing on a respective one or more virtual machines, the system comprising:

a controller, comprising a first processor, configured to perform;

accessing a computer program for controlling at least one application program to perform a task including a first sub-task to be performed by a first application program executing on a first virtual machine;

identifying, using the computer program, a first action to perform in furtherance of the first sub-task; and

providing an indication to at least one computing device executing the first virtual machine to control the first application program to at least partially perform the first action; and

the at least one computing device, communicatively coupled to the controller, comprising a second processor configured to execute the first virtual machine and to perform;

generating a first object hierarchy comprising a first plurality of objects corresponding to a first plurality of active graphical user interface (GUI) elements of the first application program, wherein the first plurality of active GUI elements comprises at least one active GUI element contained within an outer container of a first graphical interface through which user input for controlling the first application program can be provided; and

in response to receiving the indication from the controller to at least partially perform the first action, using the first object hierarchy to cause the first application program to at least partially perform the first action at least in part by;

using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform the first action in furtherance of the first sub-task, the first action programmatically emulating a user'"'"'s interaction with the first active GUI element, wherein programmatically emulating the user'"'"'s interaction with the first active GUI element comprises invoking one or more functions of the first application program or an operating system managing execution of the first application program to cause the first application program to perform the first action using the first active GUI element.

7. The system of claim 6,

wherein the task further includes a second sub-task to be performed by a second application program executing on a second virtual machine;

and

wherein the controller is further configured to provide an indication to the second virtual machine to control the second application program to at least partially perform a second action in furtherance of the second sub-task.

8. The system of claim 7, wherein the at least one computing device is configured to execute the second virtual machine and to perform:

generating a second object hierarchy comprising a second plurality of objects corresponding to a second plurality of active GUI elements of the second application program; and

in response to receiving the indication from the controller to at least partially perform the second action, using the second object hierarchy to cause the second application program to at least partially perform the second action.

9. The system of claim 6, wherein using the first object hierarchy to cause the first application program to at least partially perform the first action comprises:

accessing in the first object hierarchy, the first object corresponding to the first active GUI element of the first application program.

10. The system of claim 8, wherein the at least one computing device is configured to execute a hypervisor to manage execution of the first virtual machine and the second virtual machine.

11. A system for remotely controlling multiple application programs executing on multiple respective physical computing devices to perform a task, the system comprising:

a first computing device, comprising a first processor, and configured to execute a first application program;

a second computing device, comprising a second processor, and configured to execute a second application program; and

a controller communicatively coupled to the first and second computing devices and comprising a third processor, the controller configured to perform;

generating, based on first information obtained from the first computing device and second information obtained from the second computing device, a global object hierarchy comprising;

a first plurality of objects corresponding to a first plurality of active graphical user interface (GUI) elements of the first application program executing on the first computing device, wherein the first plurality of active GUI elements comprises at least one active GUI element contained within an outer container of a first graphical user interface through which user input for controlling the first application program can be provided, and

a second plurality of objects corresponding to a second plurality of active GUI elements of the second application program executing on the second computing device, wherein the second plurality of active GUI elements comprises at least one active GUI element contained within an outer container of a second graphical user interface through which user input for controlling the second application program can be provided;

controlling the first application program to perform a first sub-task of the task using the global object hierarchy at least in part by;

using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform a first action in furtherance of the first sub-task, the first action programmatically emulating a user'"'"'s interaction with the first active GUI element, wherein programmatically emulating the user'"'"'s interaction with the first active GUI element comprises invoking one or more functions of the first application program or an operating system managing execution of the first application program to cause the first application program to perform the first action using the first active GUI element; and

controlling the second application program to perform a second sub-task of the task using the global object hierarchy at least in part by;

using a second object of the second plurality of objects to cause, via a second active GUI element of the second plurality of active GUI elements, the second application program to at least partially perform a second action in furtherance of the second sub-task, the second action programmatically emulating the user'"'"'s interaction with the second active GUI element, wherein programmatically emulating the user'"'"'s interaction with the second active GUI element comprises invoking one or more functions of the second application program or an operating system managing execution of the second application program to cause the second application program to perform the second action using the second active GUI element.

12. The system of claim 11,

wherein the first information includes information generated on the first computing device about the first plurality of active GUI elements of the first application program;

wherein the second information includes information generated on the second computing device about the second plurality of active GUI elements of the second application program; and

wherein generating the global object hierarchy comprises combining a first local object hierarchy and a local second object hierarchy to obtain the global object hierarchy.

13. The system of claim 11, wherein controlling the first application program comprises:

accessing, in the global object hierarchy, the first object corresponding to the first active GUI element of the first application program.

14. The system of claim 13, wherein the first computing device maintains a local object hierarchy including a first local object corresponding to the first object in the global object hierarchy, and wherein using the first object comprises providing an indication to the first computing device to use the first local object to cause the first application program to at least partially perform the first action.

15. The system of claim 13, wherein accessing the first object comprises refreshing the global object hierarchy.

16. A system for remotely controlling one or more application programs executing on a respective one or more physical computing devices, the system comprising:

a controller comprising a first processor and configured to perform;

accessing a computer program for controlling at least one application program to perform a task including a first sub-task to be performed by a first application program executing on a first computing device external to the controller;

identifying, using the computer program, a first action to perform in furtherance of the first sub-task; and

providing an indication to the first computing device to control the first application program to at least partially perform the first action; and

the first computing device, communicatively coupled to the controller and comprising a second processor, configured to execute the first application program and to perform;

generating a first object hierarchy comprising a first plurality of objects corresponding to a first plurality of active graphical user interface (GUI) elements of the first application program, wherein the first plurality of active GUI elements comprises at least one active GUI element contained within an outer container of a first graphical user interface through which user input for controlling the first application program can be provided; and

in response to receiving the indication from the controller to at least partially perform the first action, using the first object hierarchy to cause the first application program to at least partially perform the first action at least in part by;

using a first object of the first plurality of objects to cause, via a first active GUI element of the first active GUI elements, the first application program to at least partially perform the first action in furtherance of the first sub-task, the first action programmatically emulating a user'"'"'s interaction with the first active GUI element wherein programmatically emulating the user'"'"'s interaction with the first active GUI element comprises invoking one or more functions of the first application program or an operating system managing execution of the first application program to cause the first application program to perform the first action using the first active GUI element.

17. The system of claim 16,

wherein the task further includes a second sub-task to be performed by a second application program executing on a second computing device different from the first computing device;

and

wherein the controller is further configured to provide an indication to the second computing device to control the second application program to at least partially perform a second action in furtherance of the second sub-task.

18. The system of claim 17, further comprising:

the second computing device, communicatively coupled to the controller, comprising a third processor configured to execute the second application program and to perform;

generating a second object hierarchy comprising a second plurality of objects corresponding to a second plurality of active GUI elements of the second application program; and

in response to receiving the indication from the controller to at least partially perform the second action, using the second object hierarchy to cause the second application program to at least partially perform the second action.

19. The system of claim 16, wherein using the first object hierarchy to cause the first application program to at least partially perform the first action comprises:

accessing in the first object hierarchy, the first object corresponding to the first active GUI element of the first application program.

20. The system of claim 19, wherein accessing the first object comprises refreshing the first object hierarchy.