Software robots for programmatically controlling computer programs to perform tasks

• US 10,268,333 B2
• Filed: 03/03/2016
• Issued: 04/23/2019
• Est. Priority Date: 03/03/2015
• Status: Active Grant

• Current Assignee: Soroco Private Limited
• Original Assignee: Soroco Private Limited
• Inventors: Nychis, George Peter, Murty, Rohan Narayan
• Primary Examiner(s): Patel, Manglesh M
• Application Number:
  US15/059,701
• Publication Number:
  US 20160259652A1
• Time in Patent Office: 1,146 Days
• Field of Search: 715704, 715200
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/32 with visual or acoustical indication of the functioning of the machine
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/36 Preventing errors by testing or debugging software
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/44 Arrangements for executing specific programs
• CPC Class Codes: G06F 9/4488 Object-oriented
• CPC Class Codes: G06F 9/451 : Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/455 : Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
• CPC Class Codes: G06F 9/45508 : Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation
• CPC Class Codes: G06F 9/45512 : Command shells
• CPC Class Codes: G06F 9/45533 : Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/45558 : Hypervisor-specific management and integration aspects

Abstract

A system, comprising at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program; and controlling the first application program to perform a first sub-task. The controlling includes automatically accessing, in the object hierarchy, a first object corresponding to an active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy, the searching comprising modifying the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task.

Claims

1. A system, comprising:

at least one computer hardware processor; and

at least one non-transitory computer-readable storage medium storing;

a computer program for controlling first and second application programs to perform a task comprising a first sub-task to be performed by the first application program and a second sub-task to be performed by the second application program; and

processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first and second application programs, the object hierarchy comprising a first object corresponding to a first active GUI element of the first application program and a second object corresponding to a second active GUI element of the second application program;

controlling the first application program, via the computer program, to perform the first sub-task at least in part by;

automatically accessing, in the object hierarchy, the first object corresponding to the first active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy using a first search string formatted in accordance with a search grammar and specifying the first object as an object to be searched in the object hierarchy, and the searching for the first object comprising;

 

searching for a third object in the object hierarchy;

 

modifying the object hierarchy, when the first application program'"'"'s graphical user interface has changed, by adding one or more new objects to the object hierarchy as descendants of the third object and/or removing one or more objects from the object hierarchy to obtain a modified object hierarchy; and

 

searching for the first object in the modified object hierarchy; and

automatically using the first object to cause, via the first active GUI element, the first application program to at least partially perform a first action in furtherance of the first sub-task; and

controlling the second application program, via the computer program, to perform the second sub-task at least in part by;

automatically accessing, in the object hierarchy, the second object corresponding to the second active GUI element of the second application program, the accessing comprising searching for the second object in the object hierarchy using a second search string formatted in accordance with the search grammar and specifying the second object as an object to be searched in the object hierarchy; and

automatically using the second object to cause, via the second active GUI element, the second application program to at least partially perform a second action in furtherance of the second sub-task.

2. The system of claim 1, wherein the search grammar comprises a token operator, an action operator, and a relationship operator.

3. The system of claim 2, wherein the first search string includes the action operator.

4. The system of claim 3, wherein the action operator indicates how the object hierarchy is to be modified during performance of the search.

5. The system of claim 1, wherein modifying the object hierarchy comprises:

using the third object to cause the first application program to perform an action causing one or more new GUI elements of the first application program to become active; and

modifying the object hierarchy by adding one or more new objects corresponding to the one or more new GUI elements as the descendants of the third object in the object hierarchy.

6. The system of claim 1, wherein:

searching for the third object is performed using a search string formatted in accordance with the search grammar, the search string including an action operator.

7. The system of claim 1, wherein the first active GUI element of the first application program is a visible GUI element of the first application program.

8. The system of claim 1, wherein the first active GUI element of the first application program is a hidden GUI element of the first application program.

9. A system, comprising:

at least one computer hardware processor; and

at least one non-transitory computer-readable storage medium storing;

a computer program for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program; and

processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program to be controlled by the computer program, the object hierarchy comprising a first object corresponding to a first active GUI element of the first application program;

controlling the first application program to perform the first sub-task at least in part by;

identifying, using the computer program, a first action to perform in furtherance of the first sub-task;

automatically accessing, in the object hierarchy, the first object corresponding to the first active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy, the searching comprising;

 

searching for a second object in the object hierarchy;

 

modifying the object hierarchy, when the first application program'"'"'s graphical user interface has changed, by adding one or more new objects to the object hierarchy as descendants of the second object and/or removing one or more objects from the object hierarchy to obtain a modified object hierarchy; and

 

searching for the first object in the modified object hierarchy; and

automatically using the first object to cause, via the first active GUI element, the first application program to at least partially perform the first action.

10. The system of claim 9, wherein searching for the first object is performed using a first search string formatted in accordance with a search grammar.

11. The system of claim 10, wherein the search grammar comprises a token operator, an action operator, and a relationship operator.

12. The system of claim 11, wherein the search string includes the action operator.

13. The system of claim 12, wherein the action operator indicates how the object hierarchy is to be modified during performance of the search.

14. The system of claim 9, wherein modifying the object hierarchy comprises:

using the second object to cause the first application program to perform an action causing one or more new GUI elements of the first application program to become active; and

modifying the object hierarchy by adding one or more new objects corresponding to the one or more new GUI elements as the descendants of the second object in the object hierarchy.

15. The system of claim 9, wherein searching for the second object is performed using a search string formatted in accordance with a search grammar, the search string including an action operator.

16. The system of claim 9, wherein the first active GUI element of the first application program is a visible GUI element of the first application program.

17. A method for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program, the method comprising:

using at least one computer hardware processor to perform;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program, the object hierarchy comprising a first object corresponding to a first active GUI element of the first application program;

controlling the first application program to perform a first sub-task of a task at least in part by;

automatically accessing, in the object hierarchy, the first object corresponding to the first active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy, the searching comprising;

searching for a second object in the object hierarchy;

modifying the object hierarchy, when the first application program'"'"'s graphical user interface has changed, by adding one or more new objects to the object hierarchy as descendants of the second object and/or removing one or more objects from the object hierarchy to obtain a modified object hierarchy; and

searching for the first object in the modified object hierarchy; and

automatically using the first object to cause, via the first active GUI element, the first application program to at least partially perform a first action in furtherance of the first sub-task.