Software robots for programmatically controlling computer programs to perform tasks

• US 10,474,313 B2
• Filed: 03/03/2016
• Issued: 11/12/2019
• Est. Priority Date: 03/03/2015
• Status: Active Grant

• Current Assignee: Soroco Private Limited
• Original Assignee: Soroco Private Limited
• Inventors: Nychis, George Peter, Murty, Rohan Narayan
• Primary Examiner(s): Patel, Manglesh M
• Application Number:
  US15/059,638
• Publication Number:
  US 20160259651A1
• Time in Patent Office: 1,349 Days
• Field of Search: 715704, 715200
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/32 with visual or acoustical indication of the functioning of the machine
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/36 Preventing errors by testing or debugging software
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/44 Arrangements for executing specific programs
• CPC Class Codes: G06F 9/4488 Object-oriented
• CPC Class Codes: G06F 9/451 : Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/455 : Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
• CPC Class Codes: G06F 9/45508 : Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation
• CPC Class Codes: G06F 9/45512 : Command shells
• CPC Class Codes: G06F 9/45533 : Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/45558 : Hypervisor-specific management and integration aspects

Abstract

A system comprising at least one hardware processor configured to perform: accessing a software robot computer program for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program; generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program; and controlling the first application program to perform the first sub-task. The controlling includes identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task; automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising refreshing the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform the first action.

Claims

1. A system, comprising:

at least one hardware processor; and

at least one non-transitory computer-readable storage medium storing;

a computer program for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program; and

processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program, the generating comprising;

identifying a first plurality of active GUI elements associated with a user interface of the first application program; and

generating, as part of the object hierarchy, a first plurality of objects corresponding to the first plurality of active GUI elements; and

controlling the first application program to perform the first sub-task at least in part by;

identifying, using the computer program, a first action to perform in furtherance of the first sub-task;

accessing, in the object hierarchy, a first object of the first plurality of objects corresponding to a first active GUI element of the first plurality of active GUI elements of the first application program;

using the first object to cause, via the first active GUI element, the first application program to at least partially perform the first action;

identifying, using the computer program, a second action to perform in furtherance of the first sub-task;

refreshing the object hierarchy to obtain a refreshed object hierarchy comprising a second plurality of objects corresponding to a second plurality of active GUI elements associated with the user interface of the first application program, the second plurality of objects including a second object not in the first plurality of objects, the second object corresponding to a second active GUI element of the second plurality of active GUI elements; and

using the second object to cause, via the second active GUI element, the first application program to at least partially perform the second action.

2. The system of claim 1, wherein the task further comprises a second sub-task to be performed by a second application program, and wherein the processor-executable instructions further cause the at least one hardware processor to control the second application program to perform the second sub-task at least in part by:

identifying, using the computer program, a third action to perform in furtherance of the second sub-task;

automatically accessing, in the object hierarchy, a third object corresponding to a third active GUI element of the second application program; and

automatically using the third object to cause, via the third active GUI element, the second application program to at least partially perform the third action.

3. The system of claim 2, wherein the task further comprises a third sub-task to be performed by a third application program, and wherein the processor-executable instructions further cause the at least one hardware processor to control the third application program to perform the third sub-task at least in part by:

identifying, using the computer program, a fourth action to perform in furtherance of the third sub-task;

automatically accessing, in the object hierarchy, a fourth object corresponding to a fourth active GUI element of the third application program; and

automatically using the fourth object to cause, via the fourth active GUI element, the third application program to at least partially perform the fourth action.

4. The system of claim 1, wherein refreshing the object hierarchy comprises refreshing only a subset of objects in the object hierarchy.

5. The system of claim 1, wherein refreshing the object hierarchy comprises refreshing only the first object and/or one or more descendant objects of the first object in the object hierarchy.

6. The system of claim 1, wherein refreshing the object hierarchy comprises adding to the object hierarchy a new object associated with an active GUI element of the first application program, wherein the second object comprises the new object.

7. The system of claim 1, wherein refreshing the object hierarchy further comprises removing a particular object from the object hierarchy, wherein the particular object is associated with an inactive GUI element of the first application program.

8. The system of claim 1, wherein refreshing the object hierarchy comprises updating at least one property of at least one object in the object hierarchy.

9. The system of claim 1, wherein refreshing the object hierarchy comprises:

determining that a particular object in the object hierarchy is invalid; and

refreshing the particular object using a reverse tree traversal technique.

10. The system of claim 1, wherein the accessing comprises searching for the first object in the object hierarchy using a search string constructed in accordance with a search grammar, the search grammar comprising at least one token operator, at least one action operator, and at least one relationship operator.

11. The system of claim 1, wherein refreshing the object hierarchy comprises:

refreshing one or more objects in the object hierarchy, wherein each of the one or more objects is a descendant of the first object, and wherein refreshing the one or more objects is performed using a plurality of threads.

12. The system of claim 11, wherein the one or more objects includes a first child object of the first object and a second child object of the first object, and wherein refreshing the one or more objects comprises:

refreshing objects in the object hierarchy that are descendants of the first child object using a first thread in the plurality of threads; and

refreshing objects in the object hierarchy that are descendants of the second child object using a second thread in the plurality of threads.

13. The system of claim 12, wherein the object hierarchy includes a first set of objects associated with the first application program and a second set of objects associated with a second application program, wherein the first set of objects includes more objects than that of the second set of objects, and wherein refreshing objects in the object hierarchy using the plurality of threads comprises using more threads in the plurality of threads to refresh objects in the first set of objects than that to refresh objects in the second set of objects.

14. The system of claim 1, wherein refreshing the object hierarchy comprises refreshing an object in the object hierarchy, and the second object comprises the refreshed object.

15. The system of claim 1, wherein the first active GUI element of the first application program is a visible GUI element of the first application program.

16. The system of claim 1, wherein the first active GUI element of the first application program is a hidden GUI element of the first application program.

17. A method for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program, the method comprising:

using at least one computer hardware processor to perform;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program, the generating comprising;

identifying a first plurality of active GUI elements associated with a user interface of the first application program; and

generating, as part of the object hierarchy, a first plurality of objects corresponding to the first plurality of active GUI elements; and

controlling the first application program to perform the first sub-task at least in part by;

identifying, using a computer program, a first action to perform in furtherance of the first sub-task;

accessing, in the object hierarchy, a first object of the first plurality of objects corresponding to a first active GUI element of the first plurality of active GUI elements of the first application program;

using the first object to cause, via the first active GUI element, the first application program to at least partially perform the first action;

identifying, using the computer program, a second action to perform in furtherance of the first sub-task;

refreshing the object hierarchy to obtain a refreshed object hierarchy comprising a second plurality of objects corresponding to a second plurality of active GUI elements associated with the user interface of the first application program, the second plurality of objects including a second object not in the first plurality of objects, the second object corresponding to a second active GUI element of the second plurality of active GUI elements; and

using the second object to cause, via the second active GUI element, the first application program to at least partially perform the second action.

18. The method of claim 17, wherein refreshing the object hierarchy comprises adding to the object hierarchy a new object associated with an active GUI element of the first application program, and the second object comprises the new object.

19. The method of claim 17, wherein refreshing the object hierarchy comprises refreshing an object in the object hierarchy, and the second object comprises the refreshed object.

20. A non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one hardware processor, cause the at least one hardware processor to perform a method for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program, the method comprising:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program, the generating comprising;

identifying a first plurality of active GUI elements associated with a user interface of the first application program; and

generating, as part of the object hierarchy, a first plurality of objects corresponding to the first plurality of active GUI elements; and

controlling the first application program to perform the first sub-task at least in part by;

identifying, using a computer program, a first action to perform in furtherance of the first sub-task;

accessing, in the object hierarchy, a first object of the first plurality of objects corresponding to a first active GUI element of the first plurality of active GUI elements of the first application program;

using the first object to cause, via the first active GUI element, the first application program to at least partially perform the first action;

identifying, using the computer program, a second action to perform in furtherance of the first sub-task;

refreshing the object hierarchy to obtain a refreshed object hierarchy comprising a second plurality of objects corresponding to a second plurality of active GUI elements associated with the user interface of the first application program, the second plurality of objects including a second object not in the first plurality of objects, the second object corresponding to a second active GUI element of the second plurality of active GUI elements; and

using the second object to cause, via the second active GUI element, the first application program to at least partially perform the second action.