Software robots for programmatically controlling computer programs to perform tasks

• US 9,965,139 B2
• Filed: 03/03/2016
• Issued: 05/08/2018
• Est. Priority Date: 03/03/2015
• Status: Active Grant

• Current Assignee: Soroco Private Limited
• Original Assignee: Soroco Private Limited
• Inventors: Nychis, George Peter, Murty, Rohan Narayan
• Primary Examiner(s): Bullock, Jr., Lewis A
• Assistant Examiner(s): Gooray, Mark A
• Application Number:
  US15/059,816
• Publication Number:
  US 20160259717A1
• Time in Patent Office: 796 Days
• Field of Search: None
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/32 with visual or acoustical indication of the functioning of the machine
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/36 Preventing errors by testing or debugging software
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/44 Arrangements for executing specific programs
• CPC Class Codes: G06F 9/4488 Object-oriented
• CPC Class Codes: G06F 9/451 : Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/455 : Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
• CPC Class Codes: G06F 9/45508 : Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation
• CPC Class Codes: G06F 9/45512 : Command shells
• CPC Class Codes: G06F 9/45533 : Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/45558 : Hypervisor-specific management and integration aspects

Abstract

A system comprising at least one computer hardware processor configured to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program; controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions; during performance of the sequence of actions, generating a log of already-performed actions in the sequence of actions; generating contextual information associated with the already performed actions, the contextual information containing at least some information not in the log and the log containing at least some information not in the contextual information; and providing to a user the log and the contextual information.

Claims

1. A system, comprising:

at least one computer hardware processor; and

at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform;

automatically generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program, the at least one application program including a first application program and a second application program, the generating comprising;

identifying active GUI elements of the at least one application program;

identifying relationships among the active GUI elements;

generating, based on the identified active GUI elements and relationships, a first portion of the object hierarchy comprising objects corresponding to active GUI elements of the first application program and a second portion of the object hierarchy comprising objects corresponding to the active GUI elements of the second application program;

accessing a task comprising a sequence of actions;

performing said sequence of actions by;

controlling the first application program to perform a first action in the sequence of actions using the first portion of the object hierarchy;

refreshing a portion of the object hierarchy according to any changes in the active GUI elements after the first action is performed;

controlling the second application program to perform a second action in the sequence of actions using the second portion of the object hierarchy; and

refreshing a portion of the object hierarchy according to any changes in the active GUI elements after the second action is performed; and

during performance of the sequence of actions,

generating a visual representation of already-performed actions in the sequence of actions;

storing the visual representation;

determining that an error in performance of the sequence of actions has occurred, the error preventing completing automated performance of the sequence of actions without human intervention; and

when it is determined that the error in performance of the sequence of actions has occurred, providing to a user the visual representation.

2. The system of claim 1, wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:

following the user taking corrective action responsive to the error, resuming automated performance of the sequence of actions.

3. The system of claim 1, wherein controlling the first application program to perform the first action in the sequence of actions comprises:

accessing a first object in the object hierarchy corresponding to a first active GUI element of the first application program; and

using the first object to cause the first application program to perform the first action in the sequence of actions.

4. The system of claim 1,

wherein identifying the active GUI elements of the at least one application program is performed based on a hierarchical representation of active GUI elements of the first application program that is provided by a first computer program, and

wherein the first computer program is an operating system.

5. The system of claim 4, wherein identifying the active GUI elements of the at least one application program comprises obtaining the hierarchical representation from the WINDOWS Automation API.

6. The system of claim 4, wherein the first computer program is a JAVA virtual machine.

7. The system of claim 1, wherein generating the object hierarchy is performed without user input.

8. A method, performed by at least one computer hardware processor, the method comprising:

using the at least one computer hardware processor to perform;

automatically generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program, the at least one application program including a first application program and a second application program, the generating comprising;

identifying active GUI elements of the at least one application program;

identifying relationships among the active GUI elements;

generating, based on the identified active GUI elements and relationships, a first portion of the object hierarchy comprising objects corresponding to active GUI elements of the first application program and a second portion of the object hierarchy comprising objects corresponding to the active GUI elements of the second application program;

accessing a task comprising a sequence of actions;

performing said sequence of actions by;

controlling the first application program to perform a first action in the sequence of actions using the first portion of the object hierarchy;

refreshing a portion of the object hierarchy according to any changes in the active GUI elements after the first action is performed;

controlling the second application program to perform a second action in the sequence of actions using the second portion of the object hierarchy; and

refreshing a portion of the object hierarchy according to any changes in the active GUI elements after the second action is performed; and

during performance of the sequence of actions,

generating a visual representation of already-performed actions in the sequence of actions;

storing the visual representation;

determining that an error in performance of the sequence of actions has occurred, the error preventing completing automated performance of the sequence of actions without human intervention; and

when it is determined that the error in performance of the sequence of actions has occurred, providing to a user the visual representation.

9. The method of claim 8, further comprising:

following the user taking corrective action responsive to the error, resuming automated performance of the sequence of actions.

10. The method of claim 8, wherein controlling the first application program to perform the first action in the sequence of actions comprises:

accessing a first object in the object hierarchy corresponding to a first active GUI element of the first application program; and

using the first object to cause the first application program to perform the first action in the sequence of actions.

11. The method of claim 8,

wherein identifying the active GUI elements of the at least one application program is performed based on a hierarchical representation of active GUI elements of the first application program that is provided by a first computer program, and

wherein the first computer program is an operating system.

12. The method of claim 11, wherein the first computer program is a JAVA virtual machine.

13. The method of claim 8, wherein generating the object hierarchy is performed without user input.

14. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:

automatically generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program, the at least one application program including a first application program and a second application program, the generating comprising;

identifying active GUI elements of the at least one application program;

identifying relationships among the active GUI elements;

generating, based on the identified active GUI elements and relationships, a first portion of the object hierarchy comprising objects corresponding to active GUI elements of the first application program and a second portion of the object hierarchy comprising objects corresponding to the active GUI elements of the second application program;

accessing a task comprising a sequence of actions;

performing said sequence of actions by;

controlling the first application program to perform a first action in the sequence of actions using the first portion of the object hierarchy;

refreshing a portion of the object hierarchy according to any changes in the active GUI elements after the first action is performed;

controlling the second application program to perform a second action in the sequence of actions using the second portion of the object hierarchy; and

refreshing a portion of the object hierarchy according to any changes in the active GUI elements after the second action is performed; and

during performance of the sequence of actions,

generating a visual representation of already-performed actions in the sequence of actions;

storing the visual representation;

determining that an error in performance of the sequence of actions has occurred, the error preventing completing automated performance of the sequence of actions without human intervention; and

when it is determined that the error in performance of the sequence of actions has occurred, providing to a user the visual representation.

15. The at least one non-transitory computer-readable storage medium of claim 14,

wherein identifying the active GUI elements of the at least one application program is performed based on a hierarchical representation of active GUI elements of the first application program that is provided by a first computer program, and

wherein the first computer is an operating system.

16. The at least one non-transitory computer-readable storage medium of claim 15, wherein the first computer program is a JAVA virtual machine.

17. The at least one non-transitory computer-readable storage medium of claim 14, wherein controlling the first application program to perform the first action in the sequence of actions comprises:

accessing a first object in the object hierarchy corresponding to a first active GUI element of the first application program; and

using the first object to cause the first application program to perform the first action in the sequence of actions.

18. The at least one non-transitory computer-readable storage medium of claim 14, wherein generating the object hierarchy is performed without user input.