Software robots for programmatically controlling computer programs to perform tasks

• US 10,156,958 B2
• Filed: 03/03/2016
• Issued: 12/18/2018
• Est. Priority Date: 03/03/2015
• Status: Active Grant

• Current Assignee: Soroco Private Limited
• Original Assignee: Soroco Private Limited
• Inventors: Nychis, George Peter, Murty, Rohan Narayan
• Primary Examiner(s): Hong, Stephen S
• Assistant Examiner(s): Roy, Sanchita
• Application Number:
  US15/059,907
• Publication Number:
  US 20160259655A1
• Time in Patent Office: 1,020 Days
• Field of Search: None
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/32 with visual or acoustical indication of the functioning of the machine
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/36 Preventing errors by testing or debugging software
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/44 Arrangements for executing specific programs
• CPC Class Codes: G06F 9/4488 Object-oriented
• CPC Class Codes: G06F 9/451 : Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/455 : Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
• CPC Class Codes: G06F 9/45508 : Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation
• CPC Class Codes: G06F 9/45512 : Command shells
• CPC Class Codes: G06F 9/45533 : Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/45558 : Hypervisor-specific management and integration aspects

Abstract

A system comprising at least one hardware processor; at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program, the generating comprising: obtaining a hierarchical representation of the active GUI elements of the application program; generating an object for each of at least some of the active GUI elements represented in the hierarchical representation; and organizing the generated objects into the object hierarchy based on relationships among active GUI elements represented in the hierarchical representation; and controlling, using software robot computer program and the generated object hierarchy, the first application program to perform the first sub-task.

Claims

1. A system, comprising:

at least one hardware processor;

at least one non-transitory computer-readable storage medium storing;

a computer program for controlling multiple application programs to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and

processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform;

accessing the computer program;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first and second application programs, the generating comprising;

obtaining a first hierarchical representation of active GUI elements of the first application program from an operating system;

generating, based on the first hierarchical representation of active GUI elements of the first application program, a first portion of the object hierarchy corresponding to active GUI elements of the first application program, the first portion of the object hierarchy comprising a first object corresponding to a first active GUI element of the first application program;

obtaining a second hierarchical representation of active GUI elements of the second application program from a source other than the operating system;

generating, based on the second hierarchical representation of active GUI elements of the second application program, a second portion of the object hierarchy corresponding to the active GUI elements of the second application program, the second portion of the object hierarchy comprising a second object corresponding to a second active GUI element of the second application program, and

combining the first portion of the object hierarchy and the second portion of the object hierarchy;

controlling the first application program to perform the first sub-task using the computer program and the generated object hierarchy at least in part by;

accessing, in the first portion of the object hierarchy, the first object corresponding to the first active GUI element of the first application program, and

using the first object to cause, via the first active GUI element, the first application program to at least partially perform a first action in furtherance of the first sub-task; and

controlling the second application program to perform the second sub-task using the computer program and the generated object hierarchy at least in part by;

accessing, in the second portion of the object hierarchy, the second object corresponding to the second active GUI element of the second application program, and

using the second object to cause, via the second active GUI element, the second application program to at least partially perform a second action in furtherance of the second sub-task.

2. The system of claim 1, wherein the second application program is an Internet browser program, and wherein the processor-executable instructions further cause the at least one hardware processor to:

obtain the second hierarchical representation via a Document Object Model (DOM) representation of a webpage displayed by the Internet browser program.

3. The system of claim 1, wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.

4. The system of claim 1, wherein accessing the first object comprises refreshing the first portion of the object hierarchy and accessing the second object comprises refreshing the second portion of the object hierarchy.

5. The system of claim 1, wherein accessing the first object comprises refreshing the object hierarchy.

6. A system for controlling multiple application programs to perform a task comprising a first sub-task and a second sub-task, the system comprising:

at least one hardware processor;

at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program and a second application program, the generating comprising;

obtaining a first hierarchical representation of active GUI elements of the first application program from an operating system;

generating, based on the first hierarchical representation of active GUI elements of the first application program, a first portion of the object hierarchy corresponding to active GUI elements of the first application program, the first portion of the object hierarchy comprising a first object corresponding to a first active GUI element of the first application program;

obtaining a second hierarchical representation of active GUI elements of the second application program from a source other than the operating system;

generating, based on the second hierarchical representation of active GUI elements of the second application program, a second portion of the object hierarchy corresponding to the active GUI elements of the second application program, the second portion of the object hierarchy comprising a second object corresponding to a second active GUI element of the second application program, and

combining the first portion and the second portion;

controlling the first application program to perform the first sub-task using the object hierarchy at least in part by;

accessing, in the first portion of the object hierarchy, the first object corresponding to the first active GUI element of the first application program, and

using the first object to cause, via the first active GUI element, the first application program to at least partially perform a first action in furtherance of the first sub-task; and

controlling the second application program to perform the second sub-task using the generated object hierarchy at least in part by;

accessing, in the second portion of the object hierarchy, the second object corresponding to the second active GUI element of the second application program, and using the second object to cause, via the second active GUI element, the second

application program to at least partially perform a second action in furtherance of the second sub-task.

7. The system of claim 6, wherein the second application program is an Internet browser program, and wherein the processor-executable instructions further cause the at least one hardware processor to:

obtain the second hierarchical representation via a Document Object Model (DOM) representation of a webpage displayed by the Internet browser program.

8. The system of claim 6, wherein accessing the first object comprises refreshing the object hierarchy.

9. A method for controlling multiple application programs to perform a task comprising a first sub-task and a second sub-task, the method comprising:

using at least one computer hardware processor to perform;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first and second application programs, the generating comprising;

obtaining a first hierarchical representation of active GUI elements of the first application program from an operating system;

generating, based on the first hierarchical representation of active GUI elements of the first application program, a first portion of the object hierarchy corresponding to active GUI elements of the first application program, the first portion of the object hierarchy comprising a first object corresponding to a first active GUI element of the first application program;

obtaining a second hierarchical representation of active GUI elements of the second application program from a source other than the operating system;

generating, based on the second hierarchical representation of active GUI elements of the second application program, a second portion of the object hierarchy corresponding to the active GUI elements of the second application program, the second portion of the object hierarchy comprising a second object corresponding to a second active GUI element of the second application program, and

combining the first portion of the object hierarchy and the second portion of the object hierarchy;

controlling the first application program to perform the first sub-task using the generated object hierarchy at least in part by;

accessing, in the first portion of the object hierarchy, the first object corresponding to the first active GUI element of the first application program, and

using the first object to cause, via the first active GUI element, the first application program to at least partially perform a first action in furtherance of the first sub-task; and

controlling the second application program to perform the second sub-task using the generated object hierarchy at least in part by;

accessing, in the second portion of the object hierarchy, the second object corresponding to the second active GUI element of the second application program, and

using the second object to cause, via the second active GUI element, the second application program to at least partially perform a second action in furtherance of the second sub-task.

10. The method of claim 9, wherein accessing the first object comprises refreshing the object hierarchy.

11. The method of claim 9, wherein accessing the first object comprises refreshing the first portion of the object hierarchy and accessing the second object comprises refreshing the second portion of the object hierarchy.

12. The method of claim 9, wherein the second application program is an Internet browser program, and obtaining a second hierarchical representation comprises:

obtaining the second hierarchical representation via a Document Object Model (DOM) representation of a webpage displayed by the Internet browser program.

13. The method of claim 9, wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.